ZTE Corporation

No.50, Software Avenue Nanjing 210012 China wei.yuehua@zte.com.cn

ZTE Corporation

No.50, Software Avenue Nanjing 210012 China zhang.zheng@zte.com.cn

Yandex

fl0w@yandex-team.ru

Individual

France pascal.thubert@gmail.com

Juniper Networks

1194 N. Mathilda Ave Sunnyvale CA 94089 United States of America prz@juniper.net

RTG rift This document discusses the properties, applicability, and operational considerations of Routing in Fat Trees (RIFT) in different network scenarios with the intention of providing a rough guide on how RIFT can be deployed to simplify routing operations in Clos topologies and their variations.

Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at .

Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents () in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.

Table of Contents

• .  Introduction
• .  Terminology
• .  Problem Statement of Routing in Modern IP Fabric Fat Tree Networks
• .  Applicability of RIFT to Clos IP Fabrics
  • .  Overview of RIFT
  • .  Applicable Topologies
    • .  Horizontal Links
    • .  Vertical Shortcuts
    • .  Generalizing to Any Directed Acyclic Graph
    • .  Reachability of Internal Nodes in the Fabric
  • .  Use Cases
    • .  Data Center Topologies
    • .  Metro Networks
    • .  Building Cabling
    • .  Internal Router Switching Fabrics
    • .  CloudCO
• .  Operational Considerations
  • .  South Reflection
  • .  Suboptimal Routing on Link Failures
  • .  Black-Holing on Link Failures
  • .  Zero Touch Provisioning (ZTP)
  • .  Miscabling
    • .  Miscabling Examples
    • .  Miscabling Considerations
  • .  Multicast and Broadcast Implementations
  • .  Positive vs. Negative Disaggregation
  • .  Mobile Edge and Anycast
  • .  IPv4 over IPv6
  • . In-Band Reachability of Nodes
  • . Dual-Homing Servers
  • . Fabric with a Controller
    • .  Controller Attached to ToFs
    • .  Controller Attached to Leaf
  • . Internet Connectivity Within Underlay
    • .  Internet Default on the Leaf
    • .  Internet Default on the ToFs
  • . Subnet Mismatch and Address Families
  • . Anycast Considerations
  • . IoT Applicability
  • . Key Management
  • . TTL/Hop Limit of 1 vs. 255 on LIEs/TIEs
• .  Security Considerations
• .  IANA Considerations
• .  References
  • .  Normative References
  • .  Informative References
• Acknowledgments
• Contributors
• Authors' Addresses

Introduction This document discusses the properties and applicability of "RIFT: Routing in Fat Trees" in different deployment scenarios and highlights the operational simplicity of the technology compared to classical routing solutions. It also documents special considerations when RIFT is used with or without overlays and/or controllers and how RIFT identifies miscablings and reroutes around node and link failures.

Terminology This document uses the terminology defined in . The most frequently used terms and their definitions from that document are listed here.

Clos / Fat Tree:

This document uses the terms "Clos" and "Fat Tree" interchangeably where it always refers to a folded spine-and-leaf topology with possibly multiple Points of Delivery (PoDs) and one or multiple Top of Fabric (ToF) planes. Several modifications such as leaf-2-leaf shortcuts and multiple level shortcuts are possible and described further in the document.

Crossbar:

Physical arrangement of ports in a switching matrix without implying any further scheduling or buffering disciplines.

Directed Acyclic Graph (DAG):

A finite directed graph with no directed cycles (loops). If links in a Clos are considered as either being all directed towards the top or bottom, each of such two graphs is a DAG.

Disaggregation:

The process in which a node decides to advertise more specific prefixes southwards, either positively to attract the corresponding traffic or negatively to repel it. Disaggregation is performed to prevent traffic loss and suboptimal routing to the more specific prefixes.

Leaf:

A node without southbound adjacencies. Level 0 implies a leaf in RIFT, but a leaf does not have to be level 0.

LIE:

This is an acronym for "Link Information Element" exchanged on all the system's links running RIFT to form ThreeWay adjacencies and carry information used to perform RIFT Zero Touch Provisioning (ZTP) of levels.

South Reflection:

Often abbreviated just as "reflection", South Reflection defines a mechanism where South Node TIEs are "reflected" from the level south back up north to allow nodes in the same level without East-West links to be aware of each other's node Topology Information Elements (TIEs).

Spine:

Any nodes north of leaves and south of ToF nodes. Multiple layers of spines in a PoD are possible.

TIE:

This is an acronym for "Topology Information Element". TIEs are exchanged between RIFT nodes to describe parts of a network such as links and address prefixes. A TIE always has a direction and a type. North TIEs (sometimes abbreviated as N-TIEs) are used when dealing with TIEs in the northbound representation, and South-TIEs (sometimes abbreviated as S-TIEs) are used for the southbound equivalent. TIEs have different types, such as node and prefix TIEs.

Problem Statement of Routing in Modern IP Fabric Fat Tree Networks Clos topologies (commonly called a Fat Tree/network in modern IP fabric considerations as a similar term for the original definition of the term Fat Tree) have gained prominence in today's networking, primarily as a result of the paradigm shift towards a centralized data-center-based architecture that delivers a majority of computation and storage services. Current routing protocols were geared towards a network with an irregular topology with isotropic properties and a low degree of connectivity. When applied to Fat Tree topologies:

• They tend to need extensive configuration or provisioning during initialization and adding or removing nodes from the fabric.
• For link-state routing protocols, all nodes including spine-and-leaf nodes learn the entire network topology and routing information, which is actually not needed on the leaf nodes during normal operation. They flood significant amounts of duplicate link-state information between spine-and-leaf nodes during topology updates and convergence events, requiring that additional CPU and link bandwidth be consumed. This may impact the stability and scalability of the fabric, make the fabric less reactive to failures, and prevent the use of cheaper hardware at the lower levels (i.e., spine-and-leaf nodes).

Applicability of RIFT to Clos IP Fabrics Further content of this document assumes that the reader is familiar with the terms and concepts used in the Open Shortest Path First (OSPF), OSPF for IPv6, and Intermediate System to Intermediate System (IS-IS) link-state protocols. outlines the requirements of routing in IP fabrics and RIFT protocol concepts.

Overview of RIFT RIFT is a dynamic routing protocol that is tailored for use in Clos, Fat Tree, and other anisotropic topologies. Therefore, a core property of RIFT is that its operation is sensitive to the structure of the fabric -- it is anisotropic. RIFT acts as a link-state protocol when "pointing north", advertising southward routes to northward peers (parents) through flooding and database synchronization. When "pointing south", RIFT operates hop-by-hop like a distance-vector protocol, typically advertising a fabric default route towards the ToF, aka superspine, to southward peers (children). The fabric default is typically the default route as described in Section "Southbound Default Route Origination" of . The ToF nodes may alternatively originate more specific prefixes (P') southbound instead of the default route. In such a scenario, all addresses carried within the RIFT domain must be contained within P', and it is possible for a leaf that acts as gateway to the Internet to advertise the default route instead. RIFT floods flat link-state information northbound only so that each level obtains the full topology of the levels that are south of it. That information is never flooded East-West or back south again, so a top tier node has a full set of prefixes from the Shortest Path First (SPF) calculation. In the southbound direction, the protocol operates like a "fully summarizing, unidirectional" path-vector protocol or, rather, a distance-vector with implicit split horizon. Routing information, normally just the default route, propagates one hop south and is "re-advertised" by nodes at next lower level.

RIFT Overview +---------------+ +----------------+ | ToF | | ToF | LEVEL 2 + ++------+--+--+-+ ++-+--+----+-----+ | | | | | | | | | ^ + | | | +-------------------------+ | Distance- | +-------------------+ | | | | | Vector | | | | | | | | + South | | | | +--------+ | | | Link-State + | | | | | | | | Flooding | | | +----------------+ | | | North v | | | | | | | | + ++---+-+ +------+ +-+----+ ++----++ | |SPINE | |SPINE | | SPINE| | SPINE| | LEVEL 1 + ++----++ ++---+-+ +-+--+-+ ++----++ | + | | | | | | | | | ^ N Distance- | +-------+ | | +--------+ | | | E Vector | | | | | | | | | +------> South | +-------+ | | | +------+ | | | | + | | | | | | | | | + v ++--++ +-+-++ ++--++ ++--++ + |LEAF| |LEAF| |LEAF| |LEAF| LEVEL 0 +----+ +----+ +----+ +----+

A spine node only has information necessary for its level, which is all destinations south of the node based on SPF calculation, the default route, and potentially disaggregated routes. RIFT combines the advantages of both link-state and distance-vector protocols:

• Fastest possible convergence
• Automatic detection of topology
• Minimal routes/information on Top-of-Rack (ToR) switches, aka leaf nodes
• High degree of ECMP
• Fast decommissioning of nodes
• Maximum propagation speed with flexible prefixes in an update

There are two types of link-state databases that are "north representation" North Topology Information Elements (N-TIEs) and "south representation" South Topology Information Elements (S-TIEs). The N-TIEs contain a link-state topology description of lower levels, and the S-TIEs simply carry default and disaggregated routes for the lower levels. RIFT also eliminates major disadvantages of link-state and distance-vector protocols with the following:

• Reduced and balanced flooding
• Level-constrained automatic neighbor discovery

To achieve this, RIFT builds on the art of IGPs, such as OSPF, IS-IS, Mobile Ad Hoc Network (MANET), and Internet of Things (IoT) to provide unique features:

• Automatic (positive or negative) route disaggregation of northward routes upon fallen leaves
• Recursive operation in the case of negative route disaggregation
• Anisotropic routing that extends a principle seen in the Routing Protocol for Low-Power and Lossy Networks (RPL) to wide superspines
• Optimal flooding reduction that derives from the concept of a "multipoint relay" (MPR) found in Optimized Link State Routing (OLSR) and balances the flooding load over northbound links and nodes

Additional advantages that are unique to RIFT are listed below. The details of these advantages can be found in RIFT.

• True ZTP
• Minimal blast radius on failures
• Can utilize all paths through fabric without looping
• Simple leaf implementation that can scale down to servers
• Key-value store
• Horizontal links used for protection only

Applicable Topologies Albeit RIFT is specified primarily for "proper" Clos or Fat Tree topologies, the protocol natively supports Points of Delivery (PoD) concepts, which, strictly speaking, are not found in the original Clos concept. Further, the specification explains and supports operations of multi-plane Clos variants where the protocol recommends the use of inter-plane rings at the ToF level to allow the reconciliation of topology view of different planes to make the Negative Disaggregation viable in case of failures within a plane. These observations hold not only in case of RIFT but also in the generic case of dynamic routing on Clos variants with multiple planes and failures in bisectional bandwidth, especially on the leaves.

Horizontal Links RIFT is not limited to pure Clos divided into PoD and multi-planes but supports horizontal (East-West) links below the ToF level. Those links are used only for last resort northbound forwarding when a spine loses all its northbound links or cannot compute a default route through them. A full-mesh connectivity between nodes on the same level can be deployed, which allows North SPF (N-SPF) to provide for any node losing all its northbound adjacencies (as long as any of the other nodes in the level are northbound connected) and still participate in northbound forwarding. Note that a "ring" of horizontal links at any level below ToF does not provide a "ring-based protection" scheme since the SPF computation would have to deal with breaking of "loops", an application for which RIFT is not intended.

Vertical Shortcuts Through relaxations of the specified adjacency forming rules, RIFT implementations can be extended to support vertical "shortcuts". The RIFT specification itself does not provide the exact details since the resulting solution suffers from either a much larger blast radius with increased flooding volumes or bow tie problems in the case of maximum aggregation routing.

Generalizing to Any Directed Acyclic Graph RIFT is an anisotropic routing protocol, meaning that it has a sense of direction (northbound, southbound, and East-West) and operates differently depending on the direction. Since a DAG provides a sense of north (the direction of the DAG) and south (the reverse), it can be used to apply RIFT -- an edge in the DAG that has only incoming vertices is a ToF node. There are a number of caveats though:

• The DAG structure must exist before RIFT starts, so there is a need for a companion protocol to establish the logical DAG structure.
• A generic DAG does not have a sense of East and West. The operation specified for East-West links and the southbound reflection between nodes are not applicable. Also, ZTP will derive a sense of depth that will eliminate some links. Variations of ZTP could be derived to meet specific objectives, e.g., make it so that most routers have at least two parents to reach the ToF.
• RIFT applies to any Destination-Oriented DAG (DODAG) where there's only one ToF node and the problem of disaggregation does not exist. In that case, RIFT operates very much like RPL , but uses link-state information for southbound routes (downwards in RPL's terms). For an arbitrary DAG with multiple destinations (ToFs), the way disaggregation happens has to be considered.
• Positive Disaggregation expects that most of the ToF nodes reach most of the leaves, so disaggregation is the exception as opposed to the rule. When this is no longer true, it makes sense to turn off disaggregation and route between the ToF nodes over a ring, a full mesh, a transit network, or a form of area zero. Then again, this operation is similar to RPL operating as a single DODAG with a virtual root.
• In order to aggregate and disaggregate routes, RIFT requires that all the ToF nodes share the full knowledge of the prefixes in the fabric. This can be achieved with a ring as suggested by RIFT, by some preconfiguration, or by using a synchronization with a common repository where all the active prefixes are registered.

Reachability of Internal Nodes in the Fabric RIFT does not require that nodes have reachable addresses in the fabric, though it is clearly desirable for operational purposes. Under normal operating conditions, this can be easily achieved by injecting the node's loopback address into Prefix North TIEs and Prefix South TIEs or other implementation-specific mechanisms. Special considerations arise when a node loses all northbound adjacencies but is not at the top of the fabric. If a spine node loses all northbound links, the spine node doesn't advertise a default route. But if the level of the spine node is auto-determined by ZTP, it will "fall down" as depicted in .

Use Cases

Data Center Topologies

Data Center Fabrics RIFT is suited for applying underlay routing in data center (DC) IP fabrics, with the vast majority of these IP fabrics being Clos architectures (and will be for the foreseeable future). It significantly simplifies operation and deployment of such fabrics as described in for environments compared to extensive proprietary provisioning and operational solutions.

Adaptations to Other Proposed Data Center Topologies

Level Shortcut . +-----+ +-----+ . | | | | .+-+ S0 | | S1 | .| ++---++ ++---++ .| | | | | .| | +------------+ | .| | | +------------+ | .| | | | | .| ++-+--+ +--+-++ .| | | | | .| | A0 | | A1 | .| +-+--++ ++---++ .| | | | | .| | +------------+ | .| | +-----------+ | | .| | | | | .| +-+-+-+ +--+-++ .+-+ | | | . | L0 | | L1 | . +-----+ +-----+

RIFT is not strictly limited to Clos topologies. The protocol only requires a sense of "compass rose directionality" either achieved through configuration or derivation of levels. So conceptually, shortcuts between levels could be included. depicts an example of a shortcut between levels. In this example, suboptimal routing will occur when traffic is sent from L0 to L1 via S0's default route and back down through A0 or A1. In order to avoid that, only default routes from A0 or A1 are used. All leaves would be required to install each other's routes. While various technical and operational challenges may require the use of such modifications, discussion of those topics is outside the scope of this document.

Metro Networks The demand for bandwidth is increasing steadily, driven primarily by environments close to content producers (server farms connection via DC fabrics) but in proximity to content consumers as well. Consumers are often clustered in metro areas with their own network architectures that can benefit from simplified, regular Clos structures. Thus, they can also benefit from RIFT.

Building Cabling Commercial edifices are often cabled in topologies that are either Clos or its isomorphic equivalents. The Clos can grow rather high with many levels. That presents a challenge for classical routing protocols (except BGP and Private Network-Network Interface (PNNI) , which is largely phased-out by now) that do not support an arbitrary number of levels, which RIFT does naturally. Moreover, due to the limited sizes of forwarding tables in network elements of building cabling, the minimum FIB size RIFT maintains under normal conditions is cost-effective in terms of hardware and operational costs.

Internal Router Switching Fabrics It is common in high-speed communications switching and routing devices to use switch fabrics that are interconnection networks inside the devices connecting the input ports to their output ports. For example, a crossbar is one of the switch fabric techniques, even though it is not feasible due to cost, head-of-line blocking, or size trade-offs. Normally, such fabrics are not self-healing or rely on 1:1 or 1+1 protection schemes, but it is conceivable to use RIFT to operate Clos fabrics that can deal effectively with interconnections or subsystem failures in such a module. RIFT is not IP specific and hence any link addressing connecting internal device subnets is conceivable.

CloudCO The Cloud Central Office (CloudCO) is a new stage of the telecom Central Office. It takes the advantage of Software-Defined Networking (SDN) and Network Function Virtualization (NFV) in conjunction with general purpose hardware to optimize current networks. The following figure illustrates this architecture at a high level. It describes a single instance or macro-node of CloudCO that provides a number of value-added services (VASes), a Broadband Access Abstraction (BAA), and virtualized network services. An Access I/O module faces a CloudCO access node and the Customer Premises Equipment (CPE) behind it. A Network I/O module is facing the core network. The two I/O modules are interconnected by a spine-and-leaf fabric .

CloudCO Architecture Example +---------------------+ +----------------------+ | Spine | | Spine | | Switch | | Switch | +------+---+------+-+-+ +--+-+-+-+-----+-------+ | | | | | | | | | | | | | | | | | +-------------------------------+ | | | | | | | | | | | | | | | | | +-------------------------+ | | | | | | | | | | | | | | | | | +----------------------+ | | | | | | | | | | | | | | | | | | | | | +---------------------------------+ | | | | | | | | | | | | | | | | | | | | | | +-----------------------------+ | | | | | | | | | | | | | | | | | | | | | | +--------------------+ | | | | | | | | | | | | | | | | +--+ +-+---+--+ +-+---+--+ +--+----+--+ +-+--+--+ +--+ |L | | Leaf | | Leaf | | Leaf | | Leaf | |L | |S | | Switch | | Switch | | Switch | | Switch| |S | ++-+ +-+-+-+--+ +-+-+-+--+ +--+-+--+--+ ++-+--+-+ +-++ | | | | | | | | | | | | | | | +-+-+-+--+ +-+-+-+--+ +--+-+--+--+ ++-+--+-+ | | |Compute | |Compute | | Compute | |Compute| | | |Node | |Node | | Node | |Node | | | +--------+ +--------+ +----------+ +-------+ | | || VAS5 || || vDHCP|| || vRouter|| ||VAS1 || | | |--------| |--------| |----------| |-------| | | |--------| |--------| |----------| |-------| | | || VAS6 || || VAS3 || || v802.1x|| ||VAS2 || | | |--------| |--------| |----------| |-------| | | |--------| |--------| |----------| |-------| | | || VAS7 || || VAS4 || || vIGMP || ||BAA || | | |--------| |--------| |----------| |-------| | | +--------+ +--------+ +----------+ +-------+ | | | ++-----------+ +---------++ |Network I/O | |Access I/O| +------------+ +----------+

The Spine-Leaf architecture deployed inside CloudCO meets the network requirements of being adaptable, agile, scalable, and dynamic.

Operational Considerations RIFT presents the features for organizations building and operating IP fabrics to simplify the operation and deployments while achieving many desirable properties of a dynamic routing protocol on such a substrate:

• RIFT only floods routing information to the devices that need it.
• RIFT allows for ZTP within the protocol. In its most extreme version, RIFT does not rely on any specific addressing and can operate using IPv6 Neighbor Discovery (ND) only for IP fabric.
• RIFT has provisions to detect common IP fabric miscabling scenarios.
• RIFT automatically negotiates Bidirectional Forwarding Detection (BFD) per link. This allows for IP and micro-BFD to replace Link Aggregation Groups (LAGs) that hide bandwidth imbalances in case of constituent failures. Further automatic link validation techniques similar to those in could be supported as well.
• RIFT inherently solves many problems associated with the use of classical routing topologies with dense meshes and high degrees of ECMP by including automatic bandwidth balancing, flood reduction, and automatic disaggregation on failures while providing maximum aggregation of prefixes in default scenarios. ECMP in RIFT eliminates the need for more Loop-Free Alternate (LFA) procedures.
• RIFT reduces FIB size towards the bottom of the IP fabric where most nodes reside. This allows for cheaper hardware on the edges and introduction of modern IP fabric architectures that encompass server multihoming and other mechanisms.
• RIFT provides valley-free routing that is loop free. A valley-free path allows for reversal of direction at most once from a packet heading northbound to southbound while permitting traversal of horizontal links in the northbound phase. This allows for the use of any such valley-free path in bisectional fabric bandwidth between two destinations irrespective of their metrics that can be used to balance load on the fabric in different ways. Valley-free routing eliminates the need for any specific micro-loop avoidance procedures for RIFT.
• RIFT includes a key-value distribution mechanism that allows for future applications such as automatic provisioning of basic overlay services or automatic key rollovers over whole fabrics.
• RIFT is designed for minimum delay in case of prefix mobility on the fabric. In conjunction with , RIFT can differentiate anycast advertisements from mobility events and retain only the most recent advertisement in the latter case.
• Many further operational and design points collected over many years of routing protocol deployments have been incorporated in RIFT such as fast flooding rates, protection of information lifetimes, and operationally recognizable remote ends of links and node names.

South Reflection South reflection is a mechanism where South Node TIEs are "reflected" back up north to allow nodes in the same level without East-West links to "see" each other. For example, in , Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF21 to ToF22 separately. Respectively, Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF22 to ToF21 separately, so ToF22 and ToF21 see each other's node information as level 2 nodes. In an equivalent fashion, as the result of the south reflection between Spine121-Leaf121-Spine122 and Spine121-Leaf122-Spine122, Spine121 and Spine 122 know each other at level 1.

Suboptimal Routing on Link Failures

Suboptimal Routing Upon Link Failure Use Case +--------+ +--------+ | ToF21 | | ToF22 | LEVEL 2 ++--+-+-++ ++-+--+-++ | | | | | | | + | | | | | | | linkTS8 +------------+ | +-+linkTS3+-+ | | | +-------------+ | | | | | | + | | +---------------------------+ | linkTS7 | | | | | + + + | | | | +-------+linkTS4+------------+ | | | | + + | | | | | | +-------------+--+ | | | | | | | linkTS6 | | +-+----+-+ +-+----+-+ ++--------+ +-+----+-+ |Spine111| |Spine112| |Spine121 | |Spine122| LEVEL 1 +-+---+--+ +-+----+-+ +-+---+---+ +-+----+-+ | | | | | | | | | +-------------+ | + ++XX+linkSL6+---+ + | | | | linkSL5 | | linkSL8 | +-----------+ | | + +---+linkSL7+-+ | + | | | | | | | | +-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+ |Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0 +-+-----+ +-+-----+ +-----+-+ +-+-----+ + + + + Prefix111 Prefix112 Prefix121 Prefix122

As shown in , as the result of the south reflection, Spine121 and Spine 122 know each other via Leaf121 or Leaf 122 at level 1. Without disaggregation mechanisms, the packet from leaf121 to prefix122 will probably go up through linkSL5 to linkTS3 when linkSL6 fails. Then, the packet will go down through linkTS4 to linkSL8 to Leaf122 or go up through linkSL5 to linkTS6, then go down through linkTS8 and linkSL8 to Leaf122 based on the pure default route. This is the case of suboptimal routing or bow tying. With disaggregation mechanisms, Spine122 will detect the failure according to the reflected node S-TIE from Spine121 when linkSL6 fails. Based on the disaggregation algorithm provided by RIFT, Spine122 will explicitly advertise prefix122 in Disaggregated Prefix S-TIE PrefixTIEElement(prefix122, cost 1). The packet from leaf121 to prefix122 will only be sent to linkSL7 following a longest-prefix match to prefix 122 directly, then it will go down through linkSL8 to Leaf122.

Black-Holing on Link Failures

Black-Holing Upon Link Failure Use Case +--------+ +--------+ | ToF 21 | | ToF 22 | LEVEL 2 ++-+--+-++ ++-+--+-++ | | | | | | | + | | | | | | | linkTS8 +--------------+ | +-+linkTS3+X+ | | | +--------------+ linkTS1 | | | | | + | + +-----------------------------+ | linkTS7 | | | + | + + + | | | linkTS2 +-------+linkTS4+X+----------+ | | + + + + | | | | linkTS5 +-+ +------------+--+ | | | + | | | linkTS6 | | +-+----+-+ +-+----+-+ ++-------+ +-+-----++ |Spine111| |Spine112| |Spine121| |Spine122| LEVEL 1 +-+---+--+ ++----+--+ +-+---+--+ +-+----+-+ | | | | | | | | + +---------------+ | + +---+linkSL6+---+ + linkSL1 | | | linkSL5 | | linkSL8 + +--+linkSL3+--+ | | + +---+linkSL7+-+ | + | | | | | | | | +-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+ |Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0 +-+-----+ +-+-----+ +-----+-+ +-----+-+ + + + + Prefix111 Prefix112 Prefix121 Prefix122

This scenario illustrates a case where double link failure occurs and black-holing can happen. Without disaggregation mechanisms, the packet from leaf111 to prefix122 would suffer 50% black-holing based on pure default route when linkTS3 and linkTS4 both fail. The packet is supposed to go up through linkSL1 to linkTS1 and then go down through linkTS3 or linkTS4 will be dropped. The packet is supposed to go up through linkSL3 to linkTS2, then go down through linkTS3 or linkTS4 will be dropped as well. This is the case of black-holing. With disaggregation mechanisms, ToF22 will detect the failure according to the reflected node S-TIE of ToF21 from Spine111\Spine112 when linkTS3 and linkTS4 both fail. Based on the disaggregation algorithm provided by RIFT, ToF22 will explicitly originate an S-TIE with prefix 121 and prefix 122 that is flooded to spines 111, 112, 121, and 122. The packet from leaf111 to prefix122 will not be routed to linkTS1 or linkTS2. The packet from leaf111 to prefix122 will only be routed to linkTS5 or linkTS7 following a longest-prefix match to prefix122.

Zero Touch Provisioning (ZTP) RIFT is designed to require a very minimal configuration to simplify its operation and avoid human errors; based on that minimal information, ZTP auto configures the key operational parameters of all the RIFT nodes, including the System ID of the node that must be unique in the RIFT network and the level of the node in the Fat Tree, which determines which peers are northward "parents" and which are southward "children". ZTP is always on, but its decisions can be overridden when a network administrator prefers to impose its own configuration. In that case, it is the responsibility of the administrator to ensure that the configured parameters are correct, i.e., ensure that the System ID of each node is unique and that the administratively set levels truly reflect the relative position of the nodes in the fabric. It is recommended to let ZTP configure the network, and when ZTP does not configure the network, it is recommended to configure the level of all the nodes to avoid an undesirable interaction between ZTP and the manual configuration. ZTP requires that the administrator points out the ToF nodes to set the baseline from which the fabric topology is derived. The ToF nodes are configured with the TOP_OF_FABRIC flag, which are initial 'seeds' needed for other ZTP nodes to derive their level in the topology. ZTP computes the level of each node based on the Highest Available Level (HAL) of the potential parent closest to that baseline, which represents the superspine. In a fashion, RIFT can be seen as a distance-vector protocol that computes a set of feasible successors towards the superspine and autoconfigures the rest of the topology. The autoconfiguration mechanism computes a global maximum of levels by diffusion. The derivation of the level of each node happens then based on LIEs received from its neighbors, whereas each node (with possible exceptions of configured leaves) tries to attach at the highest possible point in the fabric. This guarantees that even if the diffusion front reaches a node from "below" faster than from "above", it will greedily abandon already negotiated levels derived from nodes topologically below it and properly peer with nodes above. The achieved equilibrium can be disturbed massively by all nodes with the highest level either leaving or entering the domain (with some finer distinctions not explained further). It is therefore recommended that each node is multihomed towards nodes with respective HAL offerings. Fortunately, this is the natural state of things for the topology variants considered in RIFT. A RIFT node may also be configured to confine it to the leaf role with the LEAF_ONLY flag. A leaf node can also be configured to support leaf-2-leaf procedures with the LEAF_2_LEAF flag. In both cases, the node cannot be TOP_OF_FABRIC and its level cannot be configured. RIFT will fully determine the node's level after it is attached to the topology and ensure that the node is at the "bottom of the hierarchy" (southernmost).

Miscabling

Miscabling Examples

A Single-Plane Miscabling Example +----------------+ +-----------------+ | ToF21 | +------+ ToF22 | LEVEL 2 +-------+----+---+ | +----+---+--------+ | | | | | | | | | | | | +----------------------------+ | | +---------------------------+ | | | | | | | | | | | | | | | | | +-----------------------+ | | | | +------------------------+ | | | | | | | | | | | | +-+---+--+ +-+---+--+ | +--+---+-+ +--+---+-+ |Spine111| |Spine112| | |Spine121| |Spine122| LEVEL 1 +-+---+--+ ++----+--+ | +--+---+-+ +-+----+-+ | | | | | | | | | | +---------+ | link-M | +---------+ | | | | | | | | | | | +-------+ | | | | +-------+ | | | | | | | | | | | +-+---+-+ +--+--+-+ | +-+---+-+ +--+--+-+ |Leaf111| |Leaf112+-----+ |Leaf121| |Leaf122| LEVEL 0 +-------+ +-------+ +-------+ +-------+

shows a single-plane miscabling example. It's a perfect Fat Tree fabric except for link-M connecting Leaf112 to ToF22. The RIFT control protocol can discover the physical links automatically and is able to detect cabling that violates Fat Tree topology constraints. It reacts accordingly to such miscabling attempts, preventing adjacencies between nodes from being formed and traffic from being forwarded on those miscabled links at a minimum. In such scenario, Leaf112 will use link-M to derive its level (unless it is leaf) and can report links to Spine111 and Spine112 as miscabled unless the implementations allow horizontal links. shows a multi-plane miscabling example. Since Leaf112 and Spine121 belong to two different PoDs, the adjacency between Leaf112 and Spine121 cannot be formed. Link-W would be detected and prevented.

A Multiple Plane Miscabling Example +-------+ +-------+ +-------+ +-------+ |ToF A1| |ToF A2| |ToF B1| |ToF B2| LEVEL 2 +-------+ +-------+ +-------+ +-------+ | | | | | | | | | | | +-----------------+ | | | | +--------------------------+ | | | | | +------+ | | | +------+ | | | +-----------------+ | | | | | | | | +--------------------------+ | | | A | | B | | A | | B | +-----+--+ +-+---+--+ +--+---+-+ +--+-----+ |Spine111| |Spine112| +---+Spine121| |Spine122| LEVEL 1 +-+---+--+ ++----+--+ | +--+---+-+ +-+----+-+ | | | | | | | | | | +---------+ | | | +---------+ | | | | | link-W | | | | | +-------+ | | | | +-------+ | | | | | | | | | | | +-+---+-+ +--+--+-+ | +-+---+-+ +--+--+-+ |Leaf111| |Leaf112+------+ |Leaf121| |Leaf122| LEVEL 0 +-------+ +-------+ +-------+ +-------+ +--------PoD#1----------+ +---------PoD#2---------+

RIFT provides an optional level determination procedure in its ZTP mode. Nodes in the fabric without their level configured determine it automatically. However, this can have possible counter-intuitive consequences. One extreme failure scenario is depicted in , and it shows that if all northbound links of Spine11 fail at the same time, Spine11 negotiates a lower level than Leaf11 and Leaf12. To prevent such scenario where leaves are expected to act as switches, the LEAF_ONLY flag can be set for Leaf111 and Leaf112. Since level -1 is invalid, Spine11 would not derive a valid level from the topology in . It will be isolated from the whole fabric, and it would be up to the leaves to declare the links towards such spine as miscabled.

Fallen Spine +-------+ +-------+ +-------+ +-------+ |ToF A1| |ToF A2| |ToF A1| |ToF A2| +-------+ +-------+ +-------+ +-------+ | | | | | | | +-------+ | | | + + | | ====> | | X X +------+ | +------+ | + + | | | | +----+--+ +-+-----+ +-+-----+ |Spine11| |Spine12| |Spine12| +-+---+-+ ++----+-+ ++----+-+ | | | | | | | +---------+ | | | | +-------+ | | +-------+ | | | | | | | +-+---+-+ +--+--+-+ +-----+-+ +-----+-+ |Leaf111| |Leaf112| |Leaf111| |Leaf112| +-------+ +-------+ +-+-----+ +-+-----+ | | | +--------+ | | +-+---+-+ |Spine11| +-------+

Miscabling Considerations There are scenarios where operators may want to leverage ZTP and implement additional cabling constraints that go beyond the previously described topology violations. Enforcing cabling down to specific level, node, and port combinations might make it simpler for onsite staff to perform troubleshooting activities or replace optical transceivers and/or cabling as the physical layout will be consistent across the fabric. This is especially true for densely connected fabrics where it is difficult to physically manipulate those components. It is also easy to imagine other models, such as one where the strict port requirement is relaxed. illustrates an example where the first port on Leaf1 must connect to the first port on Spine1, the second port on Leaf1 must connect to the first port on Spine2, and so on. Consider a case where (Leaf1, Port1) and (Leaf1, Port2) were reversed. RIFT would not consider this to be miscabled by default; however, an operator might want to.

Additional Cabling Constraint Example +--------+ +--------+ +--------+ +--------+ | Spine1 | | Spine2 | | Spine3 | | Spine4 | +-1------+ +-1------+ +-1------+ +-1------+ + + + + | +----------+ | | | | | | | | +---------------------+ | | | | | | | | +--------------------------------+ | | | | | | | | | | | | | | | | + + + + +-1--2--3--4--+ | Leaf1 | ...... +-------------+

RIFT allows implementations to provide programmable plug-ins that can adjust ZTP operation or capture information during computation. While defining this is outside the scope of this document, such a mechanism could be used to extend the miscabling functionality. For other protocols to achieve this, it would require additional operational overhead. Consider a fabric that is using unnumbered OSPF links; it is still very likely that a miscabled link will form an adjacency. Each attempt to move cables to the correct port may result in the need for additional troubleshooting as other links will become miscabled in the process. Without automation to explicitly tell the operator which ports need to be moved where, the process becomes manually intensive and error-prone very quickly. If the problem goes unnoticed, it will result in suboptimal performance in the fabric.

Multicast and Broadcast Implementations RIFT supports both multicast and broadcast implementations. While a multicast implementation is preferred, there might cases where a broadcast implementation is optimal or even required. For example, operating systems on IoT devices and embedded devices may not have the required multicast support. Another example is containers, which do support multicast in some cases but tend to be very CPU-inefficient and difficult to tune.

Positive vs. Negative Disaggregation Disaggregation is the procedure whereby RIFT advertises a more specific route southwards as an exception to the aggregated fabric-default north. Disaggregation is useful when a prefix within the aggregation is reachable via some of the parents but not the others at the same level of the fabric. It is mandatory when the level is the ToF since a ToF node that cannot reach a prefix becomes a black hole for that prefix. The hard problem is to know which prefixes are reachable by whom. In the general case, RIFT solves that problem by interconnecting the ToF nodes so that the ToF nodes can exchange the full list of prefixes that exist in the fabric and figure out when a ToF node lacks reachability to some prefixes. This requires additional ports at the ToF, typically two ports per ToF node to form a ToF-spanning ring. RIFT also defines the southbound reflection procedure that enables a parent to explore the direct connectivity of its peers, meaning their own parents and children; based on the advertisements received from the shared parents and children, it may enable the parent to infer the prefixes its peers can reach. When a parent lacks reachability to a prefix, it may disaggregate the prefix negatively, i.e., advertise that this parent can be used to reach any prefix in the aggregation except that one. The Negative Disaggregation signaling is simple and functions transitively from ToF to Top-of-Pod (ToP) and then from ToP to Leaf. However, it is hard for a parent to figure out which prefix it needs to disaggregate because it does not know what it does not know; it results that the use of a spanning ring at the ToF is required to operate the Negative Disaggregation. Also, though it is only an implementation problem, the programming of the FIB is complex compared to normal routes and may incur recursions. The more classical alternative is, for the parents that can reach a prefix that peers at the same level cannot, to advertise a more specific route to that prefix. This leverages the normal longest prefix match in the FIB and does not require a special implementation. As opposed to the Negative Disaggregation, the Positive Disaggregation is difficult and inefficient to operate transitively. Transitivity is not needed by a grandchild if all its parents received the Positive Disaggregation, meaning that they shall all avoid the black hole; when that is the case, they collectively build a ceiling that protects the grandchild. Until then, a parent that received the Positive Disaggregation may believe that some peers are lacking the reachability and re-advertise too early or defer and maintain a black hole situation longer than necessary. In a non-partitioned fabric, all the ToF nodes see one another through the reflection and can figure out if one is missing a child. In that case, it is possible to compute the prefixes that the peer cannot reach and disaggregate positively without a ToF-spanning ring. The ToF nodes can also ascertain that the ToP nodes are each connected to at least a ToF node that can still reach the prefix, meaning that the transitive operation is not required. The bottom line is that in a fabric that is partitioned (e.g., using multiple planes) and/or where the ToP nodes are not guaranteed to always form a ceiling for their children, it is mandatory to use Negative Disaggregation. On the other hand, in a highly symmetrical and fully connected fabric (e.g., a canonical Clos Network), the Positive Disaggregation methods save the complexity and cost associated to the ToF-spanning ring. Note that in the case of Positive Disaggregation, the first ToF nodes that announce a more-specific route attract all the traffic for that route and may suffer from a transient incast. A ToP node that defers injecting the longer prefix in the FIB, in order to receive more advertisements and spread the packets better, also keeps on sending a portion of the traffic to the black hole in the meantime. In the case of Negative Disaggregation, the last ToF nodes that inject the route may also incur an incast issue; this problem would occur if a prefix that becomes totally unreachable is disaggregated.

Mobile Edge and Anycast When a physical or a virtual node changes its point of attachment in the fabric from a previous-leaf to a next-leaf, new routes must be installed that supersede the old ones. Since the flooding flows northwards, the nodes (if any) between the previous-leaf and the common parent are not immediately aware that the path via the previous-leaf is obsolete and a stale route may exist for a while. The common parent needs to select the freshest route advertisement in order to install the correct route via the next-leaf. This requires that the fabric determines the sequence of the movements of the mobile node. On the one hand, a classical sequence counter provides a total order for a while, but it will eventually wrap. On the other hand, a timestamp provides a permanent order, but it may miss a movement that happens too quickly vs. the granularity of the timing information. It is not envisioned that an average fabric supports the Precision Time Protocol in the short term nor that the precision available with the Network Time Protocol (in the order of 100 to 200 ms) may not be necessarily enough to cover, e.g., the fast mobility of a Virtual Machine (VM). Section "Mobility" of specifies a hybrid method that combines a sequence counter from the mobile node and a timestamp from the network taken at the leaf when the route is injected. If the timestamps of the concurrent advertisements are comparable (i.e., more distant than the precision of the timing protocol), then the timestamp alone is used to determine the relative freshness of the routes. Otherwise, the sequence counter from the mobile node is used if it is available. One caveat is that the sequence counter must not wrap within the precision of the timing protocol. Another is that the mobile node may not even provide a sequence counter; in which case, the mobility itself must be slower than the precision of the timing. Mobility must not be confused with anycast. In both cases, the same address is injected in RIFT at different leaves. In the case of mobility, only the freshest route must be conserved since the mobile node changes its point of attachment for a leaf to the next. In the case of anycast, the node may either be multihomed (attached to multiple leaves in parallel) or reachable beyond the fabric via multiple routes that are redistributed to different leaves. Either way, the multiple routes are equally valid and should be conserved in the case of anycast. Without further information from the redistributed routing protocol, it is impossible to sort out a movement from a redistribution that happens asynchronously on different leaves. RIFT expects that anycast addresses are advertised within the timing precision, which is typically the case with a low-precision timing and a multihomed node. Beyond that time interval, RIFT interprets the lag as a mobility and only the freshest route is retained. When using IPv6, RIFT suggests leveraging 6LoWPAN ND as the IPv6 ND interaction between the mobile node and the leaf. This not only provides a sequence counter but also a lifetime and a security token that may be used to protect the ownership of an address . When using 6LoWPAN ND , the parallel registration of an anycast address to multiple leaves is done with the same sequence counter, whereas the sequence counter is incremented when the point of attachment changes. This way, it is possible to differentiate a mobile node from a multihomed node, even when the mobility happens within the timing precision. It is also possible for a mobile node to be multihomed as well, e.g., to change only one of its points of attachment.

IPv4 over IPv6 RIFT allows advertising IPv4 prefixes over an IPv6 RIFT network. An IPv6 Address Family (AF) configures via the usual ND mechanisms and then V4 can use V6 next-hops analogous to . It is expected that the whole fabric supports the same type of forwarding of AFs on all the links. RIFT provides an indication whether a node is capable of V4-forwarding and implementations are possible where different routing tables are computed per AF as long as the computation remains loop-free.

IPv4 over IPv6 +-----+ +-----+ +---+---+ | ToF | | ToF | ^ +--+--+ +-----+ | | | | | | | +-------------+ | | | +--------+ | | + | | | | V6 +-----+ +-+---+ Forwarding |Spine| |Spine| + +--+--+ +-----+ | | | | | | | +-------------+ | | | +--------+ | | | | | | | v +-----+ +-+---+ +---+---+ |Leaf | | Leaf| +--+--+ +--+--+ | | IPv4 prefixes| |IPv4 prefixes | | +---+----+ +---+----+ | V4 | | V4 | | subnet | | subnet | +--------+ +--------+

In-Band Reachability of Nodes RIFT doesn't precondition that nodes of the fabric have reachable addresses, but operational reasons to reach the internal nodes may exist. shows an example that the network management station (NMS) attaches to Leaf1.

In-Band Reachability of Nodes +-------+ +-------+ | ToF1 | | ToF2 | ++---- ++ ++-----++ | | | | | +----------+ | | +--------+ | | | | | | ++-----++ +--+---++ |Spine1 | |Spine2 | ++-----++ ++-----++ | | | | | +----------+ | | +--------+ | | | | | | ++-----++ +--+---++ | Leaf1 | | Leaf2 | +---+---+ +-------+ | |NMS

If the NMS wants to access Leaf2, it simply works because the loopback address of Leaf2 is flooded in its Prefix North TIE. If the NMS wants to access Spine2, it also works because a spine node always advertises its loopback address in the Prefix North TIE. The NMS may reach Spine2 from Leaf1-Spine2 or Leaf1-Spine1-ToF1/ToF2-Spine2. If the NMS wants to access ToF2, ToF2's loopback address needs to be injected into its Prefix South TIE. This TIE must be seen by all nodes at the level below -- the spine nodes in -- that must form a ceiling for all the traffic coming from below (south). Otherwise, the traffic from the NMS may follow the default route to the wrong ToF Node, e.g., ToF1. In the case of failure between ToF2 and spine nodes, ToF2's loopback address must be disaggregated recursively all the way to the leaves. In a partitioned ToF, even with recursive disaggregation, a ToF node is only reachable within its plane. A possible alternative to recursive disaggregation is to use a ring that interconnects the ToF nodes to transmit packets between them for their loopback addresses only. The idea is that this is mostly control traffic and should not alter the load-balancing properties of the fabric.

Dual-Homing Servers Each RIFT node may operate in ZTP mode. It has no configuration (unless it is a ToF node at the top of the topology or if it must operate in the topology as a leaf and/or support leaf-2-leaf procedures), and it will fully configure itself after being attached to the topology.

Dual-Homing Servers +---+ +---+ +---+ |ToF| |ToF| |ToF| ToF +---+ +---+ +---+ | | | | | | | +----------------+ | | | +----------------+ | | | | | | | +----------+--+ +--+----------+ | ToR1 | | ToR2 | Spine +--+------+---+ +--+-------+--+ +---+ | | | | | | +---+ | +-----------------+ | | | | | | +-------------+ | | | | | | | +-----------------+ | | | | | +--------------+ | | | | | | | | | | | +---+ +---+ +---+ +---+ | | | | | | | | +---+ +---+ ............. +---+ +---+ SV(1) SV(2) SV(n-1) SV(n) Leaf

Sometimes people may prefer to disaggregate from ToR nodes to servers from startup, i.e., the servers have multiple routes in the FIB from startup other than default routes to avoid breakages at the rack level. Full disaggregation of the fabric could be achieved by configuration supported by RIFT.

Fabric with a Controller There are many different ways to deploy the controller. One possibility is attaching a controller to the RIFT domain from ToF and another possibility is attaching a controller from the leaf.

Fabric with a Controller +------------+ | Controller | ++----------++ | | | | +----++ ++----+ ------- | ToF | | ToF | | +--+--+ +-----+ | | | | | | | +-------------+ | | | +--------+ | | | | | | | +-----+ +-+---+ RIFT domain |Spine| |Spine| +--+--+ +-----+ | | | | | | | +-------------+ | | | +--------+ | | | | | | | | +-----+ +-+---+ ------- |Leaf | | Leaf| +-----+ +-----+

Controller Attached to ToFs If a controller is attaching to the RIFT domain from ToF, it usually uses dual-homing connections. The loopback prefix of the controller should be advertised down by the ToF and spine to the leaves. If the controller loses the link to ToF, make sure the ToF withdraws the prefix of the controller.

Controller Attached to Leaf If the controller is attaching from a leaf to the fabric, no special provisions are needed.

Internet Connectivity Within Underlay If global addressing is running without overlay, an external default route needs to be advertised through the RIFT fabric to achieve internet connectivity. For the purpose of forwarding of the entire RIFT fabric, an internal fabric prefix needs to be advertised in the Prefix South TIE by ToF and spine nodes.

Internet Default on the Leaf In the case that the internet gateway is a leaf, the leaf node as the internet gateway needs to advertise a default route in its Prefix North TIE.

Internet Default on the ToFs In the case that the internet gateway is a ToF, the ToF and spine nodes need to advertise a default route in the Prefix South TIE.

Subnet Mismatch and Address Families

Subnet Mismatch +--------+ +--------+ | | LIE LIE | | | A | +----> <----+ | B | | +---------------------+ | +--------+ +--------+ X/24 Y/24

LIEs are exchanged over all links running RIFT to perform Link (Neighbor) Discovery. A node must NOT originate LIEs on an AF if it does not process received LIEs on that family. LIEs on the same link are considered part of the same negotiation independent from the AF they arrive on. An implementation must be ready to accept TIEs on all addresses it used as the source of LIE frames. As shown in , an adjacency of nodes A and B may form without further checks, but the forwarding between nodes A and B may fail because subnet X mismatches with subnet Y. To prevent this, a RIFT implementation should check for subnet mismatch in a way that is similar to how IS-IS does. This can lead to scenarios where an adjacency, despite the exchange of LIEs in both AFs, may end up having an adjacency in a single AF only. This is especially a consideration in scenarios relating to .

Anycast Considerations

Anycast + traffic | v +------+------+ | ToF | +---+-----+---+ | | | | +------------+ | | +------------+ | | | | +---+---+ +-------+ +-------+ +---+---+ | | | | | | | | |Spine11| |Spine12| |Spine21| |Spine22| LEVEL 1 +-+---+-+ ++----+-+ +-+---+-+ ++----+-+ | | | | | | | | | +---------+ | | +---------+ | | +-------+ | | | +-------+ | | | | | | | | | | +-+---+-+ +--+--+-+ +-+---+-+ +--+--+-+ | | | | | | | | |Leaf111| |Leaf112| |Leaf121| |Leaf122| LEVEL 0 +-+-----+ ++------+ +-----+-+ +-----+-+ + + + ^ + PrefixA PrefixB PrefixA | PrefixC | + traffic

If the traffic comes from ToF to Leaf111 or Leaf121, which has anycast prefix PrefixA, RIFT can deal with this case well. However, if the traffic comes from Leaf122, it arrives to Spine21 or Spine22 at LEVEL 1. Additionally, Spine21 or Spine22 doesn't know another PrefixA attaching Leaf111, so it will always get to Leaf121 and never Leaf111. If the intention is that the traffic should be offloaded to Leaf111, then use the policy-guided prefixes defined in RIFT.

IoT Applicability The design of RIFT inherits the anisotropic design of a default route upwards (northwards) from RPL . It also inherits the capability to inject external host routes at the Leaf level using Wireless ND (WiND) between a RIFT-agnostic host and a RIFT router. Both the RPL and the RIFT protocols are meant for a large scale, and WiND enables device mobility at the edge the same way in both cases. The main difference between RIFT and RPL is that there's a single root with RPL, whereas RIFT has many ToF nodes. This adds huge capabilities for leaf-2-leaf ECMP paths but additional complexity with the need to disaggregate. Also, RIFT uses link-state flooding northwards and is not designed for low-power operation. Still, nothing prevents that the IP devices connected at the Leaf are IoT devices, which typically expose their address using WiND -- this is an upgrade from 6LoWPAN ND . A network that serves high speed / high power IoT devices should typically provide deterministic capabilities for applications such as high speed control loops or movement detection. The Fat Tree is highly reliable and, in normal conditions, provides an equivalent multipath operation; however, the ECMP doesn't provide hard guarantees for either delivery or latency. As long as the fabric is non-blocking, the result is the same, but there can be load unbalances resulting in incast and possibly congestion loss that will prevent the delivery within bounded latency. This could be alleviated with Packet Replication, Elimination, and Ordering Functions (PREOF) leaf-2-leaf, but PREOF is hard to provide at the scale of all flows and the replication may increase the probability of the overload that it attempts to solve. Note that the load balancing is not RIFT's problem, but it is key to serve IoT adequately.

Key Management As outlined in Section "Security Considerations" of , either a private shared key or a public/private key pair is used to authenticate the adjacency. Both the key distribution and key synchronization methods are out of scope for this document. Both nodes in the adjacency must share the same keys, key type, and algorithm for a given key ID. Mismatched keys will not interoperate as their security envelopes will be unverifiable. Key rollover while the adjacency is active may be supported. The specific mechanism is well documented in . As outlined in "Host Implementations" of , hosts as well as VMs acting as RIFT devices are possible. Key Management Protocols (KMPs), such as Key Value (KV) for key rollover in the fabric, use a symmetric key that can be changed easily when compromised; in which case, the symmetric key of a host is more likely to be compromised than an in-fabric networking node.

TTL/Hop Limit of 1 vs. 255 on LIEs/TIEs The use of a packet's Time to Live (TTL) (IPv4) or Hop Limit (IPv6) to verify whether the packet was originated by an adjacent node on a connected link has been used in RIFT. RIFT explicitly requires the use of a TTL/HL value of 1 or 255 when sending/receiving LIEs and TIEs so that implementers have a choice between the two. TTL=1 or HL=1 protects against the information disseminating more than 1 hop in the fabric and should be the default unless configured otherwise. TTL=255 or HL=255 can lead RIFT TIE packet propagation to more than one hop (the multicast address is already in local subnetwork range) in case of implementation problems but does protect against a remote attack as well, and the receiving remote router will ignore such TIE packet unless the remote router is exactly 254 hops away and accepts only TTL=1 or HL=1. defines a Generalized TTL Security Mechanism (GTSM). The GTSM is applicable to LIE/TIE implementations that use a TTL or HL of 255. It provides a defense from infrastructure attacks based on forged protocol packets from outside the fabric.

Security Considerations This document presents applicability of RIFT. As such, it does not introduce any security considerations. However, there are a number of security concerns in .

IANA Considerations This document has no IANA actions.

References Normative References ISO/IEC This memo documents version 2 of the OSPF protocol. OSPF is a link- state routing protocol. [STANDARDS-TRACK] This document specifies the Neighbor Discovery protocol for IP Version 6. IPv6 nodes on the same link use Neighbor Discovery to discover each other's presence, to determine each other's link-layer addresses, to find routers, and to maintain reachability information about the paths to active neighbors. [STANDARDS-TRACK] The use of a packet's Time to Live (TTL) (IPv4) or Hop Limit (IPv6) to verify whether the packet was originated by an adjacent node on a connected link has been used in many recent protocols. This document generalizes this technique. This document obsoletes Experimental RFC 3682. [STANDARDS-TRACK] This document describes the modifications to OSPF to support version 6 of the Internet Protocol (IPv6). The fundamental mechanisms of OSPF (flooding, Designated Router (DR) election, area support, Short Path First (SPF) calculations, etc.) remain unchanged. However, some changes have been necessary, either due to changes in protocol semantics between IPv4 and IPv6, or simply to handle the increased address size of IPv6. These modifications will necessitate incrementing the protocol version from version 2 to version 3. OSPF for IPv6 is also referred to as OSPF version 3 (OSPFv3). Changes between OSPF for IPv4, OSPF Version 2, and OSPF for IPv6 as described herein include the following. Addressing semantics have been removed from OSPF packets and the basic Link State Advertisements (LSAs). New LSAs have been created to carry IPv6 addresses and prefixes. OSPF now runs on a per-link basis rather than on a per-IP-subnet basis. Flooding scope for LSAs has been generalized. Authentication has been removed from the OSPF protocol and instead relies on IPv6's Authentication Header and Encapsulating Security Payload (ESP). Even with larger IPv6 addresses, most packets in OSPF for IPv6 are almost as compact as those in OSPF for IPv4. Most fields and packet- size limitations present in OSPF for IPv4 have been relaxed. In addition, option handling has been made more flexible. All of OSPF for IPv4's optional capabilities, including demand circuit support and Not-So-Stubby Areas (NSSAs), are also supported in OSPF for IPv6. [STANDARDS-TRACK] The One-way Active Measurement Protocol (OWAMP), specified in RFC 4656, provides a common protocol for measuring one-way metrics between network devices. OWAMP can be used bi-directionally to measure one-way metrics in both directions between two network elements. However, it does not accommodate round-trip or two-way measurements. This memo specifies a Two-Way Active Measurement Protocol (TWAMP), based on the OWAMP, that adds two-way or round-trip measurement capabilities. The TWAMP measurement architecture is usually comprised of two hosts with specific roles, and this allows for some protocol simplifications, making it an attractive alternative in some circumstances. [STANDARDS-TRACK] This document is one of a series concerned with defining a roadmap of protocol specification work for the use of modern cryptographic mechanisms and algorithms for message authentication in routing protocols. In particular, it defines the framework for a key management protocol that may be used to create and manage session keys for message authentication and integrity. This document is not an Internet Standards Track specification; it is published for informational purposes. Low-Power and Lossy Networks (LLNs) are a class of network in which both the routers and their interconnect are constrained. LLN routers typically operate with constraints on processing power, memory, and energy (battery power). Their interconnects are characterized by high loss rates, low data rates, and instability. LLNs are comprised of anything from a few dozen to thousands of routers. Supported traffic flows include point-to-point (between devices inside the LLN), point-to-multipoint (from a central control point to a subset of devices inside the LLN), and multipoint-to-point (from devices inside the LLN towards a central control point). This document specifies the IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL), which provides a mechanism whereby multipoint-to-point traffic from devices inside the LLN towards a central control point as well as point-to-multipoint traffic from the central control point to the devices inside the LLN are supported. Support for point-to-point traffic is also available. [STANDARDS-TRACK] The IETF work in IPv6 over Low-power Wireless Personal Area Network (6LoWPAN) defines 6LoWPANs such as IEEE 802.15.4. This and other similar link technologies have limited or no usage of multicast signaling due to energy conservation. In addition, the wireless network may not strictly follow the traditional concept of IP subnets and IP links. IPv6 Neighbor Discovery was not designed for non- transitive wireless links, as its reliance on the traditional IPv6 link concept and its heavy use of multicast make it inefficient and sometimes impractical in a low-power and lossy network. This document describes simple optimizations to IPv6 Neighbor Discovery, its addressing mechanisms, and duplicate address detection for Low- power Wireless Personal Area Networks and similar networks. The document thus updates RFC 4944 to specify the use of the optimizations defined here. [STANDARDS-TRACK] This document defines a mechanism to run Bidirectional Forwarding Detection (BFD) on Link Aggregation Group (LAG) interfaces. It does so by running an independent Asynchronous mode BFD session on every LAG member link. This mechanism allows the verification of member link continuity, either in combination with, or in absence of, Link Aggregation Control Protocol (LACP). It provides a shorter detection time than what LACP offers. The continuity check can also cover elements of Layer 3 (L3) bidirectional forwarding. This document provides the overall architecture for Deterministic Networking (DetNet), which provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency within a network domain. Techniques used include 1) reserving data-plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes along the path of the flow, 2) providing explicit routes for DetNet flows that do not immediately change with the network topology, and 3) distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet's data in spite of the loss of a path. DetNet operates at the IP layer and delivers service over lower-layer technologies such as MPLS and Time- Sensitive Networking (TSN) as defined by IEEE 802.1. Multiprotocol BGP (MP-BGP) specifies that the set of usable next-hop address families is determined by the Address Family Identifier (AFI) and the Subsequent Address Family Identifier (SAFI). The AFI/SAFI definitions for the IPv4 address family only have provisions for advertising a next-hop address that belongs to the IPv4 protocol when advertising IPv4 Network Layer Reachability Information (NLRI) or VPN-IPv4 NLRI. This document specifies the extensions necessary to allow the advertising of IPv4 NLRI or VPN-IPv4 NLRI with a next-hop address that belongs to the IPv6 protocol. This comprises an extension of the AFI/SAFI definitions to allow the address of the next hop for IPv4 NLRI or VPN-IPv4 NLRI to also belong to the IPv6 protocol, the encoding of the next hop to determine which of the protocols the address actually belongs to, and a BGP Capability allowing MP-BGP peers to dynamically discover whether they can exchange IPv4 NLRI and VPN-IPv4 NLRI with an IPv6 next hop. This document obsoletes RFC 5549. Juniper Networks Juniper Networks Hudson River Trading Individual Individual Yandex Broadband Forum Technical Report TR-384, Issue 1 Informative References 2011 IEEE International Parallel & Distributed Processing Symposium IEEE Transactions on Computers, vol. C-34, no. 10, pp. 892-901 IEEE The ATM Forum Technical Committee af-pnni-0055.001 This document describes the Optimized Link State Routing (OLSR) protocol for mobile ad hoc networks. The protocol is an optimization of the classical link state algorithm tailored to the requirements of a mobile wireless LAN. The key concept used in the protocol is that of multipoint relays (MPRs). MPRs are selected nodes which forward broadcast messages during the flooding process. This technique substantially reduces the message overhead as compared to a classical flooding mechanism, where every node retransmits each message when it receives the first copy of the message. In OLSR, link state information is generated only by nodes elected as MPRs. Thus, a second optimization is achieved by minimizing the number of control messages flooded in the network. As a third optimization, an MPR node may chose to report only links between itself and its MPR selectors. Hence, as contrary to the classic link state algorithm, partial link state information is distributed in the network. This information is then used for route calculation. OLSR provides optimal routes (in terms of number of hops). The protocol is particularly suitable for large and dense networks as the technique of MPRs works well in this context. This document discusses the Border Gateway Protocol (BGP), which is an inter-Autonomous System routing protocol. The primary function of a BGP speaking system is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASes) that reachability information traverses. This information is sufficient for constructing a graph of AS connectivity for this reachability from which routing loops may be pruned, and, at the AS level, some policy decisions may be enforced. BGP-4 provides a set of mechanisms for supporting Classless Inter-Domain Routing (CIDR). These mechanisms include support for advertising a set of destinations as an IP prefix, and eliminating the concept of network "class" within BGP. BGP-4 also introduces mechanisms that allow aggregation of routes, including aggregation of AS paths. This document obsoletes RFC 1771. [STANDARDS-TRACK] The Network Time Protocol (NTP) is widely used to synchronize computer clocks in the Internet. This document describes NTP version 4 (NTPv4), which is backwards compatible with NTP version 3 (NTPv3), described in RFC 1305, as well as previous versions of the protocol. NTPv4 includes a modified protocol header to accommodate the Internet Protocol version 6 address family. NTPv4 includes fundamental improvements in the mitigation and discipline algorithms that extend the potential accuracy to the tens of microseconds with modern workstations and fast LANs. It includes a dynamic server discovery scheme, so that in many cases, specific server configuration is not required. It corrects certain errors in the NTPv3 design and implementation and includes an optional extension mechanism. This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. This specification updates RFC 6775 -- the Low-Power Wireless Personal Area Network (6LoWPAN) Neighbor Discovery specification -- to clarify the role of the protocol as a registration technique and simplify the registration operation in 6LoWPAN routers, as well as to provide enhancements to the registration capabilities and mobility detection for different network topologies, including the Routing Registrars performing routing for host routes and/or proxy Neighbor Discovery in a low-power network. This document updates the IPv6 over Low-Power Wireless Personal Area Network (6LoWPAN) Neighbor Discovery (ND) protocol defined in RFCs 6775 and 8505. The new extension is called Address-Protected Neighbor Discovery (AP-ND), and it protects the owner of an address against address theft and impersonation attacks in a Low-Power and Lossy Network (LLN). Nodes supporting this extension compute a cryptographic identifier (Crypto-ID), and use it with one or more of their Registered Addresses. The Crypto-ID identifies the owner of the Registered Address and can be used to provide proof of ownership of the Registered Addresses. Once an address is registered with the Crypto-ID and a proof of ownership is provided, only the owner of that address can modify the registration information, thereby enforcing Source Address Validation.

Acknowledgments The authors would like to thank , , , and for providing invaluable concepts and content for this document.

Contributors The following people contributed substantially to the content of this document and should be considered coauthors: Juniper Networks

jhead@juniper.net

Juniper Networks

tverhaeg@juniper.net

Authors' Addresses ZTE Corporation

No.50, Software Avenue Nanjing 210012 China wei.yuehua@zte.com.cn

ZTE Corporation

No.50, Software Avenue Nanjing 210012 China zhang.zheng@zte.com.cn

Yandex

fl0w@yandex-team.ru

Individual

France pascal.thubert@gmail.com

Juniper Networks

1194 N. Mathilda Ave Sunnyvale CA 94089 United States of America prz@juniper.net