Routing on Flat Labels Matthew Caesar, Tyson Condie, Jayanthkumar Kannan, Karthik Lakshminarayanan, Ion Stoica, Scott Shenker Appeared in Sigcomm 2006 Presented by: Jackson Pang CS 217B, Spring 2009 Routing on Flat Labels

What’s wrong with Internet addressing today? Hierarchical addressing allows excellent scaling properties But, forces addressing to conform to network topology Since topology is not static, addresses can’t persistently identify hosts Routing on Flat Labels

Topology-Based Addressing Disadvantages: complicates Access control Topology changes Multi-homing Mobility Advantage Scalability … Area 1 Area 2 Area 4 Area 3 B J S K Q F V X A 1.1 1.2 2.1 2.2 4.2 4.1 3.3 3.2 3.1 Routing on Flat Labels

What’s wrong with Internet addressing today? The concept of location vs. identity in IP is mixed Most network applications today require persistent identity It’s hard to provide persistent identity in presence of hierarchical addressing Need to decouple identity from addressing Drastically complicates network configuration, mobility, address assignment Is hierarchy the only way to scale routing? Routing on Flat Labels

Motivation for Flat Identifiers Stable references Shouldn’t have to change when object moves Object replication Store object at many different locations Avoid fighting over names Avoid cyber squatting, typo squatting, … Current Proposed <A HREF= http://isp.com/dog.jpg >my friend’s dog</A> <A HREF= http://f0120123112/ >my friend’s dog</A> Routing on Flat Labels

Is there an alternative? Why not route on “flat” host identifiers? Assign addresses independently of network topology Benefits: No separate name resolution system required Simpler network config/allocation/mobility Simpler network-layer access controls Routing on Flat Labels

Basic idea behind ROFL Scalable routing on flat identifiers Goal #1: Scale to Internet topologies How do you fetch a web page by typing google.com? Goal #2: Support for BGP policies How do you preserve the customer – provider, peering relationships? Highly challenging problem, but solution would give a number of benefits Routing on Flat Labels

Basic mechanisms behind ROFL Goal #1: Scale to Internet topologies Mechanism: DHT-style routing, maintain source-routes to successors/fingers Provides: Scalable network routing without aggregation Goal #2: Support for BGP policies Mechanism: Intelligently choose successors/fingers to conform to ISP relationships Provides: Support for policies, operational model of BGP Routing on Flat Labels

How ROFL works ISP 4. intermediate routers may cache pointers 0x3B57E 0x3F6C0 4. intermediate routers may cache pointers 2. hosting routers participate in ROFL on behalf of hosts ISP Pointer cache: 0x3B57E 0x3BAC8 5. external pointers provide reachability across domains Pointer list: 0x3F6C0 0x3BAC8 Successor list: 0x3F6C0 0x3BAC8 3. hosting routers maintain pointers with source-routes to attached hosts’ successors/fingers 0xFA291 0x3B57E (joining host) 0x3F6C0 1. hosts are assigned topology-independent “flat” identifiers Routing on Flat Labels

Background Ideas Distributed Hash Tables Chord Canon Hashing, Consistent Hashing, DHT goals Chord Join, leave, index fingers, caching Stoica I., et al., “Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications”, IEEE/ACM Transactions on networking, 2003 Canon Support for hierarchy so that we can do BGP-style routing Ganesan, P., et al., “Canon in G Major: Designing DHTs with Hierarchical Structure”, ACM Distributed Computing Systems, 2004. Routing on Flat Labels

DHT Background: Hash Table Name-value pairs (or key-value pairs) E.g., www.cnn.com/foo.html (key) and the Web page (value) Hash table Data structure that associates keys with values lookup(key) key value value Routing on Flat Labels

DHT Background: Hash Functions Hashing Transform the key into a number And use the number to index an array Example hash function Hash(x) = x mod 101, mapping to 0, 1, …, 100 More sophisticated ones: MD5, SHA1, SHA256 Challenges (collisions, joins and leaves) What if there are more than 101 nodes? Fewer? Which nodes correspond to each hash value? What if nodes come and go over time? But ROFL assumes that we’ve got the hashing issue covered. Routing on Flat Labels

DHT Background: Consistent Hashing Large, sparse identifier space (e.g., 128 bits) Hash a set of keys x uniformly to large id space Hash nodes to the id space as well 2128-1 1 Id space represented as a ring. CH def: don’t have to change the table a whole lot when there are joins and leaves. (e.g. max K / N) Hash(name)  object_id Hash(IP_address)  node_id Routing on Flat Labels

DHT (Distributed Hash Table): Hash table spread over many nodes Distributed over a wide area Main design goals Decentralization: no central coordinator Scalability: efficient even with large # of nodes Fault tolerance: tolerate nodes joining/leaving Two key design decisions How do we map names on to nodes? How do we route a request to that node? Routing on Flat Labels

Chord Background: What is Chord? What does it do? Supports just one operation: given a key, it maps the key onto a node In short: a peer-to-peer lookup service Solves problem of locating a data item in a collection of distributed nodes, considering frequent node arrivals and departures Core operation in most p2p systems is efficient location of data items Routing on Flat Labels

Chord Characteristics (O(LogN)) Simplicity, provable correctness, and provable performance Each Chord node needs routing information about only a few other nodes Resolves lookups via messages to other nodes (iteratively or recursively) Maintains routing information as nodes join and leave the system Routing on Flat Labels

Chord Characteristics Simplicity, provable correctness, and provable performance Each Chord node needs routing information about only a few other nodes Resolves lookups via messages to other nodes (iteratively or recursively) Maintains routing information as nodes join and leave the system Routing on Flat Labels

But that doesn’t mean we replace DNS in ROFL DNS vs. Chord DNS provides a host name to IP address mapping relies on a set of special root servers names reflect administrative boundaries is specialized to finding named hosts or services Chord can provide same service: Name = key, value = IP requires no special servers imposes no naming structure can also be used to find data objects that are not tied to certain machines But that doesn’t mean we replace DNS in ROFL Routing on Flat Labels

The Base Chord Protocol Specifies how to find the locations of keys How new nodes join the system How to recover from the failure or planned departure of existing nodes Routing on Flat Labels

Chord: Recall Consistent Hashing Hash function assigns each node and key an m-bit identifier using a base hash function such as SHA-1 ID(node) = hash(IP, Port) ID(key) = hash(key) Practically, we hash the node’s public key to get the hash key Leverage on the benefits of consistent hashing Routing on Flat Labels

Chord Definitions: Successor Nodes Shows which node holds which keys identifier node X key 6 4 2 6 5 1 3 7 1 successor(1) = 1 identifier circle successor(6) = 0 6 2 successor(2) = 3 2 Routing on Flat Labels

Node Joins and Departures 6 6 4 2 6 5 1 3 7 1 successor(6) = 7 successor(1) = 3 2 1 Routing on Flat Labels

Scalable Key Location A very small amount of routing information suffices to implement consistent hashing in a distributed environment Each node need only be aware of its successor node on the circle Queries for a given identifier can be passed around the circle via these successor pointers Resolution scheme correct, BUT inefficient: it may require traversing all N nodes! Routing on Flat Labels

Acceleration of Lookups Lookups are accelerated by maintaining additional routing information Each node maintains a routing table with (at most) m entries (where N=2m) called the finger table ith entry in the table at node n contains the identity of the first node, s, that succeeds n by at least 2i-1 on the identifier circle (clarification on next slide) s = successor(n + 2i-1) (all arithmetic mod 2) s is called the ith finger of node n, denoted by n.finger(i).node Routing on Flat Labels

Finger Tables (of successors) start int. succ. keys 6 1 2 4 [1,2) [2,4) [4,0) 1 3 4 2 6 5 1 3 7 finger table start int. succ. keys 1 2 3 5 [2,3) [3,5) [5,1) finger table start int. succ. keys 2 4 5 7 [4,5) [5,7) [7,3) Routing on Flat Labels

Finger Tables - Node Joins keys start int. succ. 6 1 2 4 [1,2) [2,4) [4,0) 1 3 6 4 2 6 5 1 3 7 finger table start int. succ. keys 1 2 3 5 [2,3) [3,5) [5,1) 6 finger table start int. succ. keys 7 2 [7,0) [0,2) [2,6) 3 finger table keys start int. succ. 2 4 5 7 [4,5) [5,7) [7,3) 6 6 Routing on Flat Labels

Finger Tables - Node Departure keys start int. succ. 1 2 4 [1,2) [2,4) [4,0) 1 3 3 6 4 2 6 5 1 3 7 finger table keys start int. succ. 1 2 3 5 [2,3) [3,5) [5,1) 3 6 finger table keys start int. succ. 6 7 2 [7,0) [0,2) [2,6) 3 finger table keys start int. succ. 2 4 5 7 [4,5) [5,7) [7,3) 6 Routing on Flat Labels

Source of Inconsistencies: Concurrent Operations and Failures Basic “stabilization” protocol is used to keep nodes’ successor pointers up to date, which is sufficient to guarantee correctness of lookups Those successor pointers can then be used to verify the finger table entries Every node runs stabilize periodically to find newly joined nodes Routing on Flat Labels

Stabilization after Join n joins predecessor = nil n acquires ns as successor via some n’ n notifies ns being the new predecessor ns acquires n as its predecessor np runs stabilize np asks ns for its predecessor (now n) np acquires n as its successor np notifies n n will acquire np as its predecessor all predecessor and successor pointers are now correct fingers still need to be fixed, but old fingers will still work ns pred(ns) = n n nil succ(np) = ns pred(ns) = np succ(np) = n np Routing on Flat Labels

Failure Recovery Key step in failure recovery is maintaining correct successor pointers To help achieve this, each node maintains a successor-list of its r nearest successors on the ring If node n notices that its successor has failed, it replaces it with the first live entry in the list stabilize will correct finger table entries and successor-list entries pointing to failed node Performance is sensitive to the frequency of node joins and leaves versus the frequency at which the stabilization protocol is invoked Routing on Flat Labels

Hierarchical DHT: Support for Inter-domain Routing and BGP-ness What is a Hierarchical DHT? USA UCLA MIT CS EE Path Locality Path Convergence Efficiency, Security, Fault isolation Caching, Bandwidth Optimization Local DHTs Access Control Routing on Flat Labels

Crescendo: Convert flat DHTs to hierarchical (Canon-ization) Key idea: Recursive structure Construct bottom-up; merge smaller DHTs Lowest level: Chord CS EE Routing on Flat Labels

Crescendo: Merging two Chord rings 3 2 13 8 3 5 Black node x: Connect to y iff y closer than any other black node y = succ(x + 2^i) 2 8 10 13 12 Routing on Flat Labels

Now Back to the ROFL…. How ROFL works 0x3B57E 0x3F6C0 4. intermediate routers may cache pointers 2. hosting routers participate in ROFL on behalf of hosts ISP Pointer cache: 0x3B57E 0x3BAC8 5. external pointers provide reachability across domains Pointer list: 0x3F6C0 0x3BAC8 Successor list: 0x3F6C0 0x3BAC8 3. hosting routers maintain pointers with source-routes to attached hosts’ successors/fingers 0xFA291 0x3B57E (joining host) 0x3F6C0 1. hosts are assigned topology-independent “flat” identifiers Routing on Flat Labels

Identifiers Identity tied to public/private key pair Everyone can know the public key Only authorized parties know the private key Self-certifying identifier: hash of public key Host associates with a hosting router Proves it knows private key, to prevent spoofing Router joins the ring on the host’s behalf Anycast Multiple nodes have the same identifier Routing on Flat Labels

AS and BGP Policies (A review) Economic relationships Peering Provider/customer Isolation routing contained within hierarchy Routing on Flat Labels

Isolation in ROFL (Canon) Traffic between two hosts traverses no higher than their lowest common provider in the AS hierarchy Accomplished by carefully merging the rings External Successor Destination External Successor Joining host Source Internal Successor Routing on Flat Labels

Policy support in ROFL (BGP support) Goal: prefer peer route over provider route Mechanism: Convert peering relationships to Virtual ASes A Source Destination VirtAS A C B B C Peering link Source Destination Routing on Flat Labels

Scalability in ROFL Two extensions to improve locality: Maintain proximity-based fingers in a policy-safe fashion Pointer caching strategies: prefer nearby, popular pointers 0xF64B pc: 0xA153 pc: 0x3BAC 0xF64B 0xA153 0x3BAC 0x3B57 0xA153 0x3B57 0xF64B 0x3BAC Routing on Flat Labels

Evaluation Intra-domain Inter-domain Trace based on “Rocketfuel” over 4 large ISPs with 2-3 millions of hosts Each host w/ 128-bit ID Routers have 9Mbits of memory for pointer cache and finger tables Inter-domain Using Routeviews, inter-AS topology graph are derived Ran simulations with 30 thousand nodes and extrapolated results to to 600 million hosts Routing on Flat Labels

Metric ROFL BGP+DNS Join overhead Latency State Failure 450 per-host “lightweight joins”: 14 per-host typically 0 per host 40,000 per prefix Latency Baseline: 135ms With pointer caches: 70ms With lookup: 137ms No lookup: 54ms State 2 million pointer cache entries in core, ~100 entries at edge 150 thousand entries at core and edge 77 million DNS entries at root servers Failure Cached pointers equivalent of backup paths, failures only affect successors and fingers Undergoes convergence process, failures have global impact Routing on Flat Labels

Metric ROFL BGP+DNS Join overhead Latency State Failure 450 per-host “lightweight joins”: 14 per-host typically 0 per host 40,000 per prefix Latency Baseline: 135ms With pointer caches: 70ms With lookup: 137ms No lookup: 54ms State 2 million pointer cache entries in core, ~100 entries at edge 150 thousand entries at core and edge 77 million DNS entries at root servers Failure Cached pointers equivalent of backup paths, failures only affect successors and fingers Undergoes convergence process, failures have global impact Routing on Flat Labels

Metric ROFL BGP+DNS Join overhead Latency State Failure 450 per-host “lightweight joins”: 14 per-host typically 0 per host 40,000 per prefix Latency Baseline: 135ms With pointer caches: 70ms With lookup: 137ms No lookup: 54ms State 2 million pointer cache entries in core, ~100 entries at edge 150 thousand entries at core and edge 77 million DNS entries at root servers Failure Cached pointers equivalent of backup paths, failures only affect successors and fingers Undergoes convergence process, failures have global impact Routing on Flat Labels

Metric ROFL BGP+DNS Join overhead Latency State Failure 450 per-host “lightweight joins”: 14 per-host typically 0 per host 40,000 per prefix Latency Baseline: 135ms With pointer caches: 70ms With lookup: 137ms No lookup: 54ms State 2 million pointer cache entries in core, ~100 entries at edge 150 thousand entries at core and edge 77 million DNS entries at root servers Failure Cached pointers equivalent of backup paths, failures only affect successors and fingers Undergoes convergence process, failures have global impact Routing on Flat Labels

Metric ROFL BGP+DNS Join overhead Latency State Failure 450 per-host “lightweight joins”: 14 per-host typically 0 per host 40,000 per prefix Latency Baseline: 135ms With pointer caches: 70ms With lookup: 137ms No lookup: 54ms State 2 million pointer cache entries in core, ~100 entries at edge 150 thousand entries at core and edge 77 million DNS entries at root servers Failure Cached pointers equivalent of backup paths, failures only affect successors and fingers Undergoes convergence process, failures have global impact Routing on Flat Labels

Contributions: Clean-slate design with novel ideas A world without RIR’s to issue IP addresses! The design includes both inter-domain and intra-domain routing issues. Did not side-step essential the features. ROFL is very attractive for new Internet usage patterns such as: multicast, mobility, multi-homing. Secret Sauces: Brings up design issues we will face in a clean-slate approach Put up a good fight to challenge the traditional IP and DNS-based infrastructure Extension of already powerful ideas such as: DHT, Chord, Hierarchical DHT (combined 1000+ citations). Honesty about “conservation of dirt” – didn’t bother to sweep them elsewhere Fundamental computer science ideas: hashing, caching, recursion, doubly linked-lists Routing on Flat Labels

ROFL Weaknesses: Identity tied to public / private key pair PKI notion in Internet: who are the trusted authorities? Unlikely but possible hash collision Identity -> hash translation Per-AS link-state information assumed to be current and precise. Assumes existence of OSPF-like link-state information Accomplished via the “control messages” Needs large pointer cache, but ok with current technology? Inter-AS routing needs large bloom filter for each neighboring AS Issues with quickly updating bloom filters on joins and leaves? Stretch: which is the ratio of the hop count of the selected path to that of the optimal path. Routing on Flat Labels

Special Thanks for the presentation material: References Stoica I., et al., “Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications”, IEEE/ACM Transactions on networking, 2003 Ganesan, P., et al., “Canon in G Major: Designing DHTs with Hierarchical Structure”, Distributed Computing Systems, 2004. Proceedings. Special Thanks for the presentation material: Jennifer Rexford, Princeton Markus Böhning, UC Berkeley Routing on Flat Labels