1 Koorde: A Simple Degree Optimal DHT Frans Kaashoek, David Karger MIT Brought to you by the IRIS project

2 DHT Routing Distributed hash tables Implement hash table interface Map any ID to the machine responsible for that ID (in a consistent fashion) Standard primitive for P2P Machines not all aware of each other Each tracks small set of “neighbors” Route to responsible node via sequence of “hops” to neighbors

3 Performance Measures Degree How many neighbors nodes have Hop count How long to reach any destination node Fault tolerance How many nodes can fail Maintenance overhead E.g., making sure neighbors are up Load balance How evenly keys distribute among nodes

4 Tradeoffs With larger degree, hope to achieve Smaller hop count Better fault tolerance But higher degree implies More routing table state per node Higher maintenance overhead to keep routing tables up to date Load balance “orthogonal issue”

5 Current Systems Chord, Kademlia, Pastry, Tapestry O(log n) degree O(log n) hop count O(log n) ratio load balance Chord: O(1) load balance with O(log n) “virtual nodes” per real node Multiplies degree to O(log 2 n)

6 Outliers CAN Degree d O(dn 1/d ) hops Viceroy O(log n) hop count Constant average degree But some nodes have degree log n

7 Lower Bounds to Shoot For Theorem: if max degree is d, then hop count is at least log d n Proof: < d h nodes at distance h Allows degree O(1) and O(log n) hops Or deg. O(log n) and O(log n / loglog n) hops Theorem: to tolerate half nodes failing, (e.g. net partition) need degree  (log n) Pf: if less, some node loses all neighbors Might as well take O(log n / loglog n) hops!

8 Koorde New routing protocol Shares almost all aspects with Chord But, meets (to within constant factor) all lower bounds just mentioned: Degree 2 and O(log n) hops Or degree log n and O(log n / loglog n) hops and fault tolerant Like Chord, O(log n) load balance or constant with O(log n) times degree

9 Chord Review Chord consists of Consistent hashing to assign IDs to nodes Good load balance Efficient routing protocol to find right node Fast join/leave protocol Few data items shifted Fault tolerance to half of nodes failing Efficient maintenance over time ■ Koorde routing protocol to find right node

10 Consistent Hashing Assign ID to “successor” node on ring 49 Assign doc with hash 49 to node 51

11 Chord Routing Each node keeps successor pointer Also keeps power- of-two “fingers” neighbors providing shortcuts So log n fingers

12 Chord Lookups

13 Koorde Idea Chord acts like a hypercube Fingers flip one bit Degree log n (log n different flips) Diameter log n Koorde uses a deBruijn network Fingers shift in one bit Degree 2 (2 possible bits to shift in) Diameter log n

14 De Bruijn Graph Nodes are b -bit integers ( b = log n ) Node u has 2 neighbors (bit shifts): 2u mod 2 b and 2u+1 mod 2 b

15 De Bruijn Routing Shift in destination bits one by one b hops complete route Route from 000 to 110:

16 Routing Code Procedure u.LOOKUP( k, toShift ) /* u is machine, k is target key toShift is target bits not yet shifted in */ if k  u then Return u /* as owner for k */ else /* do de Bruijn hop */ t = u ° topBit ( toShift ) Return t.lookup(k, toshift  1) Initially call self.LOOKUP( k,k )

17 Summary Each node has 2 outgoing neighbors Also two incoming Can show good routing load balance Need b = log n bits for n distinct nodes So log n hops to route

18 Problems to Solve Want b -bit ring, b >> log n, to avoid colliding identifiers as nodes join Implies use b >> log n hops Worse, most nodes not present to route! Solutions Imaginary routing: present nodes simulate routing actions of absent nodes Short cuts: use gaps to start route with most of destination bits already shifted in

19 Imaginary routing Node u holds two pointers Successor on ring One finger: predecessor of 2u (mod 2 b ) On sparse ring, is also predecessor of 2u+1 So handles both de Bruijn edges Node u “owns” all imaginary nodes between self and (real) successor Simulates de Bruijn routing from those imaginary nodes to others by forwarding to the others’ real owners

20 Code Procedure u.LOOKUP( k, toShift, i ) if k  ( u,u.successor ] then return u.successor /* as bucket for k */ else if i  ( u,u.successor ] then /* i belongs to u; do de Bruijn hop */ return u.finger.LOOKUP ( k, toshift  1, i ° topBit ( toShift )) else /* i doesn’t belong to u; forward it */ return u.successor.LOOKUP ( k, toShift, i ) Initially call self.LOOKUP( k,k,self )

21 True route tracks imaginary start target successor finger (< double) imaginary (double)

22 Correctness Once b de Bruijn steps happen, done At this point, i = k Will follow successors to bucket for k Successor steps delay de Bruijn steps, but not forever After finite number of successor steps, reach predecessor of i Conclude: all necessary de Bruijn steps happen in finite time. So correct.

23 How long? Only b de Bruijn steps Just bound (expected) number of successor steps per de Bruijn step Nodes randomly distributed on ring So node expects to own size 1/n interval So distance to imaginary node on de Bruijn step is 1/n De Bruijn step doubles everything, makes distance 2/n Expect 2 nodes in interval of that size

24 Few Successor Steps start target 1/n < 2/n

25 Summary Each de Bruijn hop followed by 2 successor hops (in expectation) b de Bruijn hops Conclude 2b successor hops so 3b hops in total Expectation argument extends to “with high probability” argument (same bounds) Remaining problem: b>>log n, too big

26 Exploit Address Blocks Only n real nodes Each owns ~ 1/n “block” of keyspace Within that block, only top log n bits “significant”; low bits arbitrary So set low bits to high bits of target Then just have to shift out log n most significant bits So log n de Bruijn hops, So O(log n) hops in total

27 Example Start at u = … Successor …. u “owns” imaginary 00101****** Target …. Set imaginary start … Only need to shift out hops, independent of b

28 Summary Koorde uses 2 neighbors per node (one successor, one finger) And requires O(log n) routing hops with high probability

29 Variant: Koorde- K We used a binary de Bruijn Network Generalizes to other base K:

30 Analysis To represent n distinct node ids need log K n base- K digits Suggests log K n hops to route Same problem as Koorde: b >> log K n Same solution: imaginary routing Node u points at predecessor( Ku ) Same analysis: K de Bruijn hops interspersed with successor hops

31 Successor Hops Now de Bruijn hop multiplies ids by K So expect K nodes between finger and next imaginary node Implies K successor hops per de Bruijn hop Gives K log K n hops---no good To avoid successor hops, u fingers predecessor( Ku ) and following K nodes Allows K successor hops by one finger Gives O(log K n) hops as desired

32 Summary Using K fingers per node, can achieve O(log K n) = O(log n / log K) routing hops As discussed earlier, degree log n is necessary (and sufficient) for fault tolerance (and is degree of most previous systems) So, O(log n / log log n ) hops

33 Summary: What do we Gain? Lower degree for same number of hops Storage isn’t really an issue But lower degree should translate into lower maintenance traffic Lower hop count for same degree And tunable Other systems also have tunable hop count But at low hop counts (high degree) their extra log factor in degree does matter

34 What do we lose? Chord is “self stabilizing” From successors, can build entire routing system quickly by “pointer jumping” to find fingers Koorde is not Given only successor pointers, no clear fast way to find fingers Not a problem for joins, because joiner can use lookup to find its finger But could be a problem if massive changes

35 More Info