1. Distributed Hash Table Systems Hui Zhang University of Southern California

2. CS551 - DHT2 Outline History A brief description of DHT systems  General definition  Basic components Chord, a DHT system. Proximity routing in DHTs Reading list for interested audience

3. CS551 - DHT3 History: client-server model serverindex tabledata client aclient bclient cclient d query data transferring

4. CS551 - DHT4 History: peer-to-peer model (Napster) server index table dataclient a client b client c client ddata query data transferring

5. CS551 - DHT5 History: peer-to-peer model (Gnutella) query data transferring index table dataclient a client b client c client d index table data index table data index table data

6. CS551 - DHT6 History: peer-to-peer model (DHT systems) query data transferring index table dataclient a client b client c client d index table data index table data index table data

7. CS551 - DHT7 a new class of peer-to-peer routing infrastructures - CAN, Chord, Pastry, Tapestry, Viceroy, Kademlia, etc. support a hash table-like functionality on Internet-like scale - given a key, map it onto a node numerous applications, services, and infrastructures are proposed to be built on top of a DHT system. - web caching, - naming service, - event notification, - indirection services, - DoS attack prevention, - application-layer multicast, - etc. DHT systems - definition

8. CS551 - DHT8 Basic DHT components an overlay space and its partition approach a routing protocol  local routing table  next-hop decision a base hash function  variants: proximity-aware, locality-preserving, etc.

9. CS551 - DHT9 Consistent hashing [Karger et al. STOC97] data server Overlay space Hashing

10. CS551 - DHT10 Overlay space - one-dimensional unidirectional key space 0 – 2 m -1.  Given two m-bit identifiers x and y, d(x, y)=(y-x+ 2 m ) % 2 m - Each node is assigned a random m-bit identifier. - Each node is responsible for all keys equal to or before its identifier until its predecessor node’s identifier in the key space. Chord [Stoica et al. Sigcomm2001]

11. CS551 - DHT11 A Chord network with 8 nodes and 8-bit key space 0 32 64 96 128 160 192 224 256 Network node 0 256 Data 120 Chord – an example (m=8)

12. CS551 - DHT12 Routing table (finger table) - (at most) m entries. The i th entry of node n contains the pointer to the first node that succeeds n by at least 2 (i-1) on the key space, 1  i  m. Next-hop decision: - For the given target key k, find the closest finger before (to) k and forward the request to it. - Ending condition: The request terminates when k lies between the ID range of current node and its successor node. - The routing path length is O(log n) for a n-nodes network with high probability (w.h.p.). Chord – routing protocol

13. CS551 - DHT13 Chord – routing table setup A Chord network with 8 nodes and m(=8)-bit key space Network node 255 0 64 128 32 192 96 160224 [1,2) Range 1 [2,4) Range 2 [4,8) Range 3 [8,16) Range 4 [16,32) Range 5 [32,64) Range 6 [64,128) Range 7 [128,256) Range 8 Data Pointer

14. CS551 - DHT14 Chord – a lookup for key 120 Network node 0 64 128 32 192 96 160224 120 Data 255 Overlay routing physical link A Chord network with N(=8) nodes and m(=8)-bit key space

15. CS551 - DHT15 2.Updating fingers of existing nodes Chord – node joining 3.Transferring keys Between the new node and its successor node. 1.Initializing fingers

16. CS551 - DHT16 Chord – stabilization mechanism Refresh the pointer to the immediate successor up to date. - guarantee correctness of lookups Refresh the pointers to the rest (m-1) fingers up to date. - keep lookups fast.

17. CS551 - DHT17 Chord – successors-list mechanism

18. CS551 - DHT18 Chord – simulation result [Stoica et al. Sigcomm2001]

19. CS551 - DHT19 The fraction of lookups that fail as a function of the fraction of nodes that fail. [Stoica et al. Sigcomm2001] Chord – “failure” experiment

20. CS551 - DHT20 Outline History A brief description of DHT systems  General definition  Basic components Chord, a DHT system. Proximity routing in DHTs Reading list for interested audience

21. CS551 - DHT21 Overlay performance of Chord Expected routing table size per node - O(log N) entries Expected hops per request - O(log N) hops

22. CS551 - DHT22 Latency stretch in Chord Network node 0 64 128 32 192 96 160224 120 Data 255 Overlay routing physical link A Chord network with N(=8) nodes and m(=8)-bit key space 64 U.S.AChina 0 128 96 80

23. CS551 - DHT23 Latency stretch [Ratnasamy et al. 2001] = latency for each lookup on the overlay topology average latency on the underlying topology In Chord,  (logN) hops per lookup in average   (logN) stretch in original Chord.  Could Chord do better, e.g., O(1) stretch, without much change?

24. CS551 - DHT24 Three categories of proximity routing [Gummadi2003] Proximity identifier selection  By carefully assign IDs to the nodes, make the overlay topology congruent to the underlying network topology. Proximity route selection  When forwarding a request message, the choice of the next hop depends on both the overlay distance and the underlying distance. Proximity neighbor selection  The neighbors in the routing table are chosen based on their proximity.

25. CS551 - DHT25 Underlying network topology decides the performance of proximity routing Chord node The worse-case scenario: equal-distance Physical link Router

26. CS551 - DHT26 Latency expansion Let N u (x) denote the number of nodes in the network G that are within latency x of node u. - power-law latency expansion: N u (x) grows (i.e. ``expands'‘) proportionally to x d, for all nodes u.  Examples: ring (d=1), mesh (d=2). - exponential latency expansion: N u (x) grows proportionally to  x for some constant  > 1.  Examples: random graphs.

27. CS551 - DHT27 “Latency-stretch” theorem - I “Bad news” Theorem If the underlying topology G is drawn from a family of graphs with exponential latency expansion, then the expected latency of Chord is  (LlogN), where L is the expected latency between pairs of nodes in G. Chord node The worse-case scenario: equal-distance Physical link Router

28. CS551 - DHT28 “Latency-stretch” theorem - II “Good news” Theorem If (1) the underlying topology G is drawn from a family of graphs with d-power-law latency expansion, and (2) for each node u in the Chord network, it samples (log N) d nodes in each range with uniform randomness and keeps the pointer to the nearest node for future routing, then the expected latency of a request is O(L), where L is the expected latency between pairs of nodes in G. then the latency stretch of Chord is O(1).

29. CS551 - DHT29 Two remaining questions How does each node efficiently achieve (log N) d samples from each range? Do real networks have power-law latency expansion characteristic?

30. CS551 - DHT30 Uniform sampling in terms of ranges For a routing request with  (log N) hops, final node t will be a random node in  (log N) different ranges Node x: the node at hop x Node 0: the request initiator Node t: the request terminator routing path Node 0 node t is in its range-d Node 1 node t is in its range-x (< d) Node 2 node t is in its range-y (< x) Node t

31. CS551 - DHT31 Lookup-Parasitic Random Sampling 1.Recursive lookup. 2.Each intermediate hop appends its IP address to the lookup message. 3.When the lookup reaches its target, the target informs each listed hop of its identity. 4.Each intermediate hop then sends one (or a small number) of pings to get a reasonable estimate of the latency to the target, and update its routing table accordingly.

32. CS551 - DHT32 LPRS-Chord: convergence time Convergence Time

33. CS551 - DHT33 LPRS-Chord: topology with power-law expansion Ring Stretch (at time 2logN)

34. CS551 - DHT34 One remaining question Do real networks have power-law latency expansion characteristic?

35. CS551 - DHT35 Internet router-level topology: latency measurement Approximate link latency by geographical latency - assign geo-locations to nodes using Geotrack [Padmanabhan2001]. A large router-level topology dataset - 320,735 nodes, mapped to 603 distinct cities all over the world. - 92,824 node pairs are sampled to tractably compute the latency expansion of this large topology.

36. CS551 - DHT36 Internet router-level topology: latency expansion

37. CS551 - DHT37 LPRS-Chord on router-level topology Stretch on the router-level subgraphs (at time 2logN)

38. CS551 - DHT38 Outline History A brief description of DHT systems  General definition  Basic components Chord, a DHT system. Proximity routing in DHTs Reading list for interested audience

39. CS551 - DHT39 A few interesting papers on DHT K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy, S. Shenker, I. Stoica. “The Impact of DHT Routing Geometry on Resilience and Proximity.” Sigcomm 2003. A. Gupta, B. Liskov, and R. Rodrigues. “Efficient Routing for Peer-to-Peer Overlays.” NSDI 2004. H. Balakrishnan, K. Lakshminarayanan, S. Ratnasamy, S. Shenker, I. Stoica, M. Walfish. “A Layered Naming Architecture for the Internet.” Sigcomm 2004.

40. CS551 - DHT40 Thank you!

41. CS551 - DHT41 Backup slides

42. CS551 - DHT42 A simple random sampling solution Network node 0 A Chord network with m-bit key space 2 m -1 Pointer 2 m-1 2 m-2 Distance measurement

43. CS551 - DHT43 A simple random sampling solution Network node 0 A Chord network with m-bit key space 2 m -1 Pointer 2 m-1 2 m-2 Distance measurement

44. CS551 - DHT44 Term definition (II) Range - for a given node in a Chord overlay with ID j, its i-th range R i (j) is the interval [j+2 i-1, j+2 i ) on the key space, where 1  i  m. Frugal routing 1. after each hop, the search space for the target reduces by a constant factor, and 2. If w is an intermediate node in the route, v is the destination, and v  R i (w), then the node after w in the route depends only on w and i.

45. CS551 - DHT45 LPRS-Chord: simulation methodology Phase 1. N nodes join the network one-by-one. Phase 2. each node on average inserts four documents into the network. Phase 3. each node generates, on average 3logN data requests one-by-one. - LPRS actions are enabled only in Phase 3 - Performance measurement begins at Phase 3

46. CS551 - DHT46 Comparison of 5 sampling strategies: definitions Consider a lookup that is initiated by node x 0, then forwarded to node x 1, x 2,..., and finally reaches the request terminator, node x n : 1. Node x i samples node x n, 0  i < n 2. Node x n samples nodes x 0, …, x n-1 3. Node x i samples node x i-1, 1  i  n 4. Node x 0 samples nodes x n 5. Node x i samples node x 0, 0 < i  n

47. CS551 - DHT47 Comparison of 5 sampling strategies: simulation result

48. CS551 - DHT48 Zipf-ian document popularity

49. CS551 - DHT49 Impact of skewed request distributions