A Scalable Content-Addressable Network Authors: S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Shenker University of California, Berkeley Presenter: Andrzej Kochut

Main features of CAN Distributed, Internet-scale hash table No form of centralized control Scalable – amount of information stored at each node independent of the total number of nodes in the system Does not impose any form of hierarchical name structure Can be implemented at the application level

Basic operations of CAN Inserting, updating, deleting of (key,value) pairs Retrieving value associated with a given key Adding new nodes to CAN Handling departing nodes Dealing with node failures

Basic CAN design

Design of CAN Virtual d-dimensional coordinate space on a d-torus At any instant the entire coordinate space is partitioned among all the nodes in the system Each node contains chunk of the hash table and information about its neighbors in the d-space Uniform hash function is used to map key values to points in the d-dimensional space

Example 2-space partition

Design of CAN To store (K,V) pair the hash value P of K is computed and the pair is stored in the node that owns point P To retrieve the value given K, the hash value P of K is computed. If the requesting node does not own point P, the request is routed towards the node that owns P The set of immediate neighbors serves as a coordinate routing table that enables routing between arbitrary points in the d-space

Routing in CAN Intuitively – following the straight path through the Cartesian space from source to destination Node maintains coordinate routing table that holds IP addresses and zones’ coordinate of its neighbors in the space Two nodes are neighbors if their coordinate spans overlap along d-1 dimensions and abut along one dimension CAN message contains destination coordinate. Node greedy forwards it to the neighbor with coordinates closest to the destination coordinate

Routing in CAN For the d-dimensional space equally partitioned into n nodes the average routing path is (d/4)*n (1/d) Individual nodes maintain 2d neighbors The path length growth proportionally to the O(n (1/d) ) Many different routes between two points

CAN construction New node is allocated its own portion of the coordinate space in three steps: –Find a node already in the CAN – look up the CAN domain name in DNS –Pick zone to join to and route request to its owner using CAN mechanisms –Split the zone between old and new node –The neighbors of the split zone must be notified so the routing can include the new node

Finding and splitting a zone Randomly choose a point P in the space Send join request destined for P Route the request using CAN mechanisms Split the zone occupied by the owner of P assuming certain ordering of the dimensions, i.e. first X then Y Transfer (key, value) pairs from the half of the zone to the new node

Joining the routing Update neighbor information in the old and new node Inform all neighbors about changes in the zone – every node in the system sends immediate update message followed by periodic refreshes

Node departure and recovery Normal procedure – explicit hand over of (key,value) database to one of the neighbors Node failure – immediate takeover procedure: –Failure detected as a lack of update messages –Each neighbor starts timer with proportion to the node’s zone size –After timer expires the node extends its own zone to contain the failed neighbor’s zone and sends TAKEOVER message to all failed node’s neighbors –On receive of the TAKEOVER node cancels its timer if the sender’s zone size is smaller than his own. Otherwise it sends it’s own TAKEOVER message.

Recovery cont. CAN state may become inconsistent if multiple adjacent nodes fail simultaneously In such cases perform an expanding ring search for any nodes residing beyond the failure region (?) If it fails initiate repair mechanism (?) If a node holds more than one zone initiate the background zone-reassignment algorithm

Drawbacks of the basic CAN design Not fault tolerant No recovery mechanism Lack of load-balancing mechanisms (no caching and replication) Does not consider the underlying IP topology while building overly network

Design improvements

Realities: multiple coordinate spaces Topologically-sensitive construction of the CAN Multiple hash functions Better routing metrics Overloading coordinate zones More uniform partitioning Caching and replication

Multiple coordinate spaces Each node is assigned a different zone in each reality (coordinate space) Improved robustness - point P is unreachable only if P in all realities is unreachable Improved routing – to forward a message, a node checks all its neighbors on each reality and forwards the message to the neighbor with coordinates closest to the destination Increased per-node state

Topologically-sensitive construction of the CAN Assumes existence of a well known set of machines (i.e. DNS root name servers) called landmarks Each node that joins the CAN adds itself to the region associated with its perceived ordering of landmarks Reduces latency stretch (ratio of the CAN latency to the IP latency) Leads to unbalanced partition of space between nodes

Multiple hash functions Many different hash functions to map (key, value) pairs to the points in space Data replicated accordingly Improved fault tolerance Increase in the size of (key,value) database and in the size of the query traffic Data consistency issues

Effects of the improvements

Effects of the improvements (cont.)

Better routing metrics Each node measures the network-level RTT to each of its neighbors Message forwarded to the neighbor with the highest progress to RTT ratio

Overloading coordinate zones More than one node associated with each zone Each node maintains list of its peers in the zone and neighboring information about selected node in each of its neighboring zones Adding new nodes – if the zone to which a new node joins contains less than MAXPEERS nodes, the new node joins the zone without any space- splitting Neighboring nodes in the adjacent zones are chosen based on the RTT

Advantages of the zone overloading Reduced path length Reduced per-hop latency – node has multiple choices in neighbors selection Improved fault-tolerance

More uniform partitioning Attempt to have equal partitioning of the space between nodes Existing occupant of the zone that a new node joins redirects the request to the zone with the highest volume among his and all of his neighbors Not true load balancing – does not reflect popularity of the data

Caching and replication Each node maintains cache of the most popular requests A node that is being overloaded by requests to the particular data key replicates this key to all of its neighbors No description of the way to check validity of the cached and replicated data

Performance tests System size – 2 18 nodes Topology generated by GT modeler Results only from simulations

Performance test results

Conclusion and presenter’s opinion Addresses the problems of scalable routing and indexing Test results only from simulation Unresolved security issues (i.e. denial of service attacks) Lack of search techniques (i.e. keyword searching) Not clearly specified recovery techniques Not clearly specified data consistency issues (replication, caching) Does not address the problem of node’s connection quality