A Scalable Content-Addressable Network University of Athens Department of Informatics and Telecommunications Postgraduate Program Course: Distributed Systems Supervisor: Professor Alex Delis A Scalable Content-Addressable Network In Proceedings of ACM SIGCOMM 2001 S. Ratnasamy, P. Francis, M. Handley, R. Karp, S. Shenker Zacharias Boufidis boufidis@di.uoa.gr June 3rd, 2002

Scope CAN: Content-Addressable Network: A distributed infrastructure for providing hash table-like functionality on Internet-like scales A distributed indexing scheme for mapping file names to location Content-Addressable Network: storing content at a “point” / routing between points Department of Informatics and Telecommunications - University of Athens

Outline Motivation / Objectives CAN Design: insertion, retrieval, deletion Design Optimizations Performance Evaluation Related Work Discussion Department of Informatics and Telecommunications - University of Athens

Motivation - Applicability P2P file sharing systems - challenge: huge amount of resources - drawback: centralized indexing scheme (Napster: single point of failure/control, Gnutella: flooding => scalabilitty) Large-scale storage management systems (e.g OceanStore) - Efficient insertion and retrieval of content Location-independent naming schemes (vs. DNS) - Naming scheme decoupled from the name resolution process Application-layer multicasting - No need for distribution trees - Scalability (group sizes larger than 1000 nodes, no single source service model)

Objectives Indexing mechanism: totally distributed scalable (per-node storage and communication costs grow slowly as the network size grows) fault-tolerant self-organizing query speed (low-latency - efficient routing) locality (queries for nearby nodes stay local) guarantees for content retrieval Department of Informatics and Telecommunications - University of Athens

Design Overview Basic idea: a virtual d-dimensional Cartesian coordinate space dynamically partioned => zone per node Data stored as <key,value> pairs Insertion: Key k uniform hash function point P in a zone Retrieval: Key k uniform hash function point P in a zone or routing the request until it reaches the node in which zone P lies

Assumptions Associated DNS domain name - external mechanism upon joining A priori knowledge of the deterministic hash function Department of Informatics and Telecommunications - University of Athens

Design - CAN Routing Routing table: IP address & virtual coordinate zone of neighbours (coordinate zones adjoining) Greedy forwarding (recursively) of msg[P(k), dst_coordinates] to neighbour with coordinates closest to P Department of Informatics and Telecommunications - University of Athens

Design - CAN Construction (1) Node Insertion (of N1) 1. N1: find_anyCANnode(N2) 2. N1: choose_point(rand, P) 3. Route to N3 owning P 4. N1: JOIN request to N3 5. N3: zone_splitting(), send(<key,value> from half_zone) to N1, send(neighbout_info (IPaddr)) to N1, inform_neighrbours Pros: only neighbours affected => scalability

Design - CAN Construction (2) Node departure 1. Explicit handover of a zone and <k,v> pairs => possible merge of zones 2. Node failure: Robustness: immediate takeover based on metrics (smallest current zone, least-loaded node, connectivity, etc.) (-) loss of <k,v> pairs solution: periodic update messages Expanding ring seach first to avoid inconsistencies Background zone-reassignment => 1 zone/node

Design Considerations Low per-node state [O(d), independent of n] vs. short path lengths [O(d (n^1/d)) hops] d: dimensions, n: nodes Design Goal: lookup latency = (Avg. # CAN hops) x (Avg. per hop latency) comparable to IP latency => techniques for reducing the CAN routing latency => (+) robustness (routing, data availability) (-) per-node state (-) system complexity

Design Improvements (1) Multi-dimensioned coordinate spaces - More dimensions => more neighbours per node Goal: reduction of routing path length (# hops) => reduction of path latency (+) routing fault tolerance (-) more state per node (routing table) Multiple realities - More independent coordinate spaces => allocation of multiple different zones per node (replication of hash tables on every reality) => selection of min CAN hop route => reduction of Avg. path length =>path latency (+) data availability (+) enhanced routing fault tolerance (-) more state per node The first solution outperforms in terms of routing efficiency

Design Improvements (2) Refinement of CAN routing metrics RTT-weighted routing: - Goal: reduction of per-hop latency metric: progress(Cartesian distance) / RTT - Simulation results: 24-40% improvement Department of Informatics and Telecommunications - University of Athens

Design Improvements (3) Overloading coordinate zones - When joining, zone sharing (if possible) instead of splitting - State info: neighbour list + peer list Selection of a peer’s neighbour based on measured lowest RTT - Hash tables: replication vs. Partitioning tradeoffs: availability, size of data stores, consistency (+) Reduced # hops (+) Reduced per hop latency (+) Improved fault tolerance (-) System complexity (-) Additional control traffic reduced path length - Simulation results: 45% improvement of per-hop latency

Design Improvements (4) Multiple hash functions - Mapping a single key to multiple nodes (replication) => parallel queries (+) Reduced query latency (+) Increased data availability (-) Increased size of the <key, value> database (-) Increased query traffic Topologically-sensitive CAN construction - Node insertion based on RTT from landmarks (instead of random insertion) metric: latency stretch = CAN latency / network level latency (+) Reduced path latency (-) Uneven load distribution => need for load balancing

Design Improvements (5) Uniform partitioning - Volume-based zone splitting (+) Achieves some form of load balancing (-) «Hot spot» problem: some <key, value> pairs are more popular Caching & Replication for «hot spot» management - Caching of recently accessed keys (belonging to other nodes) - Replication: active pushing out of popular keys to neighbours within a region (+) Increased data availability (+) Reduced query latency (+) Load balancing Department of Informatics and Telecommunications - University of Athens

Performance Evaluation (1) Critical factors: 1. increase in # of dimensions d => reduction of path length 2. Use of RTT-weighted routing => optimization of next-hop forwarding => reduction of path latency Department of Informatics and Telecommunications - University of Athens

Performance Evaluation (2) Effect of link delay distribution on latency stretch: Remarks: 1. Increase in # of nodes => slow increase of latency stretch (i.e., slow increase of total path latency) 2. Random delay: the largest latency stretch 3. Larger backbone => reduced density of CAN nodes => less effect of RTT-weighted routing => degraded gains Department of Informatics and Telecommunications - University of Athens

Related Work LS and DV routing Hierarchical routing (BGP) (-) Widespread, frequent (in case of topology changes) dissemination of topology info Hierarchical routing (BGP) (-) Fault tolerance CAN: Self-configured routing Plaxton’s Algorithm (~prefix-based routing) - Similar routing hops & routing table size (-) Neighbour discovery (global knowledge of topology) Geographic routing (-) Geographic forwarding needs real-world location service - difficulty in implementing neighbour relationships (no radio coverage feature available in CANs)

Conclusion Content-Addressable Networks - Added Value: Scalable routing and efficient indexing Completely distributed, self-organizing, fault tolerant Department of Informatics and Telecommunications - University of Athens

Discussion (1) Paper Evaluation Plus: - Novelty - Comparison to best rival scheme & other competing indexing schemes Minus: - Limited evaluation criteria - Drawbacks of the proposed solution? -“Chalk and Cheese”: indexing techniques & routing algorithms Department of Informatics and Telecommunications - University of Athens

Discussion (2) Evaluation of indexing technique - Open issues - Index length? - Index construction time? (Convergence speed) - Index construction = f(size of data store)? - Index update cost? - Extensibility? (types of queries supported, apart from keyword search) - Memory/processing requirements? - Effect of caches compared to “cold start”? - Stability (concurrent joins) &Consistencies: a) intermediate consistencies b) simultaneous updates? - Ease of implementation?

Discussion (3) Evaluation of routing algorithm - Open issues - Routing loops? - Control of routing paths? (recursive routing) - Routing of query responses? Department of Informatics and Telecommunications - University of Athens

Discussion (4) Other Issues - Security (DoS attacks, AAA) - Parameter tuning needed to achieve scalability (Cannot vary d as n increases - n not known by any node) - CAN maintenance protocol overhead? (Cost of update operation) - Accommodation of administrative boundaries? (handling of key value pairs?) - Initial knowledge of the deterministic hash function? Ways to be changed dynamically? Implications? (total reconstruction of the CAN?) - Specification of inter-update times, caching TTL values, etc.

Discussion (5) Enhancements - Initialization of alternate paths from intermediate nodes - Employment of hierarchical model or islands of CANs: - The uniform manipulation fo CAN nodes does not represent the state of the Internet - even P2P systems need not be purel P2P - some degree of loose organization might be required - Dynamic parameter tuning to achieve scalability & efficient routing - Landmark hierarchal routing or GLS spatial hierarchy? - Stronger coupling of routing algorithms to underlying topology and node capability (some knowledge about the network could really help) - Derivation of appropriate network-layer models - QoS-routing based on CANs (idea: hashing IPaddr & ToS field, use one reality per QoS metric, etc.)

Thank You! Department of Informatics and Telecommunications - University of Athens