Wide-Area Cooperative Storage with CFS Robert Morris Frank Dabek, M. Frans Kaashoek, David Karger, Ion Stoica MIT and Berkeley

Target CFS Uses Serving data with inexpensive hosts: open-source distributions off-site backups tech report archive efficient sharing of music node Internet node

How to mirror open-source distributions? Multiple independent distributions Each has high peak load, low average Individual servers are wasteful Solution: aggregate Option 1: single powerful server Option 2: distributed service But how do you find the data?

Design Challenges Avoid hot spots Spread storage burden evenly Tolerate unreliable participants Fetch speed comparable to whole-file TCP Avoid O(#participants) algorithms Centralized mechanisms [Napster], broadcasts [Gnutella] CFS solves these challenges

The Rest of the Talk Software structure Chord distributed hashing DHash block management Evaluation Design focus: simplicity, proven properties

CFS Architecture Each node is a client and a server (like xFS) Clients can support different interfaces File system interface Music key-word search (like Napster and Gnutella) node clientserver node clientserver Internet

Client-server interface Files have unique names Files are read-only (single writer, many readers) Publishers split files into blocks Clients check files for authenticity [SFSRO] FS Clientserver Insert file f Lookup file f Insert block Lookup block node server node

Server Structure DHash stores, balances, replicates, caches blocks DHash uses Chord [SIGCOMM 2001] to locate blocks DHash Chord Node 1Node 2 DHash Chord

Chord Hashes a Block ID to its Successor N32 N10 N100 N80 N60 Circular ID Space Nodes and blocks have randomly distributed IDs Successor: node with next highest ID B33, B40, B52 B11, B30 B112, B120, …, B10 B65, B70 B100 Block ID Node ID

Basic Lookup N32 N10 N5 N20 N110 N99 N80 N60 N40 “Where is block 70?” “N80” Lookups find the ID’s predecessor Correct if successors are correct

Successor Lists Ensure Robust Lookup N32 N10 N5 N20 N110 N99 N80 N60 Each node stores r successors, r = 2 log N Lookup can skip over dead nodes to find blocks N40 10, 20, 32 20, 32, 40 32, 40, 60 40, 60, 80 60, 80, 99 80, 99, , 110, 5 110, 5, 10 5, 10, 20

Chord Finger Table Allows O(log N) Lookups N80 ½ ¼ 1/8 1/16 1/32 1/64 1/128 See [SIGCOMM 2000] for table maintenance

DHash/Chord Interface lookup() returns list with node IDs closer in ID space to block ID Sorted, closest first server DHash Chord Lookup(blockID)List of finger table with

DHash Uses Other Nodes to Locate Blocks N40 N10 N5 N20 N110 N99 N80 N50 N60 N68 Lookup(BlockID=45)

Storing Blocks Long-term blocks are stored for a fixed time Publishers need to refresh periodically Cache uses LRU disk: cacheLong-term block storage

Replicate blocks at r successors N40 N10 N5 N20 N110 N99 N80 N60 N50 Block 17 N68 Node IDs are SHA-1 of IP Address Ensures independent replica failure

Lookups find replicas N40 N10 N5 N20 N110 N99 N80 N60 N50 Block 17 N Lookup(BlockID=17) RPCs: 1. Lookup step 2. Get successor list 3. Failed block fetch 4. Block fetch

First Live Successor Manages Replicas N40 N10 N5 N20 N110 N99 N80 N60 N50 Block 17 N68 Copy of 17 Node can locally determine that it is the first live successor

DHash Copies to Caches Along Lookup Path N40 N10 N5 N20 N110 N99 N80 N60 Lookup(BlockID=45) N50 N RPCs: 1. Chord lookup 2. Chord lookup 3. Block fetch 4. Send to cache

Caching at Fingers Limits Load N32 Only O(log N) nodes have fingers pointing to N32 This limits the single-block load on N32

Virtual Nodes Allow Heterogeneity Hosts may differ in disk/net capacity Hosts may advertise multiple IDs Chosen as SHA-1(IP Address, index) Each ID represents a “virtual node” Host load proportional to # v.n.’s Manually controlled Node A N60N10N101 Node B N5

Fingers Allow Choice of Paths N80 N48 100ms 10ms Each node monitors RTTs to its own fingers Tradeoff: ID-space progress vs delay N25 N90 N96 N18N115 N70 N37 N55 50ms 12ms Lookup(47)

Why Blocks Instead of Files? Cost: one lookup per block Can tailor cost by choosing good block size Benefit: load balance is simple For large files Storage cost of large files is spread out Popular files are served in parallel

CFS Project Status Working prototype software Some abuse prevention mechanisms SFSRO file system client Guarantees authenticity of files, updates, etc. Napster-like interface in the works Decentralized indexing system Some measurements on RON testbed Simulation results to test scalability

Experimental Setup (12 nodes) One virtual node per host 8Kbyte blocks RPCs use UDP To vu.nl lulea.se ucl.uk To kaist.kr,.ve Caching turned off Proximity routing turned off

CFS Fetch Time for 1MB File Average over the 12 hosts No replication, no caching; 8 KByte blocks Fetch Time (Seconds) Prefetch Window (KBytes)

Distribution of Fetch Times for 1MB Fraction of Fetches Time (Seconds) 8 Kbyte Prefetch 24 Kbyte Prefetch40 Kbyte Prefetch

CFS Fetch Time vs. Whole File TCP Fraction of Fetches Time (Seconds) 40 Kbyte Prefetch Whole File TCP

Robustness vs. Failures Failed Lookups (Fraction) Failed Nodes (Fraction) (1/2) 6 is Six replicas per block;

Future work Test load balancing with real workloads Deal better with malicious nodes Proximity heuristics Indexing Other applications

Related Work SFSRO Freenet Napster Gnutella PAST CAN OceanStore

CFS Summary CFS provides peer-to-peer r/o storage Structure: DHash and Chord It is efficient, robust, and load-balanced It uses block-level distribution The prototype is as fast as whole-file TCP