Large Scale Sharing Marco F. Duarte COMP 520: Distributed Systems September 19, 2004

Introduction P2P sharing systems are very popular In P2P, all nodes have identical capabilities and responsibilities Popular approaches are partially centralized, do not scale well, or do not provide desired anonymity Scalability of systems critical Need for decentralized, load-balancing architectures

Features desired in a P2P sharing system Decentralized architecture – no single point of failure Scalability – bandwidth and load balancing Fault tolerance – content replication Anonymity for users – posters, readers, storers Resilient against DoS attacks

Freenet provides anonymity No requester, provider information implicit in communication Presence of a file in a node does not imply authorship Popular files are replicated to improve locality Does not intend to provide permanent storage

Freenet Queries Files receive FileIDs (160- bit SHA-1 hash of “file identifier”) Queries have pseudo- unique random identifiers (QueryIDs) and hops-to-live count. Routing tables contain table of previously retrieved FileIDs and their locations Queries are routed to location with closest FileID at each stage; loops are detected with QueryID FileIDNode Address ?

Freenet Queries: Lookups and Stores Copies of the file are stored at all nodes File record for a is added to routing tables Writes perform lookup, insert file along path if no match found a e b

Freenet Properties FileID-based clustering allows for improved routing as usage increases LRU-like capacity management: rarely used files are purged from the system Random nature of FileIDs allow for diversity of information at nodes Attempts to supplant existing files will lead to real file propagation Anonymity features: File ownership assumed randomly by other nodes Minimal routing information necessary at each hop Hops-to-live count of 1 updated randomly

Freenet Problems Files that are stored in the network may not be found. Freenet does not provide reliable storage No notion of locality in routing Simulations do not involve file insertion or node discovery

PAST: Reliable Distributed Storage Customizable file persistence High availability and load balancing Efficient Routing and Storage Allocation Uses FileIDs generated from hashes like in Freenet Uses owner credentials to verify identity of authors Interface: Insert, Lookup, Reclaim

PAST Architecture FileID computed from hash of filename, owner’s public key and a random salt. Each node receives a pseudorandom NodeID, independent of the node properties. Owner specifies number k of replicas of a file to store in the system on insert. File is stored in the k nodes with NodeIDs closest to the FileID. Routing provided by Pastry.

Pastry: Routing for P2P Networks Paths with less than hops Delivery guaranteed under at most node failures Flexible proximity metric. Each node contains: Leaf set – l nodes with closest NodeIDs Routing table – set of neighbors organized by NodeIDs Neighborhood set – l closest nodes Each NodeID is paired with its network address Direct routes to neighbors and l closest NodeIDs

Pastry: Example Routing table organized by similarity to NodeID. Neighborhood set used for node addition/recovery. Queries are forwarded to a numerically closer node (by shared NodeID header, and NodeID proximity).

Pastry Routing Table 0=2 M Leaf Set Neighborhood Set

Pastry Routing Example 0=2 M ? Other nodes exist but are not shown

Pastry Node Insertion Example 0=2 M Neighborhood Set Leaf Set

Pastry Node Removal Example 0=2 M

PAST Insertions 0=2 M Insert File, FileID 3130 Owner 3130: File, Certificate 3130: File, Certificate 3130: File, Certificate fileID = Insert(name, owner-credentials, k, file) Insert File K times

PAST Insertions 0=2 M Owner k Store Receipts k Store Receipts k Store Receipts fileID = Insert(name, owner-credentials, k, file)

PAST Semantics fileID = lookup(fileID) Routed to NodeID = FileID First of k closest nodes found returns file, credentials Reclaim(fileID, owner-credentials) Same semantics as Insert Owner issues Reclaim Certificate Storing nodes issue Reclaim Receipt Changes in leaf sets will trigger changes in replica locations A new node creates “pointers” to files it should contain; migration is gradual

Load Balancing in PAST: Replica Diversion 3130 Leaf Set 3201Leaf Set

Load Balancing in PAST: File Diversion 3130 Leaf Set 3201Leaf Set Change ID by changing salt Policies for acceptance of replicas and diverted replicas, and selection of diverted replica node. Maximum ratio of file size to free space for insertion t pri, t div

Caching in PAST Highly popular files might demand more replicas than specified. Files located “far away” only need to be fetched once locally Unused disk space is allocated as cache. Caching performance degrades gradually with increased utilization Cache insertion policy similar to diversion policies.

PAST Performance: t pri comparison, t div =0.05

PAST Performance: Ratio of File Diversions

PAST Performance: Ratio of Replica Diversions

PAST Performance: Failed Insertions

PAST Performance: Cache Hits

Conclusions Content based routing improves scalability of distributed storage systems. Need for user authentication in distributed systems. Caching is crucial for system performance. Diversion allows for graceful performance degradation. Need file mutability, file search or indexing services