1. A Peer-to-Peer File System OSCAR LAB

2. Overview A short introduction to peer-to-peer (P2P) Systems Ivy: a read/write P2P file system (OSDI’02)

3. What is P2P ? An architecture of equals (as opposed to client/server), each peer/node acts as – Client – Server – Router Harness aggregate resources (e.g., CPUs, memory, disk capacities) among peers/nodes

4. What is P2P ? Technical trends Creation of huge pool of available latent resources Increasing processing power of PCs Decreasing cost and increasing capacity of disk space Widespread penetration of broadband

5. P2P Systems Centralized: have a centralized directory service – E.g., Napster – Limits scalability and poses a single point of failure Decentralized and Untructured – No precise control over the network topology or data placement – E.g., Gnutella – Controlled message flooding, limiting scalability

6. P2P Systems Decentralized and Structured – Tightly control the network topology and data placement – Loosely structured: Freenet (the file placement is based on hints) – Highly structured: Pastry, Chord, Tapestry, and CAN

7. Decentalized and Highly Structured P2P Systems Precise control of the network topology and data placement A distributed hash table (DHash) – Each node has a host-ID (hash of the public key or IP addr.) – Each file/object has a file-ID (hash of the file pathname) – Both files and nodes are mapped into the Dhash – Basic interface put(key, value) get(key)

8. Decentalized and Highly Structured P2P Systems A location and routing infrastructure – Application-level, routed by an ID not IP address – Routing effciency: O(logN) Advantages – Good scalability (O(logN) in routing effciency and routing table) – Reliability – Self-maintenance (node addition/removal) – Good performance (compared to other P2P systems) Issues – Routing performance (compared to IP routing) – Security – Other issues ……

9. P2P Applications Content delivery systems Application-level multicast Publishing/file sharing systems P2P storage systems (e.g., PAST, CFS, OceanStore) P2P file systems

10. Ivy: A Read/Write P2P File System Introduction Design Issues Performance Evaluation Summary

11. Introdcution Challenges: – Previous P2P systems are either read-only or one single writer, so multiple writers pose file system consistency issue – Unreliable participants render locking unattractive (for consistency) – Undo/ignore untrusty participants’ modifications – Security over untrusted storage of nodes – Resolve update conflicts due to network partition – High availability vs. strong consistency

12. Design Issues DHash infrastructure Log-based metadata and data NFS-like file system

13. DHash A distributed P2P hash table Stores participant’s logs Basic operations – put(key, value) – get(key) – E.g., key = content-hash of a log, value = log record

14. Log Data Structure One log per participant – A log contains all of one participant’s modifications (log records) to a file system data and metadata – Each log record is a content-hash block – Each participant appends log records only to its own log, but reads from all participants’ logs Ignore some untrusty participant’s modifications by without reading its log

15. Log Data Structure

18. Using the Log Append a log record – Derive a log record from a NFS request – Its prev field points to the last record – Insert the new log record into DHash – Sign a new log-head pointing to the new log record – Insert the new log-head into DHash

19. Using the Log File system creation – Create a new log with an End record – An Inode record with random i-number for the root directory – A log-head – Using the root i-number as the NFS root file handle

20. Using the Log File creation – Request: create (directory i-number, file name) – An Inode record with a new random i-number – A Link record – Return the NFS client with the i-number as a file handle – If write the file, create a Write record File read – Request: read (i-number, offset, length) – Scan logs accumulating data from Write records overlapping the range of data to be read, while ignoring data hiddened by SetAttr records that indicate file trucation.

21. Using the Log File name lookup – Request: open (directory i-number, file name) – Scan logs for a corresponding Link record – First encounter a corresponding Unlink record, indicating that the file doesn’t exist File attributes – File length, mtime, ctime, etc. – Scan logs to incrementally compute attributes

22. User Cooperation: Views View: the set of logs comprising a file system View block – A DHash content-hash block containing pointers to all log-heads in the view – Contains the root directory i-number – One Property: immutable (different file systems with different view blocks ) Name a file system with the content-hash key of its view block, like self-certifying file system (SFS)

23. Combining Logs Problem: – concurrent updates result in conflicts, how to order log records ? Solution: Version Vector in each log record – Detect update conflicts – E.g., (A:5, B:7) < (A:6, B:7)  compatible – (A:5, B:7) vs. (A:6, B:6)  concurrent version vectors, order them by comparing the public keys of two logs

24. Snapshots Problem ? – have to traverse the entire log to answer requests (high overhead and inefficiency). Solution: snapshots – Avoid traversing the entire log – Consistent state of the file system – Private per participant, periodically construct it – Stored in DHash, sharing contents among snapshots – Contains a file map, a set of i-nodes, and some data blocks, see Figure 2

25. Snapshot Data Structure

26. Snapshots Building snapshots – perform all log records newer than the previous snapshot Using snapshots – First traverse log records newer than current snapshot – If this can’t fulfill a NFS request, further search information in current snapshot – Mutually-trusted participants can share snapshots

27. Cache Consistency Most updates are immediately visible – Store the new log record and update the new log-head before replying to an NFS request – Query the latest log-heads for latest updates upon each NFS operation Modified close-to-open consistency for file reads/writes – Open()  fectch all log-heads for subsequent reads/writes – Write()  write data on its cache, defers writing data to DHash – Close()  push log records (if any by writes), update log- head

28. Exclusive Create Requirement: create directory entries be exclusive – Some applications use this semantics to implement locks Solution:

29. Partitioned Updates Close-to-open consistency guaranteed only if network is fully connected How if network partitioned? – Maximize availability (by allowing concurrent updates) – Compromise consistency – After partition heals, using Version Vectors – Application-level solver to resolve conflicts (Harp)

30. Security and Integrity Form another view to exlcude bad/misbehavoring/malicious participants Using content-hash key and public-hash key to protect data integrity

31. Evaluation Goal: understand the cost of Ivy’s design in terms of network latency and cryptographic operations Workload: Modified Andrew Benchmark (MAB) Performance in a WAN

32. Many Logs, One Writer The number of logs has relatively little impact – Because Ivy fetches the log-heads/log-records in parallel

33. Many DHash Servers More impact, since more messages are required to fetch log- records

34. Many Writers More impact, have to fetch other participants’ newly logged updates

35. Summary Log-based data/metadata, avoiding using locking Close-to-open consistency Tradeoff between high availabilty and strong consistency Allow concurrent updates, detect and reslove update conflicts Performance: 2-3 times slower than NFS Limitations ? – Small scale: limited to the number of logs – Hard to hide wide-area network latency

36. Thanks