1. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 11 DISTRIBUTED FILE SYSTEMS

2. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Server Architectures (1) Figure 11-1. (a) The remote access model. (b) The upload/download model.

3. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Server Architectures (2) Figure 11-2. The basic NFS architecture for UNIX systems.

4. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File System Model (1) Figure 11-3. An incomplete list of file system operations supported by NFS. Subsumed by create Subsumed by remove Replaced by open

5. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File System Model (2) Figure 11-3. An incomplete list of file system operations supported by NFS. Stateful server

6. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cluster-Based Distributed File Systems (2) Figure 11-5. The organization of a Google cluster of servers. Metadata es hints Master does not try to stay consistent all the time – update info by polling Massive numbers of servers = highly likely one or more is down! Chunks replicated Master keeps metainfo in RAM – like logging file system; reduces I/O

7. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Symmetric Architectures Figure 11-6. The organization of the Ivy distributed file system. Key choice: are files or blocks what is stored on servers?

8. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Cluster-Based Distributed File Systems (1) Figure 11-4. The difference between (a) distributing whole files across several servers and (b) striping files for parallel access.

9. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Remote Procedure Calls in NFS Figure 11-7. (a) Reading data from a file in NFS version 3. (b) Reading data using a compound procedure in version 4. No transaction semantics – just list of requests

10. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The RPC2 Subsystem (1) Figure 11-8. Side effects in Coda’s RPC2 system. Access to reference variables, devices, other side effects

11. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 The RPC2 Subsystem (2) Figure 11-9. (a) Sending an invalidation message one at a time. (b) Sending invalidation messages in parallel.

12. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File-Oriented Communication in Plan 9 Figure 11-10. Files associated with a single TCP connection in Plan 9. Examples:Server: write “announce ” to file ctl to accept connections Client: write “connect ! ” to file ctl to initiate a connection Both: write data to data file to send, read from data file to receive TCP connection represented by a subdirectory with files for actions, data TCP as unstructured bytes stream matches file paradigm well

13. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Naming in NFS (1) Figure 11-11. Mounting (part of) a remote file system in NFS. Path names depend on client mount points – FS view not uniform Only explicitly exported subdirectories can be mounted remotely

14. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Naming in NFS (2) Figure 11-12. Mounting nested directories from multiple servers in NFS. Handles do not include server reference Name resolution iterative in NFSv3

15. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Mount Time When to mount remote file system? Boot time + Consistent view of FS - May do unnecessary work - Takes longer to boot On explicit command by user + Give user control - Require user to know & do things Automount + “Subdirectories magically appear” - “Subdirectories magically appear”

16. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Automounting (1) Figure 11-13. A simple automounter for NFS. Automount daemon always involved in every file operation!

17. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Automounting (2) Figure 11-14. Using symbolic links with automounting. Automount daemon only involved in references to tmp_mnt path, and only on mount/unmount

18. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Constructing a Global Name Space Figure 11-15. Junctions in GNS. Similar to mount points, but deals with different file systems in WAN Explicitly separates resolution from file access Location service (per file system) handles mapping from name to location

19. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Semantics of File Sharing (1) Figure 11-16. (a) On a single processor, when a read follows a write, the value returned by the read is the value just written.

20. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Semantics of File Sharing (2) Figure 11-16. (b) In a distributed system with caching, obsolete values may be returned.

21. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Semantics of File Sharing (3) Figure 11-17. Four ways of dealing with the shared files in a distributed system. * - change = new version *WTH? Now have to deal with which version you get, resolving divergent paths At least no changes are lost – but they may get buried

22. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File Locking (1) Figure 11-18. NFSv4 operations related to file locking. Lock requests succeed or fail – no blocking Requester must poll until success or give up – no auto retry Lock manager will enqueue requester (if asked) – requester still needs to poll

23. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File Locking (2) – Share Res'n Figure 11-19. The result of an open operation with share reservations in NFS. (a) When the client requests shared access given the current denial state. Used to support Windows – independent from NFS locks Request includes access type and new denial state

24. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 File Locking (3) Figure 11-19. The result of an open operation with share reservations in NFS. (b) When the client requests a denial state given the current file access state. Open operation may fail for either access or denial request Granularity is at file level – way too coarse!

25. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Sharing Files in Coda Figure 11-20. The transactional behavior in sharing files in Coda.

26. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Side Caching (1) Figure 11-21. Client-side caching in NFS.

27. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Side Caching (2) Figure 11-22. Using the NFSv4 callback mechanism to recall file delegation.

28. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Client-Side Caching in Coda Figure 11-23. The use of local copies when opening a session in Coda.

29. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Server Replication in Coda Figure 11-24. Two clients with a different Accessible Volume Storage Group (AVSG) for the same replicated file. Coda takes optimistic approach for partition, uses Coda Version Vector (CVV) to resolve

30. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Structured Peer-to-Peer Systems Figure 11-25. Balancing load in a peer-to-peer system by replication. R checks its files to see if any should be copied at P, depending on P's load; pointers are cached.

31. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Handling Byzantine Failures Figure 11-26. The different phases in Byzantine fault tolerance. Client builds a quorum certificate from at least 2K replies from 3K others, with total order: (operation, view, number) of request

32. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Fault Tolerance Prob(accessible) = Σ i = m n nini ( ) p n-i (1-p) i Erasure Correction Coding (n,m) – can survive r=n-m failures: Storage increase = n/m times more storage Prob(accessible) =1 - p (r+1) Replication (r extra copies) – can survive r failures: Storage increase = r+1 times more storage Assumption: nodes fail independently with probability p

33. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 High Availability in Peer-to-Peer Systems Figure 11-27. The ratio r rep /r ec as a function of node availability a.

34. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Security in NFS Figure 11-28. The NFS security architecture. Basic Unix remote access “security” = pass uid and gid plus groups in clear.... D-H to establish session key

35. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Secure RPCs Figure 11-29. Secure RPC in NFSv4.

36. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Access Control Figure 11-30. The various kinds of users and processes distinguished by NFS with respect to access control.

37. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Decentralized Authentication (1) Figure 11-31. The organization of SFS.

38. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Decentralized Authentication (2) Figure 11-32. A self-certifying pathname in SFS. Host ID = H(LOC,K + s)

39. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Secure Lookups in DHT-Based Systems Secure routing requires that three issues are dealt with: 1.Nodes are assigned identifiers in a secure way. 2.Routing tables are securely maintained. 3.Lookup requests are securely forwarded between nodes. Sybil attack

40. Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5 Secure Collaborative Storage Figure 11-33. The principle of storage claims in the Samsara peer-to-peer system. Claims computed using hash Storage validated using challenge/hash