Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 11 DISTRIBUTED FILE SYSTEMS

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-Server Architectures (1) Figure (a) The remote access model. (b) The upload/download model.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-Server Architectures (2) Figure The basic NFS architecture for UNIX systems.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File System Model (1) Figure An incomplete list of file system operations supported by NFS. Subsumed by create Subsumed by remove Replaced by open

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File System Model (2) Figure An incomplete list of file system operations supported by NFS. Stateful server

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Cluster-Based Distributed File Systems (2) Figure The organization of a Google cluster of servers. Metadata es hints Master does not try to stay consistent all the time – update info by polling Massive numbers of servers = highly likely one or more is down! Chunks replicated Master keeps metainfo in RAM – like logging file system; reduces I/O

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Symmetric Architectures Figure The organization of the Ivy distributed file system. Key choice: are files or blocks what is stored on servers?

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Cluster-Based Distributed File Systems (1) Figure The difference between (a) distributing whole files across several servers and (b) striping files for parallel access.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Remote Procedure Calls in NFS Figure (a) Reading data from a file in NFS version 3. (b) Reading data using a compound procedure in version 4. No transaction semantics – just list of requests

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved The RPC2 Subsystem (1) Figure Side effects in Coda’s RPC2 system. Access to reference variables, devices, other side effects

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved The RPC2 Subsystem (2) Figure (a) Sending an invalidation message one at a time. (b) Sending invalidation messages in parallel.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File-Oriented Communication in Plan 9 Figure Files associated with a single TCP connection in Plan 9. Examples:Server: write “announce ” to file ctl to accept connections Client: write “connect ! ” to file ctl to initiate a connection Both: write data to data file to send, read from data file to receive TCP connection represented by a subdirectory with files for actions, data TCP as unstructured bytes stream matches file paradigm well

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Naming in NFS (1) Figure Mounting (part of) a remote file system in NFS. Path names depend on client mount points – FS view not uniform Only explicitly exported subdirectories can be mounted remotely

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Naming in NFS (2) Figure Mounting nested directories from multiple servers in NFS. Handles do not include server reference Name resolution iterative in NFSv3

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Mount Time When to mount remote file system? Boot time + Consistent view of FS - May do unnecessary work - Takes longer to boot On explicit command by user + Give user control - Require user to know & do things Automount + “Subdirectories magically appear” - “Subdirectories magically appear”

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Automounting (1) Figure A simple automounter for NFS. Automount daemon always involved in every file operation!

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Automounting (2) Figure Using symbolic links with automounting. Automount daemon only involved in references to tmp_mnt path, and only on mount/unmount

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Constructing a Global Name Space Figure Junctions in GNS. Similar to mount points, but deals with different file systems in WAN Explicitly separates resolution from file access Location service (per file system) handles mapping from name to location

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Semantics of File Sharing (1) Figure (a) On a single processor, when a read follows a write, the value returned by the read is the value just written.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Semantics of File Sharing (2) Figure (b) In a distributed system with caching, obsolete values may be returned.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Semantics of File Sharing (3) Figure Four ways of dealing with the shared files in a distributed system. * - change = new version *WTH? Now have to deal with which version you get, resolving divergent paths At least no changes are lost – but they may get buried

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File Locking (1) Figure NFSv4 operations related to file locking. Lock requests succeed or fail – no blocking Requester must poll until success or give up – no auto retry Lock manager will enqueue requester (if asked) – requester still needs to poll

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File Locking (2) – Share Res'n Figure The result of an open operation with share reservations in NFS. (a) When the client requests shared access given the current denial state. Used to support Windows – independent from NFS locks Request includes access type and new denial state

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved File Locking (3) Figure The result of an open operation with share reservations in NFS. (b) When the client requests a denial state given the current file access state. Open operation may fail for either access or denial request Granularity is at file level – way too coarse!

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Sharing Files in Coda Figure The transactional behavior in sharing files in Coda.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-Side Caching (1) Figure Client-side caching in NFS.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-Side Caching (2) Figure Using the NFSv4 callback mechanism to recall file delegation.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Client-Side Caching in Coda Figure The use of local copies when opening a session in Coda.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Server Replication in Coda Figure Two clients with a different Accessible Volume Storage Group (AVSG) for the same replicated file. Coda takes optimistic approach for partition, uses Coda Version Vector (CVV) to resolve

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Structured Peer-to-Peer Systems Figure Balancing load in a peer-to-peer system by replication. R checks its files to see if any should be copied at P, depending on P's load; pointers are cached.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Handling Byzantine Failures Figure The different phases in Byzantine fault tolerance. Client builds a quorum certificate from at least 2K replies from 3K others, with total order: (operation, view, number) of request

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Fault Tolerance Prob(accessible) = Σ i = m n nini ( ) p n-i (1-p) i Erasure Correction Coding (n,m) – can survive r=n-m failures: Storage increase = n/m times more storage Prob(accessible) =1 - p (r+1) Replication (r extra copies) – can survive r failures: Storage increase = r+1 times more storage Assumption: nodes fail independently with probability p

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved High Availability in Peer-to-Peer Systems Figure The ratio r rep /r ec as a function of node availability a.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Security in NFS Figure The NFS security architecture. Basic Unix remote access “security” = pass uid and gid plus groups in clear.... D-H to establish session key

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Secure RPCs Figure Secure RPC in NFSv4.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Access Control Figure The various kinds of users and processes distinguished by NFS with respect to access control.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Decentralized Authentication (1) Figure The organization of SFS.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Decentralized Authentication (2) Figure A self-certifying pathname in SFS. Host ID = H(LOC,K + s)

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Secure Lookups in DHT-Based Systems Secure routing requires that three issues are dealt with: 1.Nodes are assigned identifiers in a secure way. 2.Routing tables are securely maintained. 3.Lookup requests are securely forwarded between nodes. Sybil attack

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved Secure Collaborative Storage Figure The principle of storage claims in the Samsara peer-to-peer system. Claims computed using hash Storage validated using challenge/hash