1. Andrew File System (AFS)
Craig Shih & Todor Avramov
University of Washington, Bothell
CSS534: Parallel Programming
December 8, 2016

2. Outline Overview Consistency Architecture
Synchronization and Caching Issues
Scalability and Performance
AFS vs. NFS
References
Q&A

3. Overview
Introduced by researchers at Carnegie-Mellon University in the 1980’s
Distributed File System with Main goal of Scalability
Client-side (Venus), Server-side (Vice)
Basic idea of AFS – whole-file caching on local disk of client machine
2 version (AFSv1 and AFSv2)
4 major implementations
Transarc(IBM) (deprecated)
OpenAFS
ARLA
Linux Kernel Source Code (early stages of development)
AFS v1 – was originally called the ITC distributed file system – had basic design in place, but didn’t scale desirably
AFS client-side (venus)
Server side – vice
AFSv1 – onlyl cached file contents, not directories.
When file is called to be open – directory is passed to the server (vice) at which point the entire file is cached onto local disk. When finished (close is called) any changes would be stored back on server. Next time if the same file is called, would be more efficient since it checks if its been changed, if not will just open the local disk’s cached copy.
Two main problems with VERSION 1 – Path-traversal costs are too high – had to traverse whole path – too much cpu used
The client issues too many TestAuth protocol messages. – too many checks to see if local copy can be used – more often than not, local copy can be used
These two main issues limited scalability
AFS Version 2
Introduced Callback – to reduce number of client/server interactions. – simply a promise from the server to the client that the server will inform the client when a file that the client is caching has been modified.
Also introduced notion of a file –dentifier (FID) – instead of pathnames to specify which file a client was interested – FID would be used
FID consists of a Volume ID, a file ID, and a “uniquifier” – essentially using /home/remzi/notes.txt as an example – the first fetch of notes.txt will cache directory home, establish callback for that directory, then do the same with remzi, and then do the same with the actual notes.txt file. This way any changes to the entire path will be marked for callback. Server will notify client if any changes occur. This reduces the need to make multiple hits on the server once initial caching has been established.

4. AFS Version 2 Introduced Callbacks
Reduces number of client/server interactions
Is a promise from the server to the client that the server will inform client when a file that the client has cached has been modified.
Introduced notion of File Identifier (FID)
Replaced whole pathnames as a way to specify file location to server
FID consists of a Volume Identifier, File Identifier, and a “uniquifier”
”uniquifier” allows the reuse of Identifiers if a file is deleted.
Introduced Callbacks
Reduces number of client/server interactions
Is a promise from the server to the client that the server will inform client when a file that the client has cached has been modified.
Introduced notion of File Identifier (FID)
Replaced whole pathnames as a way to specify file location to server
FID consists of a Volume Identifier, File Identifier, and a “uniquifier”
”uniquifier” allows the reuse of Identifiers if a file is deleted.
Ex. /home/craig/todor/hello.txt
Will cache each directory (home, craig, todor)
Will then cache the file hello.txt and establish callback for that as well.
Server will then notify client if any changes occur.
Reduces the need to make multiple hits on the server once initial caching has been established

5. Callback/ FID Example

6. Consistency Guarantees
Update Visibility
When will the server be updated with a new version of the file?
Cache Staleness
Once the server has a new version, how long before clients see the new version instead of an older cached copy?
Two cases to consider:
Consistency between processes on different machines
Consistency between processes on the same machine
AFS Uses Weak (but practical) Consistency Model
Guarantees that after the completion of an operation, the next operation performed anywhere in the network will see the updated file system state

7. Consistency Mechanisms
New data written to a file is not stored back at the file server and visible to other clients until the file is closed.
Typical UNIX read/write semantics on the same machine – writes to a file are immediately visible to other local processes
Open/Close Granularity
Client nodes that already have the data cached on their local disks will not have to retrieve from the file server.
Minimizes network communication and file server load
Whole-file caching on local disk
Every time a file is modified in the client side, in addition to storing the file in the local cache, a copy is sent to the file server
Minimizes the effects of a client failure
Write-through Cache
When the server becomes aware that a particular file is modified by a client, the server breaks callbacks (initiates invalidation callbacks) for any clients with local cached copies of that file.
Subsequent opens on those clients require a re-fetch of the new version of the file from the server
Callbacks

8. Consistency Timeline

9. AFS Architecture
Vice
RPC
RPC

10. System Call Interception in AFS

11. Vice (File Server)

12. Synchronization and Caching Issues
Callbacks depend on reliable delivery of messages from server to clients Issues:
Network is unreliable and may prevent delivery of callback; breaking messages
Client may crash
File server may crash
Fixes
After recovery, treat all cache contents as suspect
Cache Timeout/Periodic polling (10 minutes)
> Synchronization: Update and Callback at the same time for the same file?
Locking the cache entries? (Too complicated and may result in deadlock)
Add a callback sequence number at the file server? (Best solution)
Discard the callback? (Ad hoc solution)
What happens when client updates a file and sends it to the fileserver while at the same time the file server calls the client to break a callback promise associated with the same file?

13. Scalability and Performance
Enterprise deployments may exceed 25,000 clients
Able to support about 50 clients per server
Files commonly accessed locally. File reads usually from local disk cache
Client-side performance came close to local performance

14. AFS vs NFS Performance
Assumptions: small and medium files fit into memory of a client – large files fit on a local disk but not in client memory.
Access across the network to the remote server for a file block takes Lnet time units. Access to local memory takes Lmem. And access to local disk takes Ldisk.
General assumption is that Lnet > Ldisk < Lmem

15. AFS vs NFS Qualitative AFS NFS Supported Clients Requires Disks
Diskless
Network Traffic
As required (much less)
Periodic (high)
Server Load
Client Response Times
Equal
OS Support
Minimal
More
Caching Protocol
Whole Files
Blocks of Files
Caching Mechanism
Client Disks
Client Memory
Security
Kerberos
Weaker more primitive
Admin Management
Simpler
Difficult

16. References http://pages.cs.wisc.edu/~remzi/OSTEP/dist-afs.pdf
Distributed Systems: Concepts and Design, Edition 4, Addison-Wesley Copyright © George Coulouris, Jean Dollimore, Tim Kindberg, Pearson Education 2005, Chapter 8.4 Andrew File System

17. Questions