1. CENG334 Introduction to Operating Systems
13/03/07
CENG334 Introduction to Operating Systems
Network File System
Erol Sahin
Dept of Computer Eng.
Middle East Technical University
Ankara, TURKEY
Slides adapted from: Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google, Inc. Summer 2007
Except as otherwise noted, the content of this presentation is © Copyright University of Washington and licensed under the Creative Commons Attribution 2.5 License.

2. Distributed Filesystems
Support access to files on remote servers
Must support concurrency
Make varying guarantees about locking, who “wins” with concurrent writes, etc...
Must gracefully handle dropped connections
Can offer support for replication and local caching
Different implementations sit in different places on complexity/feature scale

3. NFS
First developed in 1980s by Sun Presented with standard UNIX FS interface Network drives are mounted into local directory hierarchy

4. NFS Protocol Initially completely stateless
Operated over UDP; did not use TCP streams
File locking, etc., implemented in higher-level protocols
Modern implementations use TCP/IP & stateful protocols

5. Server-side Implementation
NFS defines a virtual file system
Does not actually manage local disk layout on server
Server instantiates NFS volume on top of local file system
Local hard drives managed by concrete file systems (EXT, ReiserFS, ...)

6. NFS Locking NFS v4 supports stateful locking of files
Clients inform server of intent to lock
Server can notify clients of outstanding lock requests
Locking is lease-based: clients must continually renew locks before a timeout
Loss of contact with server abandons locks

7. NFS Client Caching
NFS Clients are allowed to cache copies of remote files for subsequent accesses
Supports close-to-open cache consistency
When client A closes a file, its contents are synchronized with the master, and timestamp is changed
When client B opens the file, it checks that local timestamp agrees with server timestamp. If not, it discards local copy.
Concurrent reader/writers must use flags to disable caching

8. NFS: Tradeoffs NFS Volume managed by single server
Higher load on central server
Simplifies coherency protocols
Full POSIX system means it “drops in” very easily, but isn’t “great” for any specific need

9. CENG334 Introduction to Operating Systems
13/03/07
CENG334 Introduction to Operating Systems
Google File System
Topics:
Erol Sahin
Dept of Computer Eng.
Middle East Technical University
Ankara, TURKEY
Slides adapted from Alex Moshchuk, University of Washington

10. Motivation Google needed a good distributed file system
Redundant storage of massive amounts of data on cheap and unreliable computers
Why not use an existing file system?
Google’s problems are different from anyone else’s
Different workload and design priorities
GFS is designed for Google apps and workloads
Google apps are designed for GFS

11. Assumptions High component failure rates “Modest” number of HUGE files
Inexpensive commodity components fail all the time
“Modest” number of HUGE files
Just a few million
Each is 100MB or larger; multi-GB files typical
Two kinds of reads: large streaming reads and small random reads
Applications batch and sort their small reads to advance steadily through a file
Files are write-once, mostly appended to
Perhaps concurrently
Multiple clients concurrently append to the same file
High sustained throughput favored over low latency

12. Interface
Implements a familiar file system interface, though it does not implement POSIX API.
Supports operations such as:
create
delete
open, close
read, write
Moreover
snapshot: creates a copy of a file of directory tree at low cost
record append: allows multiple clients to append data to the same file concurrently while guaranteeing the atomicity of each append.

13. GFS Architecture

14. GFS Architecture Replicas Master GFS Master Client C1 C0 C3 C4 C5
Essentially it is a large-scale distributed system. It has two main components. One of them is master which is responsible for metadata like filename and so on and then chunk server responsible for storing chunk of files.
GFS shares many of the same goals as previous distributed file systems such as performance, scalability, reliability, and availability. GFS shares many of the same goals
as previous distributed file systems such as performance, scalability, reliability, and availability.
First, component failures are the norm rather than the exception. The file system consists of hundreds or even thousands of storage machines built from inexpensive commodity parts and is accessed by a comparable number of client machines. The quantity and quality of the components virtually guarantee that some are not functional at any given time and some will not recover from their current failures. We have seen problems caused by application bugs, operating system bugs, human errors, and the failures of disks, memory, connectors, networking, and power supplies. Therefore, constant monitoring, error detection, fault tolerance, and automatic recovery must be integral to the system.
Second, files are huge by traditional standards. Multi-GB files are common. Each file typically contains many application objects such as web documents. When we are regularly working with fast growing data sets of many TBs comprising billions of objects, it is unwieldy to manage billions of approximately KB-sized files even when the file system could support it. As a result, design assumptions and parameters such as I/O operation and blocksizes have to be revisited.
Third, most files are mutated by appending new data rather than overwriting existing data. Random writes within a file are practically non-existent. Once written, the files are only read, and often only sequentially. A variety of data share these characteristics. Some may constitute large repositories that data analysis programs scan through. Some may be data streams continuously generated by running applications. Some may be archival data. Some may be intermediate results produced on one machine and processed on another, whether simultaneously or later in time. Given this access pattern on huge files, appending becomes the focus of performance optimization and atomicity guarantees, while caching data blocks in the client loses its appeal.
Fourth, co-designing the applications and the file system API benefits the overall system by increasing our flexibility. For example, we have relaxed GFS’s consistency model to vastly simplify the file system without imposing an onerous burden on the applications. We have also introduced an atomic append operation so that multiple clients can append concurrently to a file without extra synchronization between them.
Master manages metadata
Data transfers happen directly between clients/chunkservers
Files broken into chunks (typically 64 MB)
Chunks replicated across three machines for safety

15. GFS Design Decisions Single master to coordinate access, keep metadata
Simple centralized management
Files stored as chunks
Fixed size (64MB)
Reliability through replication
Each chunk replicated across 3+ chunkservers
No data caching
Little benefit due to large data sets, streaming reads
Familiar interface, but customize the API
Simplify the problem; focus on Google apps
Add snapshot and record append operations

16. Single master From distributed systems we know this is a:
Single point of failure
Scalability bottleneck
GFS solutions:
Shadow masters
Minimize master involvement
never move data through it, use only for metadata
and cache metadata at clients
large chunk size
master delegates authority to primary replicas in data mutations (chunk leases)
Simple, and good enough!

17. Metadata (1/2) Global metadata is stored on the master
File and chunk namespaces
Mapping from files to chunks
Locations of each chunk’s replicas
All in memory (64 bytes / chunk)
Fast
Easily accessible

18. Metadata (2/2)
Master has an operation log for persistent logging of critical metadata updates
persistent on local disk
replicated
checkpoints for faster recovery

19. Mutations Mutation = write or append Goal: minimize master involvement
must be done for all replicas
Goal: minimize master involvement
Lease mechanism:
master picks one replica as primary; gives it a “lease” for mutations
primary defines a serial order of mutations
all replicas follow this order
Data flow decoupled from control flow

20. Atomic record append Client specifies data
GFS appends it to the file atomically at least once
GFS picks the offset
works for concurrent writers
Used heavily by Google apps
e.g., for files that serve as multiple-producer/single-consumer queues

21. Relaxed consistency model (1/2)
“Consistent” = all replicas have the same value
“Defined” = replica reflects the mutation, consistent
Some properties:
concurrent writes leave region consistent, but possibly undefined
failed writes leave the region inconsistent
Some work has moved into the applications:
e.g., self-validating, self-identifying records

22. Relaxed consistency model (2/2)
Simple, efficient
Google apps can live with it
what about other apps?
Namespace updates atomic and serializable

23. Master’s responsibilities (1/2)
Metadata storage
Namespace management/locking
Periodic communication with chunkservers
give instructions, collect state, track cluster health
Chunk creation, re-replication, rebalancing
balance space utilization and access speed
spread replicas across racks to reduce correlated failures
re-replicate data if redundancy falls below threshold
rebalance data to smooth out storage and request load

24. Master’s responsibilities (2/2)
Garbage Collection
simpler, more reliable than traditional file delete
master logs the deletion, renames the file to a hidden name
lazily garbage collects hidden files
Stale replica deletion
detect “stale” replicas using chunk version numbers

25. Fault Tolerance High availability Data integrity fast recovery
master and chunkservers restartable in a few seconds
chunk replication
default: 3 replicas.
shadow masters
Data integrity
checksum every 64KB block in each chunk

26. Performance

27. Deployment in Google
Many GFS clusters hundreds/thousands of storage nodes each Managing petabytes of data GFS is under BigTable, etc.

28. Conclusion
GFS demonstrates how to support large-scale processing workloads on commodity hardware
design to tolerate frequent component failures
optimize for huge files that are mostly appended and read
feel free to relax and extend FS interface as required
go for simple solutions (e.g., single master)
GFS has met Google’s storage needs… it must be good!

29. More info.. http://labs.google.com/papers/gfs.html