Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung

Paper highlights Presents a DFS tailored to a very specific workload Mostly huge files Mostly append-only updates Mostly sequential reads Client will try to order non-sequential read Non-standard client API

The environment Component failures are frequent Large system built from commodity parts Files are huge Multi-GB are frequent Most files are mutated by appending Large repositories, data streams, archival data GFS API co-designed along with applications

Design assumptions (I) Built from many cheap commodity components Frequent failures Modest number of very large files 100 MB or more Two kinds or reads Large sequential reads (100's of KB and more) Small random reads Often batched and sorted by applications

Design assumptions Many large sequential writes Append data to files Random small writes are rare Must implement well-defined semantics for multiple clients that concurrently append data to the same file Sustained bandwidth is more important than latency

Interface Familiar POSIX API create, delete, open, close, read and write Two additional operations snapshot Creates a copy of a file or a directory record append: Allows multiple clients to append data to the same file at the same time Guarantees the atomicity of updates

Architecture (I) Single master + multiple chunkservers Files divided into fixed-size chunks Each chunk has a unique immutable 64-bit chunk handle Assigned by the master at chunk creation Chunks are Stored as Linux files Replicated (default is three replicas)

Architecture (II) The master Maintains all metadata: Namespace, access control, mapping from files to chunks, chunk locations Controls all system-wide activities: Garbage collection, chunk migration Communicates to chunkservers through HearthBeats Having a single master simplify the design

Architecture (III) Clients interact with Master for all system metadata operations Directly to chunkservers for all data transfers No data caching anywhere in the system Most applications stream through huge files Would be ineffective Clients cache metadata

Overview

A typical interaction Client translates byte offset into chunk index within file Easy because chunks have fixed-sizes Sends request to master with file name + chunk index Master replies with chunk handle and chunk replica locations Typical requests are for multiple chunks

Chunk sizes 64 MB (that's megabytes!) Advantages Reduces need for clients to interact with master Many applications exhibit spatial locality Disadvantage: May cause hot spots if many clients have to access the same one-chunk file Only happened for some executables Solution was to have more replicas

Metadata Master stores in memory File and chunk namespaces Mapping from files to chunks Locations of each chunk replicas First two are kept persistent by logging mutations on a operation log Stored on the master's local disk Replicated on remote hosts Chunk replica locations are collected from chunk servers each time the master reboots

In-memory data structures Make master operations fast Let master perform periodic maintenance operations in an efficient manner Chunk garbage collection Re-replication after a chunkserver failure Chunk migration to balance load and disk space Less than 64 bytes per 64MB chunk

Chunk locations (I) Non-persistent Can be stored in main memory Master polls chunkservers At startup Periodically after that (HeartBeat messages) Greatly simplifies the system design No need to keep master and chunkservers synchronized

Chunk locations (II) Main principle Each chunkserver has the final word over what chunks it stores on does not sore on its local disk Eliminates consistency issues when a chunkserver crashes or another integrates the system

Operation log (I) Historical record of critical metadata changes Only persistent record of metadata Provides a logical timeline File chunks and their versions are identified by the logical time at which they were created Replicated on several machines Changes cannot be visible to clients until metadata changes ae made persistent

Operation log (II) When master needs to recover the state of its file system, it will replay the log To minimize recovery times, log must kept short Master checkpoints its state Each time the log reaches a maximum size Checkpoint is a B-tree Can be directly loaded in main memory

Consistency model Relaxed consistency A file region is consistent if all clients will always see the same data on all replicas Product of concurrent successful writes A file region is said to be defined if It is consistent Clients will see the mutation writes in their entirety Product of serial writes

Namespace mutations Atomic Handled exclusively by the master Uses locks Serial order defined by the master's operation log

Data mutations Can be Writes Record appends Writes write at an application specified write offset Record appends cause data to be appended at most once at an offset of GFS choosing GFS may insert padding or duplicate appends

From the rest of the paper Shadow masters Additional masters that "follow" the actual master by implementing the actions recorded on the master's operation log Not perfect mirrors Offer read-only access to GFS

You are not responsible for the rest of the paper