The Google File System Ghemawat, Gobioff, Leung via Kris Molendyke CSE498 WWW Search Engines LeHigh University.

Slides:



Advertisements
Similar presentations
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia.
Advertisements

Question Scalability vs Elasticity What is the difference?
Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
The google file system Cs 595 Lecture 9.
THE GOOGLE FILE SYSTEM CS 595 LECTURE 8 3/2/2015.
G O O G L E F I L E S Y S T E M 陳 仕融 黃 振凱 林 佑恩 Z 1.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google Jaehyun Han 1.
The Google File System Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani 1CS5204 – Operating Systems.
GFS: The Google File System Brad Karp UCL Computer Science CS Z03 / th October, 2006.
NFS, AFS, GFS Yunji Zhong. Distributed File Systems Support access to files on remote servers Must support concurrency – Make varying guarantees about.
The Google File System (GFS). Introduction Special Assumptions Consistency Model System Design System Interactions Fault Tolerance (Results)
Google File System 1Arun Sundaram – Operating Systems.
Lecture 6 – Google File System (GFS) CSE 490h – Introduction to Distributed Computing, Winter 2008 Except as otherwise noted, the content of this presentation.
The Google File System. Why? Google has lots of data –Cannot fit in traditional file system –Spans hundreds (thousands) of servers connected to (tens.
The Google File System and Map Reduce. The Team Pat Crane Tyler Flaherty Paul Gibler Aaron Holroyd Katy Levinson Rob Martin Pat McAnneny Konstantin Naryshkin.
1 The File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung (Google)
GFS: The Google File System Michael Siegenthaler Cornell Computer Science CS th March 2009.
Large Scale Sharing GFS and PAST Mahesh Balakrishnan.
The Google File System.
Google File System.
Northwestern University 2007 Winter – EECS 443 Advanced Operating Systems The Google File System S. Ghemawat, H. Gobioff and S-T. Leung, The Google File.
Case Study - GFS.
Google Distributed System and Hadoop Lakshmi Thyagarajan.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
The Hadoop Distributed File System: Architecture and Design by Dhruba Borthakur Presented by Bryant Yao.
1 The Google File System Reporter: You-Wei Zhang.
CSC 456 Operating Systems Seminar Presentation (11/13/2012) Leon Weingard, Liang Xin The Google File System.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
Homework 1 Installing the open source cloud Eucalyptus Groups Will need two machines – machine to help with installation and machine on which to install.
The Google File System Presenter: Gladon Almeida Authors: Sanjay Ghemawat Howard Gobioff Shun-Tak Leung Year: OCT’2003 Google File System14/9/2013.
The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
Outline for today  Administrative  Next week: Monday lecture, Friday discussion  Objective  Google File System  Paper: Award paper at SOSP in 2003.
MapReduce and GFS. Introduction r To understand Google’s file system let us look at the sort of processing that needs to be done r We will look at MapReduce.
CENG334 Introduction to Operating Systems Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY Network File System Except as.
Presenters: Rezan Amiri Sahar Delroshan
GFS : Google File System Ömer Faruk İnce Fatih University - Computer Engineering Cloud Computing
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.
GFS. Google r Servers are a mix of commodity machines and machines specifically designed for Google m Not necessarily the fastest m Purchases are based.
HADOOP DISTRIBUTED FILE SYSTEM HDFS Reliability Based on “The Hadoop Distributed File System” K. Shvachko et al., MSST 2010 Michael Tsitrin 26/05/13.
Presenter: Seikwon KAIST The Google File System 【 Ghemawat, Gobioff, Leung 】
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture Chunkservers Master Consistency Model File Mutation Garbage.
Google File System Robert Nishihara. What is GFS? Distributed filesystem for large-scale distributed applications.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition, Lecture 24: GFS.
Google File System Sanjay Ghemwat, Howard Gobioff, Shun-Tak Leung Vijay Reddy Mara Radhika Malladi.
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Presenter: Chao-Han Tsai (Some slides adapted from the Google’s series lectures)
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M th October, 2008.
Dr. Zahoor Tanoli COMSATS Attock 1.  Motivation  Assumptions  Architecture  Implementation  Current Status  Measurements  Benefits/Limitations.
1 CMPT 431© A. Fedorova Google File System A real massive distributed file system Hundreds of servers and clients –The largest cluster has >1000 storage.
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung
Cloud Computing Platform as a Service The Google Filesystem
File and Storage Systems: The Google File System
Google File System.
GFS.
The Google File System (GFS)
Google Filesystem Some slides taken from Alan Sussman.
Google File System CSE 454 From paper by Ghemawat, Gobioff & Leung.
Gregory Kesden, CSE-291 (Storage Systems) Fall 2017
Gregory Kesden, CSE-291 (Cloud Computing) Fall 2016
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung Google Presented by Jiamin Huang EECS 582 – W16.
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
The Google File System (GFS)
THE GOOGLE FILE SYSTEM.
by Mikael Bjerga & Arne Lange
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP’03, October 19–22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae.
The Google File System (GFS)
Presentation transcript:

The Google File System Ghemawat, Gobioff, Leung via Kris Molendyke CSE498 WWW Search Engines LeHigh University

Operating Environment Component Failure It’s the norm, not the exception Expect: Application, OS bugs Human error Hardware failure:  Disks  Memory  Power Supply Need for: Constant monitoring Error detection Fault tolerance Automatic Recovery

Non-standard data needs Files are huge Typically multi-GB Data sets can be many TB Cannot deal with typical file sizes Would mean billions of KB sized files- I/O problems! Must rethink block sizes

Non-standard writing File mutation Data is often appended, rather than overwritten Once a file is written it is read only, typically read only sequentially Optimization focuses on appending to large files and atomicity, not on caching

File System API & Applications Applications & API are co-designed Increases flexibility Goal is simple file system, light burden on applications Atomic append built into API to make it easy on applications with multiple clients to append with minimal synchronization

Design Assumptions Built from cheap commodity hardware Expect large files: 100MB to many GB Support large streaming reads and small random reads Support large, sequential file appends Support producer-consumer queues for many-way merging and file atomicity Sustain high bandwidth by writing data in bulk

Interface and operations Interface resembles standard file system- hierarchical directories and pathnames Standard operations such as create, delete, open, close, read, write Non-standard operations: Snapshot: low cost copy of directory tree or file Record append: allows multiple clients to append data concurrently while guaranteeing atomicity without blocking (more later)

GFS Architecture 1 master, at least 1 chunkserver, many clients

Master Responsibilities Maintain metadata: information like namespace, access control info, mappings from files to chunks, chunk locations Activity control Chunk lease management Garbage collection Chunk migration Communication with chunks Heartbeats: periodic syncing with chunkservers Lack of caching Block size too big (64 MB) Applications stream data (Clients will cache metadata, though)

Chunks Identified by 64 bit chunk handles Globally unique and assigned by master at chunk creation time Triple redundancy: copies of chunks kept on multiple chunkservers Chunk size: 64 MB! Plain Linux file Advantages Reduces R/W & communication between clients & master Reduces network overhead- persistent TCP connection Reduces size of metadata on master Disadvantage Hotspots- small files of just one chunk that many clients need

Metadata 3 main types to keep in memory File and chunk namespaces Mappings from files to chunks Locations of chunk’s replicas First two are also kept persistently in log files to ensure reliability and recoverability Chunk locations are held by chunkservers- master polls them for locations with a heartbeat occasionally (and on startup)

What About Consistency? GFS Guarantees File namespace mutations are atomic (file creation…) File Regions Consistent: all clients see same data, regardless which replicated chunk they use Defined: after data mutation (writes or record appends) if it is consistent & all clients will see the entire mutation effect

System Operations Leases Mutations are done at all chunk’s replicas Master defines primary chunk which then decides serial order to update replicas Timeout of 60 seconds- can be extended with help of heartbeats from master

Write Control & Data Flow 1. Client asks master for chunkserver 2. Master replies with primary & secondaries (client caches) 3. Client pushes data to all 4. Clients sends write to primary 5. Primary forwards serially 6. Secordaries return completed write 7. Primary replies to client

Decoupling Data & flow control are separated Use network efficiently- pipeline data linearly along chain of chunkservers Goals Fully utilize each machine’s network bandwidth Avoid network bottlenecks & high-latency Pipeline is “carefully chosen” Assumes distances determined by IP addresses

Atomic record appends Record append GFS appends data to a file at least once atomically (as a single continuous sequence of bytes) Similar to Unix O_APPEND without race condition concerns when faced with multiple concurrent writers Extra logic to implement this behavior Primary checks to see if appending data exceeds current chunk boundary If so, append new chunk padded out to chunk size and tell client to try again next chunk When data fits in chunk, write it and tell replicas to do so at exact same offset Primary keeps replicas in sync with itself

Master operation Executes all namespace operations Manages chunk replicas throughout system Placement decisions Creation Replication Load balancing chunkserver Reclaim unused storage throughout system

Replica Placement Issues Worries Hundreds of chunkservers spread over many machine racks Hundreds of clients accessing from any rack Aggregate bandwidth of rack < aggregate bandwidth of all machines on rack Want to distribute replicas to maximize data Scalability Reliability Availability

Where to Put a Chunk Considerations It is desirable to put chunk on chunkserver with below-average disk space utilization Limit the number of recent creations on each chunkserver- avoid heavy write traffic Spread chunks across multiple racks

Re-replication & Rebalancing Re-replication occurs when the number of available chunk replicas falls below a user-defined limit When can this occur? Chunkserver becomes unavailable Corrupted data in a chunk Disk error Increased replication limit Rebalancing is done periodically by master Master examines current replica distribution and moves replicas to even disk space usage Gradual nature avoids swamping a chunkserver

Collecting Garbage Lazy Update master log immediately, but… Do not reclaim resources for a bit (lazy part) Chunk is first renamed, not removed Periodic scans remove renamed chunks more than a few days old (user-defined) Orphans Chunks that exist on chunkservers that the master has no metadata for Chunkserver can freely delete these

Lazy Garbage Pros/Cons Pros Simple, reliable Makes storage reclamation a background activity of master- done in batches when the master has the time (lets master focus on responding to clients, putting speed at priority) A simple safety net against accidental, irreversible deletion Con Delay in deletion can make it harder to fine tune storage in tight situations

Stale replica detection Missed mutations on a chunk make it stale Chunk version number- makes it possible to detect stale chunks New leases increment chunk version number Stale chunks are removed in garbage collection

Fault Tolerance Cannot trust hardware Goal is high availability. How? Fast recovery Replication Fast Recovery Master & chunkservers can restore state and restart in a matter of seconds regardless of failure type No distinction between failure types at all Chunk Replication Default to keep replicas on at least 3 different racks

More than One Master! Master replication Really only one master in control- replicas are nearly synchronous heirs to the throne Replicas spread over multiple machines After failure can restart almost instantly Controlled by a monitoring infrastructure outside of GFS Shadow Master Provides read-only access to the FS when master is down May lag slightly- hence shadow, not mirror

Data Integrity Chunkservers use checksumming Each chunkserver responsible to verify its own copies of data Chunkservers will not propagate errors Will not return data to requester on checksum failure Bad chunks collected with garbage Master replicates a good copy of the chunk to replace bad Little effect on performance

Performance Environment GFS Cluster 1 master 2 master replicas 16 chunkservers 16 clients Hardware Dual 1.4 GHz PIII 2 GB memory 2 80 GB HD (5400 RPM) 100 Mbps Connections 2 switches  19 GFS machines connected to one  16 clients connected to other  Switches connected by 1Gbps link

Aggregate Throughput 10 MB/s 94 MB/s 6.3 MB/s 35 MB/s 6 MB/s 4.8 MB/s Reads Random 4 MB region from 320 GB, 256 times Writes 1 GB by each client in 1 MB sizes

Real World Clusters Cluster Stats Performance after 1 week % Operations Performed % Master Request Types Cluster X: R&D machine, Cluster Y: Production Data Processing 67% 97% 50% 45% 95% 85%

Final Issues More infrastructure needed to keep users from interfering with each other Linux & HD problems thanks to IDE protocol versions- motivated checksumming Additional concerns with costs of certain calls and locking issues

Conclusions GFS is radically different than traditional FS Component failure is normal Optimize for huge files Append as much as possible Much attention paid to Monitoring Replication Recovery Separate control and data flow Minimize master involvement with common operations to avoid bottleneck Design is a success and widely used in Google