Google Bigtable A Distributed Storage System for Structured Data Hadi Salimi, Distributed Systems Laboratory, School of Computer Engineering, Iran University of Science and Technology,

Introduction BigTable is a distributed storage system for managing structured data. Scales to Petabytes of data and thousands of machines. Developed and in use at Google since Used for more than 60 Google products. 2

Data Model (row, column, time) => string Row, column, value are arbitrary strings. Every read or write of data under a single row key is atomic (regardless of the number of different columns being read or written in the row). Columns are dynamically added. Timestamps for different versions of data. – Assigned by client application. – Older versions are garbage-collected. Example: Web map 3

Tablets Rows are sorted lexicographically. Consecutive keys are grouped together as “tablets”. – Allows data locality. – Example rows: com.google.maps/index.html and com.google.maps/foo.html are likely to be in same tablet. 4

Column Families Column keys are grouped into sets called “column families”. Column key is named using syntax: family:qualifier Access control and disk/memory accounting are at column family level Example: “anchor:cnnsi.com” 5

API Data Design – Creating/deleting tables and column families – Changing cluster, table and column family metadata like access control rights Client Interactions – Write/Delete values – Read values – Scan row ranges – Single-row transactions (e.g., read/modify/write sequence for data under a row key) Map/Reduce integration. – Read from Big Table; Write to Big Table. 6

Building Blocks SSTable file: Data structure for storage – Maps keys to values – Ordered. Enables data locality for efficient writes/reads. – Immutable. On reads, no concurrency control needed. Need to garbage collect deleted data. – Stored in Google File System (GFS), and optionally can be mapped into memory. Replicates data for redundancy. Chubby: Distributed lock service. – Store the root tablet, schema info, access control list – Synchronize and detect tablet servers 7

Implementation 3 components: 1.Client library 2.Master Server (exactly 1). Assigns tablets to tablet servers. Detecting the addition and expiration of tablet servers. Balancing tablet-server load Garbage collection of GFS files Schema changes such as table and column family creations. 3.Tablet Servers (multiple, dynamically added/removed) Handles read and write requests to the tablets that it has loaded Splits tablets that have grown too large. Each tablet MB. 8

Tablet Location How to know which node to route client request? 3-level hierarchy – One file in Chubby for location of Root Tablet – Root tablet contains location of Metadata tablets – Metadata table contains location of user tablets Row: [Tablet’s Table ID] + [End Row] Key: [Node ID] Client library caches tablet locations. 9

Tablet Assignment Master keeps track of tablet assignment and live servers Chubby – Tablet server creates & locks a unique file. – Tablet server stops serving if loses lock. – Master periodically checks tablet servers. If fails, master tries to lock the file and un-assigns the tablet. – Master failure does not change tablets assignments. Master restart 10

Tablet Serving Write 1.Check well-formedness of request. 2.Check authorization in Chubby file. 3.Write to “tablet log” (i.e., a transaction log for “redo” in case of failure). 4.Write to memtable (RAM). 5.Separately, “compaction” moves memtable data to SSTable. And truncates tablet log. 11 Read 1.Check well-formedness of request. 2.Check authorization in Chubby file. 3.Merge memtable and SSTables to find data. 4.Return data.

Compaction In order to control size of memtable, tablet log, and SSTable files, “compaction” is used. 1.Minor Compaction. Move data from memtable to SSTable. Truncate tablet log. 2.Merging Compaction. Merge multiple SSTables and memtable to a single SSTable. 3.Major Compaction. Remove deleted data. 12

Refinements Locality group. – Client can group multiple column families into a locality group. Enables more efficient reads since each locality group is a separate SSTable. Compression. – Client can choose to compress at locality group level. Two level caching in servers – Scan cache ( K/V pairs) – Block cache (SSTable blocks read from GFS) Bloom filter – Efficient check if a SSTable contain data for a row/column pair. Commit log implementation – Each tablet server has a single commit log (not one-per-tablet). 13

Performance Evaluation Random reads are slowest. Need to access SSTable block from disk. Writes are faster than reads. Commit log is append-only. Reads require merging of SSTables and memtable. Scans reduce number of read operations. 14

Performance Evaluation: Scaling Not linear, but not bad up to 250 tablet servers. Random read has worst scaling. Block transfers saturate network. 15

Conclusions Satisfies goals of high-availability, high-performance, massively scalable data storage. API. Successfully used by various Google products (>60). Additional features in progress: – Secondary indexes – Cross data center replication. – Deploy as a hosted service. Advantages of the custom development: – Significant flexibility due to own data model. – Can remove bottlenecks and inefficiencies as they arise. 16

Big Table Family Tree 17 Non-relational DBs (HBase, Cassandra, MongoDB, etc.) Column-oriented data model. Multi-level storage (commit log, RAM table, SSTable) Tablet management (assignment, splitting, recovery, GC, Bloom filters) Google related technologies and open-source equivalents GFS => Hadoop Distributed File System (HDFS) Chubby => Zookeeper Map/Reduce => Apache Map/Reduce

Any Question ? 18