1. A Distributed Storage System for Structured Data
Bigtable
A Distributed Storage System for Structured Data

2. Credit
Based on a paper by Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber
And the following presentations:
by Jeffery Dean in UW in September 2005
Big Table : A Distributed Storage System for Structured Data by Pouria Pirzadeh and Vandana Ayyalasomayajula of University of California, Irvine (BigtableLacture)
Google Bigtable - CSE, IIT Bombay by S. Sudarshan (bigtable-uw-presentaion)

3. What we do today Motivation Overview Data Model Client API
Building blocks
Implementation

4. Motivation Google scale is huge
Petabytes of data
Many incoming requests
Very different demands – in data size, workloads, and configurations
No commercial system big enough
Even if there was one, utilization would be expansive
Might have made design choices that doesn’t fit Google requirements
Motivation:
huge scale:
Personalized Search. records user queries and clicks across a variety of Google properties such as web search, images, and news. Users can browse their search histories, and ask for personalized search results based on their historical Google usage patterns.
very different demands
data size <from URLs to web pages to satellite imagery>
throughput-oriented batch-processing – Google Earth use a table to store raw imagery, which later processed latency-sensitive serving of data to end users -
No solution:
Based on “Big Table : A Distributed Storage System for Structured Data"

5. Overview Bigtable is widely applicable Scalable
Used by more than sixty products in Google
Scalable
Uses petabytes of data and thousands of servers
Use components already existing in Google
Commodity servers
Google technologies
Provides only a simple data model
Instead supports dynamic control over data layout and format
Google products and projects, including Google Analytics, Google Finance, Orkut, Personalized Search, Writely, and Google Earth.
dynamic control - clients can control the locality of data through choice of schema and the data location (disk/nenory) through schema parameters.
SQL

6. Data model
Big table is divided to clusters, each containing a set of tables
(row, colomn, timestamp)->data
Row’s name and column’s name are aribitrary strings
Data is uninterpreted array of bytes
Figure taken from Paper
Based on “Big Table : A Distributed Storage System for Structured Data"

7. Rows and Tablets Rows are ordered in lexicographic order
The rows are partitioned to tablet
Each tablet consist of a specific row range
Used as the basic unit for distribution and load balancing (later)
Operations on a single row are atomic
Figure taken from Paper

8. Columns and Columns Families
Columns are grouped into sets called column families
Which are basic unit of access control
Rarely changed
Can contain one column, or many
Column name family:qualifier
Same column family
Access control and both disk and memory accounting are performed at the column-family level.
The language family – contains only one column key, page’s language ID
The anchor family - each column key in this family represents a single anchor. The qualifier is the name of the referring site; the cell contents is the link text.
Figure taken from Paper

9. Different versions of the data, taken in different times
Timestamps
Each cell in a Bigtable can contain multiple versions of the same data
These versions are indexed by timestamp
Bigtable provide automatic garbage-collection for this data:
Keep only the last n versions of a cell
Keep only new-enough versions
column contents: column to the times at which these page versions were actually crawled. The garbage-collection mechanism described above lets us keep only the most recent three versions of every page
Different versions of the data, taken in different times
Figure taken from Paper

10. Client API Manage the schema Basic commands
Create and delete tables and column families
Change cluster, table, and column family metadata
Basic commands
Write or delete values
Look up values from individual rows
Scanning a subset of the data in a table
Scanner - iterate over multiple column families, several mechanisms for limiting the rows, columns, and timestamps produced by a scan
single-row transactions - perform atomic read-modify-write sequences on data stored under a single row key.
Execution of client-supplied scripts - written in a language developed at Google. does not allow client scripts to write back into Bigtable. does allow various forms of data transformation, filtering based on arbitrary expressions, and summarization via a variety of operators.

11. Client API More advanced commands Single-row transactions
Using cells as integer counter
Execution of client-supplied scripts in the address spaces of the servers
Scanner - iterate over multiple column families, several mechanisms for limiting the rows, columns, and timestamps produced by a scan
single-row transactions - perform atomic read-modify-write sequences on data stored under a single row key.
Execution of client-supplied scripts - written in a language developed at Google. does not allow client scripts to write back into Bigtable. does allow various forms of data transformation, filtering based on arbitrary expressions, and summarization via a variety of operators.

12. Building Blocks Google File System (GFS) A cluster management system
Large-scale distributed file system.
Used to store the on-disk files.
A cluster management system
Bigtable cluster run in a pool of machines
Such pool usually run other applications as well
So bitable depends on the cluster management system to manage its run
Schedule jobs
Deal with machine failures
Monitor machines status
And more..

13. Building Blocks Chubby
Highly-available and persistent distributed lock service.
Can store directories and small files.
Used for
Storing the Bigtable schema information
Tracking the master and tablet servers
And more
Chubby - Each directory or file can be used as a lock, and reads and writes to a file are atomic

14. Implementation - Master
Three major components:
Client library
One master server
Many tablet servers
Master server
One per cluster
Manage Tablet Servers
Garbage-collect of files in GFS
Handles schema changes
3 Components.
Manage Tablet Servers - Assign tablets to tablet servers. Balance tablet-server load. Detect the addition and expiration of tablet servers.
Handles schema changes - table and column family creations.
Based on “Big Table : A Distributed Storage System for Structured Data"

15. Implementation - Tablet Server
Manages a set of tablets
It handles read and write requests to the tablets that it has loaded
And splits tablets that have grown too large
Clients communicate with servers directly
Master lightly loaded
Can be dynamically added or removed from a cluster
Tablets moves between servers
Based on “Big Table : A Distributed Storage System for Structured Data"

16. Tablet Location Given a row key, how can clients find a tablet?
One approach: use the Master Server
Problem: Master becomes a bottleneck in large system
Instead: use special table containing tablet location info
But how can we find this special table?
Since tablets move around from server to server, given a row, how do clients find the right server?
Based on Jeff Dean’s Lecture

17. Tablet Location – Cont. 3-level hierarchy for location storing
Metadata table contains location of user tablets
Root tablet contains location of Metadata tablets
One file in Chubby for location of Root Tablet
Row’s key: Tablet’s Table ID, End Row
Client library caches tablet locations
Moves up the hierarchy if location is stale
Also prefetches tablet locations
for range queries
If the client’s cache is stale, the location algorithm could take up to six round-trips, because stale cache entries are only discovered upon misses (assuming that METADATA tablets do not move very frequently).
tablet locations are stored in memory
Figure taken from Paper
Based on Jeff Dean’s Lecture

18. (row, column, timestamp) -> data
Editing a table
Tablet
Memtable
Sorted buffer
(row, column, timestamp) -> data
Memory
Sstable – keys and values are arbitrary byte strings.
provide look up specified key, or iteration over all key/value pairs in a specified key range
look up requires only one read from disk - as the index is in memory. or can be loaded entirely to memory.
GFS
SSTables
Immutable, ordered map
(row, column, timestamp) -> data
Tablet Log
Append-only log

19. Write Write Request Tablet New Write Delete Delete Write Write Delete
Memory
Memtable
When a write operation arrives at a tablet server:
checks that it is well-formed, and that the sender is authorized to perform the mutation
A valid mutation is written to the commit log.
After the write has been committed, its contents are inserted into the memtable..
Deletes:
Represented by a special deletion entries
The data is deleted later
New Write
Delete
Delete
Write
Write
GFS
Tablet Log
SSTables

20. Read Read Request Tablet Row’s Data Write Delete Delete Write Write
Memory
Memtable
Row’s data
When a read operation arrives at a tablet server:
The server checked for well-formedness and proper authorization.
A valid read operation is executed on a merged view of the sequence of SSTables and the memtable.
Since the SSTables and the memtable are lexicographically sorted data structures, the merged view can be formed efficiently.
Incoming read and write operations can continue while tablets are split.
New Write
Delete
Delete
Write
Write
GFS
Tablet Log
SSTables

21. What the Tablet Log is for?
Tablet recovery
When a tablet server crashes, its tablets are moved to other tablet servers
The SSTables are on the disk, so they survive
But what of the memtable?
The tablet server need to reconstruct the memtable
Done by applying all of the updates that in the tablet log that haven’t been written to SSTable yet
The locations of the tablet’s list of SSTables and tablet log are kept in the METADATA table

22. Minor Compaction
Convert the memtable into an SSTable
Tablet
Write
Delete
Memtable
Memory
When the memtable size reaches a threshold, the memtable is frozen, a new memtable is created, and the frozen memtable is converted to an SSTable and written to GFS.
Reduce memory usage
Reduce the amount of data that has to be read from the commit log during recovery
Incoming read and write operations can continue while compactions occur
Row’s data
Write
Delete
Delete
Write
Write
GFS
SSTables
Tablet Log

23. Merging Compaction Merging compaction Reduce number of SSTables
Convert the memtable and some of the SSTable to a new SSTable
Tablet
Write
Delete
Memtable
Memory
Merging compaction
Reduce number of SSTables
Without it, read operations might need to merge updates from an arbitrary number of SSTables
Executed periodically
The input SSTables and memtable are discarded when the compaction has finished
Write
Delete
Delete
Write
Write
GFS
SSTables
Tablet Log

24. Major Compaction Major compaction
Convert the memtable and all the SSTable to a new SSTable
Tablet
Write
Delete
Memtable
Memory
Major compaction
rewrites all SSTables into exactly one SSTable
No deletion records, only live data
allow Bigtable to reclaim resources used by deleted data,
ensure that deleted data disappears from the system in time
Write
Delete
Delete
Write
Write
GFS
SSTables
Tablet Log

25. Questions?

26. Refinement – Locality Groups
Remember column families?
Locality groups group together several column families
Each locality group is kept in a separate SSTable
Allows more efficient reads
Each read access smaller SSTables
Can be declared in-memory
Instead of on the disk
Useful to small locality groups
For example, page metadata in Webtable (such as language and checksums) can be in one locality group, and the contents of the page can be in a different group: an application that wants to read the metadata does not need to read through all of the page contents.
SSTables for in-memory locality groups are loaded lazily into the memory of the tablet server. Once loaded, column families that belong to such locality groups can be read without accessing the disk.
This feature is useful for small pieces of data that are accessed frequently: we use it internally for the location column family in the METADATA table.
Based on “Big Table : A Distributed Storage System for Structured Data"

27. Refinement – Bloom Filters
Read operation has to read from all the SSTables of the tablet
Can result in many disk accesses
Clients can create Bloom filters for SSTable of specific locality group
Filters kept in the memory
Bloom filter
Probabilistic data structure that is used to test whether an element is a member of a set
Can’t have false negative
Don’t allow deletions
Figure taken from Wikipedia

28. Refinement - Caching Two levels of caching Scan cache Block cache
Higher-level cache
Caches the key-values read from the SSTables
Useful when reading the same data over and over
Block cache
Lower-level cache
Caches the SSTable blocks that were read from the GFS
Useful when scanning the data or when reading different columns in the same locality group within the same row

29. Refinement - Immutability
SSTables are immutable
Simplifies caching
No need for synchronization of accesses when reading from SSTables
Garbage collection of SSTables done by master
On tablet split, child tablets share the SSTables of the father
Only memtable accessed by both reads and write
But it allows concurrent read/write
Based on “Google Bigtable - CSE, IIT Bombay"

30. Questions?