1. Data Management in the Cloud Paul Szerlip

2. The rise of data Think about this o For the past two decades, the largest generator of data was humans -- now it's our devices Cheap sensors o Cellphones are packed with sensory information  Images, video, audio, etc Expensive sensors o DZero, high energy physics, generates 1 TB a day How do you deal with that much data? [1,2]

3. Data in the cloud Storing the data o Bigtable, S3, NoSQL, etc Processing the data o MapReduce, Hadoop, etc

4. Good data management in the cloud Availability o Accessible in cases of partial network failure or datacenter failure Scalability o Support for massive database sizes - spread across many servers Elasticity o Scaling up and scaling down Performance o Efficient system storage utilization Multitenancy o Many applications on the same hardware

5. Good data management (continued) Load and Tenant Balancing o Moving load between servers Fault Tolerance o Tolerating network or hardware failures Running in heterogeneous environment o Dealing with hardware degredation Flexible query interface o Providing ways to access both SQL and non-SQL languages

6. Overarching Themes Frustration with ACID on the cloud o (Atomicity, consistency, isolation, durability) Hard to maintain ACID guarantees with data replication over large geographic distances [1] o Consistency, Availability, Tolerance to partitions, choose 2 Rise of NoSQL (a misnomer) [2] o Eventually consistent can be okay, some ACID properties are relaxed or left to application developers

7. Investigating 3 Systems Bigtable (Google) o And quick look at MapReduce Amazon:S3/SimpleDB Open source NoSQL alternatives: o Cassandra (key-value) o MongoDB (document)

8. Bigtable Distributed storage designed to scale to petabyte size databases spread across thousands of servers [1] Used extensively by Google Not fully relational o "Sparse, distributed, persistent multidimentional sorted map" [1] Uses Google File System (GFS) under the hood Index using row keys o Tablet = range of row keys, used for load balancing

9. Bigtable Diagram [2]

10. Bigtable GFS o SSTable  Provides a persistent immutable ordered map o Chubby provides locking mechanism  Ensures single master  Location of bigtable data  Storing schema information and access control lists Each Bigtable is allocated to one master, and many multiple tablet servers o Master assigns tablets to different tablet servers, dynamically based on server load o Tablets handle read-write

11. MapReduce Introduced by Google in 2004 [1] Often used to operate on Bigtable data [1] A means to process large amounts of data in a distributed environment in a highly parallelized manner

12. MapReduce Steps 1.Input files split into M pieces, multiple copies of program started on cluster 2.One copy is master, M map tasks, R reduce tasks assigned to idle workers 3.Worker reads file split contents, passes to map function - results buffered in memory 4.Buffered results written to local disk periodically, partitioned into R regions by partitioning function, locations passed to master

13. MapReduce (continued) 5.Reduce worker notified about location, reads buffered data from map workers, sorts so that same keys are grouped together 6.Reduce worker passes key and intermediate values to Reduce function, output is appended to final output file 7.After all map and reduce tasks completed, master wakes up user program

14. S3 - Simple Storage Service "Infinite" store for objects of variable size [1] Organized in 2 levels o Buckets  Like folders, you can save any number of objects in them o Objects  Byte container (up to 5 GB) and metadata (up to 2KB) Limited search o Single bucket, name only

15. SimpleDB Organized into domains (tables) where you can insert data, get data, or run queries [1] Each domain has items which are descibed by attribute name/value pairs No schema API Access- o CreateDomain, DeleteDomain, PutAttributes, DeleteAttributes, GetAttributes, and Select Meant for fast reads Keeps multiple copies of the domains

16. NoSQL What does this mean? o More about relaxing ACID than being "No" SQL [2] Lots of open source NoSQL systems o Zynga was big on NoSQL Why to use them? o Excellent elasticity o Flexible data models - often schema-less o CHEAP (relative to RDBMS) o (if you have lots of frequent and small writes)

17. Types of NoSQL Key-value o Redis, Cassandra, etc. Document store o CouchDB, mongoDB, etc Graph dbs, object stores o Won't go into these much

18. Cassandra Highly scalable, eventually consistent, distributed, structured, key-value store [1] Open sourced by Facebook (2008) [1] ColumnFamily based o Column is a tuple of {key, value, timestamp} o ColumnFamilies contain many columns, all referenced by row-key Kind of like a hybrid of Dynamo and Bigtable [1]

19. MongoDB Document-oriented o High input read/write o High availability o Scalability o Flexible query language

20. References [1] Sakr, S., Liu, A., Batista, D.M., Alomari, M., A Survey of Large Scale Data Management Approaches in Cloud Environments, IEEE Communications, 2011. [2] Cloud Computing: Theory and Practice (our lecture notes) [3] http://www.mongodb.org/display/DOCS/Intro duction