1. Introduction to Data Center Computing Derek Murray October 2010

2. What we’ll cover Techniques for handling “big data” –Distributed storage –Distributed computation Focus on recent papers describing real systems

3. Example: web search WWW CrawlingIndexingQuerying

4. A system architecture? Computers Network Storage

5. Data Center architecture Server Rack switch Server Rack switch Server Rack switch Core switch

6. Distributed storage High volume of data High volume of read/write requests Fault tolerance

7. Brewer’s CAP theorem (2000) Consistency Partition Tolerance Availability

8. The Google file system (2003) GFS Master Chunk server Client

9. Dynamo (2007) Client

10. Distributed computation Parallel distributed processing Single Program, Multiple Data (SPMD) Fault tolerance Applications

11. Task farming Master Worker Storage

12. MapReduce (2004)

13. Dryad (2007) Arbitrary directed acyclic graph (DAG) Vertices and channels Topological ordering

14. DryadLINQ (2008) Language Integrated Query (LINQ) var table = PartitionedTable.Get (“…”); var result = from x in table select x * x; int sumSquares = result.Sum();

15. Scheduling issues Heterogeneous performance Sharing a cluster fairly Data locality

16. Percolator (2010) Built on Google BigTable Transactions via snapshot isolation Per-column notifications (triggers)

17. Skywriting and C IEL (2010) Universal distributed execution engine Script language for distributed programs Opportunities for student projects…

18. References Storage –Ghemawat et al., “The Google File System”, Proceedings of SOSP 2003 –DeCandia et al., “Dynamo: Amazon’s Highly-Available Key-value Store”, Proceedings of SOSP 2007 Computation –Dean and Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, Proceedings of OSDI 2004 –Isard et al., “Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks”, Proceedings of EuroSys 2007 –Yu et al., “DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language”, Proceedings of OSDI 2008 –Olston et al., “Pig Latin: A Not-So-Foreign Language for Data Processing”, Proceedings of SIGMOD 2008 –Murray and Hand, “Scripting the Cloud with Skywriting”, Proceedings of HotCloud 2010 Scheduling –Zaharia et al., “Improving MapReduce Performance in Heterogeneous Environments”, Proceedings of OSDI 2008 –Isard et al., “Quincy: Fair Scheduling for Distributed Computing Clusters”, Proceedings of SOSP 2009 –Zaharia et al., “Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling”, Proceedings of EuroSys 2010 Transactions –Peng and Dabek, “Large-Scale Incremental Processing using Distributed Transactions and Notifications”, Proceedings of OSDI 2010

19. Conclusions Data centers achieve high performance with commodity parts Efficient storage requires application- specific trade-offs Data-parallelism simplifies distributed computation on the data

20. Questions Now or after the lecture –Email Derek.Murray@cl.cam.ac.uk –Web http://www.cl.cam.ac.uk/~dgm36/