1. Paula Ta-Shma, IBM Haifa Research 1 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Big Data and Map Reduce Paula Ta-Shma IBM Haifa Research Storage Systems 1/5/2013

2. Paula Ta-Shma, IBM Haifa Research 2 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Outline  Historical Context behind Map Reduce  What is Big Data ?  The Map Reduce Framework  Connections with Storage Cloud

3. Paula Ta-Shma, IBM Haifa Research 3 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Historical Context  Relational Database Management Systems (RDBMS) –Researched in 70s, products in 80s and beyond –Relational (tabular) data model –Query Language : SQL -Efficient Query Processing: Indexing, Query Evaluation Strategies –Transactions, Consistency –Concurrency Control –Security and Authorization –Can be implemented on top of file systems -Provide higher level of abstraction and functionality than file systems  Example Use Cases –Banking, Stock trading, Personnel Management, Inventory Management, Manfuacturing Data, etc. –The list is very long SELECT Name FROM Accounts GROUP BY Name HAVING SUM(Balance) < 0 NameBalance ($) Bob5000.00 Alice-389.27 Fred-800.00 Alice2980000.00 Accounts

4. Paula Ta-Shma, IBM Haifa Research 4 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Historical Context Cont.  Business Intelligence –Extract value from large amounts of data –Banking use case example -Identify and actively retain and pursue profitable customers -Analyze the performance of sales personnel, tellers and account managers -etc. –Massive query processing to analyze data across multiple dimensions -Requires read access to large amounts of data -Typically long running queries, can interfere with transactions –Work on a snapshot of data -Deployed as physically separate Data Warehousing systems -Mission critical -Data warehousing products in early 90s

5. Paula Ta-Shma, IBM Haifa Research 5 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom New Requirements in Internet Era  Massive amounts of data  Unstructured (e.g. text) and semi-structured data (e.g. XML)  Analysis capabilities beyond what is possible in SQL  LOW COST $$$Capital ExpensesOperational Expenses HardwareUse commodity hardware, scale out instead of scale up. Make it easy to manage hardware which will fail often. Treat failure case as the norm, automatic failover. SoftwareDBMS software is complex and expensive, transactions, concurrency control etc. not needed for many tasks Make it easy to write ‘queries’ on a distributed infrastructure.

6. Paula Ta-Shma, IBM Haifa Research 6 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Map Reduce  Invented by Google –Inspired by functional programming languages map and reduce functions –Seminal paper: Dean, Jeffrey & Ghemawat, Sanjay (OSDI 2004), "MapReduce: Simplified Data Processing on Large Clusters"  Used at Google to completely regenerate Google's index of the World Wide Web. –It replaced the old ad hoc programs that updated the index and ran the various analyses.  Uses: –distributed pattern-based searching, distributed sorting, web link-graph reversal, term- vector per host, web access log stats, inverted index construction, document clustering, machine learning, statistical machine translation  Hadoop: –Open source implementation which matches Google’s specifications

7. Paula Ta-Shma, IBM Haifa Research 7 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Source: IBM InfoSphere BigInsights slides, by Bruce Brown https://www-950.ibm.com/events/wwe/grp/grp004.nsf/vLookupPDFs/Bruce%20Brown%20-%20BigInsights-1-16-12-external/$file/Bruce%20Brown%20-%20BigInsights-1-16-12-external.pdf

8. Paula Ta-Shma, IBM Haifa Research 8 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Source: IBM InfoSphere BigInsights slides, by Bruce Brown https://www-950.ibm.com/events/wwe/grp/grp004.nsf/vLookupPDFs/Bruce%20Brown%20-%20BigInsights-1-16-12-external/$file/Bruce%20Brown%20-%20BigInsights-1-16-12-external.pdf

9. Paula Ta-Shma, IBM Haifa Research 9 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Map Reduce In Detail  Map Reduce material taken from Distributed Systems Course, MapReduce lecture by Paul Krzyzanowski –http://www.seas.gwu.edu/~gparmer/courses/f12_3411/distrib-5-mapreduce.pdfhttp://www.seas.gwu.edu/~gparmer/courses/f12_3411/distrib-5-mapreduce.pdf

10. Paula Ta-Shma, IBM Haifa Research 10 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom HDFS Architecture Source http://hadoop.apache.org/docs/r1.0.4/hdfs_design.htmlhttp://hadoop.apache.org/docs/r1.0.4/hdfs_design.html

11. Paula Ta-Shma, IBM Haifa Research 11 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Integrating Hadoop with Object Storage  Implement Hadoop FileSystem API  Leave MapReduce framework unchanged –=> no changes needed for user applications –=> work with Hadoop based technologies -Hive, Pig Latin, HBase, Jaql, and others Hadoop FileSystem API (create,open,close,read,write,seek,get block locations…) Hadoop Distributed File System (HDFS) S3FileSystem CDMI FileSystem Hadoop Map Reduce invokes implements Application HBase, Jaql,…

12. Paula Ta-Shma, IBM Haifa Research 12 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom Amazon Elastic Map Reduce Source: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-what-is-emr.htmlhttp://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-what-is-emr.html

13. Paula Ta-Shma, IBM Haifa Research 13 “Advanced Topics on Storage Systems” - Spring 2013, Tel-Aviv University http://www.eng.tau.ac.il/semcom The End