Map Reduce

History 2003 - Google releases the Google File System 2004 - Google publishes white paper detailing the MapReduce paradigm 2006 - Yahoo creates Hadoop (named after creator's son's stuffed elephant) 2011 - Hadoop is scaled to tens of thousands of nodes 2016 - Hadoop market > $800 million, big data market > $23 billion http://grigory.us/big-data-class.html

MapReduce Based on functional programming (LISP), it utilizes parallel operations to make processing huge amounts of data feasible. This runtime system handles many of the messy engineering aspects of parallelization, fault tolerance, data distribution, load balancing and management of task communication. Motivation: implementing hundreds of special-purpose computations on large datasets Computing inverted indexes from Web content collected via Web crawling (which pages point at this page) Extracting statistics from Web logs (frequency distribution of search topics, by region, by type of user) Easy to conceptualize, hard to implement because of size of dataset.

MapReduce Part 2 An underlying model of the data is assumed. This model treats an object of interest in the form of a unique key that has associated content or value (key-value pair). Many computations can be expressed as applying a map operation to each logical record that produces a set of intermediate key-value pairs. Then apply a reduce operation to all the values that share the same key (the purpose of sharing a key is to combine derived data). The ideas of MapReduce are not new (most are from the 1950's) but the implementation for distributed systems was. http://dme.rwth-aachen.de/de/research/projects/mapreduce

Example in Python Problem: We want the word count (number of particular word occurrences) for a document Map: for each word in the document emit the word-count pair def get_words(part_of_doc): for word in part_of_doc: yield word, 1 Reduce: combine the shared key and values def combine_counts(word, values): return word, sum(values) Note this is a different syntax than Project 6.

MapReduce Use Cases Distributed Grep: Grep looks for a text pattern in a document Map: Output a line if it matches the pattern Reduce: Identity function (output everything it is given) Reverse Web-Link Graph Purpose is to output (target URLs, source URLs) pairs for each link to a target page found in a page named source. Map: Output each link found Reduce: Concatenates the list of all source URLs associated with a given target URL and outputs (target, list of sources) Inverted Index Find all of the documents that contain specific words Map: Parses each document and outputs each (word, document_id) pair Reduce: Takes all the pairs of a given word, sorts them by document_id, emits a (word, list of document_ids)