MapReduce Simplified Data Processing on Large Clusters Google, Inc. Presented by Prasad Raghavendra

Introduction Model for processing large data sets. Contains Map and Reduce functions. Runs on a large cluster of machines. A lot of MapReduce programs are executed on Google’s cluster everyday.

Motivation  Very large data sets need to be processed. - The whole Web, billions of Pages  Lots of machines - Use them efficiently.

Processing of Large Data Sets For example: - Counting access frequency to URLs: Input: list(RequestURL) Output: list(RequestURL, total_number) - Distributed Grep - Distributed Sort

Programming model Input & Output: each a set of key/value pairs Programmer specifies two functions: map (in_key, in_value) -> list(out_key, intermediate_value) Name comes from map function in LISP Ex. (map 'list #’+ '(1 2 3) '(1 2 3)) => (2 4 6) -Processes input key/value pair -Produces set of intermediate pairs m ap(document, content) { for each word in content emit(word, “1”) }

reduce (out_key, list(intermediate_value)) -> list(out_value) Name comes from reduce function in LISP Ex. (reduce #’+ '( )) => 15 - Combines all intermediate values for a particular key - Produces a set of merged output values (usually just one) reduce(word, values) { result = 0; for each value in values result += value emitString(w, result) }

Example The problem of counting the number of occurrences of each word in a large collection of documents. Page 1: the weather is good Page 2: today is good Page 3: good weather is good

Map output Worker 1: (the 1), (weather 1), (is 1), (good 1). Worker 2: (today 1), (is 1), (good 1). Worker 3: (good 1), (weather 1), (is 1), (good 1).

Reduce Input Worker 1: (the 1) Worker 2: (is 1), (is 1), (is 1) Worker 3: (weather 1), (weather 1) Worker 4: (today 1) Worker 5: (good 1),(good 1), (good 1), (good 1)

Reduce Output Worker 1: (the 1) Worker 2: (is 3) Worker 3: (weather 2) Worker 4: (today 1) Worker 5: (good 4)

Example 2

Implementation

Flow of MapReduce Operation The MapReduce library in the user program splits the input files into M pieces(16,64 MB). One of the copies of the program is special. The master. The rest are workers. A worker who is assigned a map task parses key/value pairs out of the input data. Periodically, the buffered pairs are written to local disk. When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data. The output of the Reduce function is appended to a final output file. When all map tasks and reduce tasks have been completed, the master wakes up the user program.

Problem: Stragglers Often some machines are late in their replies - slow disk, overloaded, etc Approach: - when only few tasks left to execute, start backup tasks - a task completes when either primary or backup completes task Performance: - without backup, sort (->) takes 44% longer

Partition Function Defines which worker processes which keys - default: hash(key2) mod R Other partition functions useful: - sort: prefix of k bytes of line - idea: based on known/sampled distribution of key2 to evenly distribute processed keys

Combiner Function Problem: intermediate results can be quite verbose e.g., (“the”, 1) could occur many times in previous example Approach: perform a local reduction before writing intermediate results typically, combiner same function as reduce func This will reduce the run-time because less writing to disk and across the network

Performance Scan 10^ byte records to extract records matching a rare pattern (92K matching records) : 150 seconds. Sort 10^ byte records (modeled after TeraSort benchmark) : normal 839 seconds.

Fault Tolerance Crash of worker all - even finished - tasks are redone Crash of leader crash of leader process -> restart process with checkpoint crash of leader machine-> unlikely - restart computation redo computation

Conclusion MapReduce has proven to be a useful abstraction Easy to use Very large variety of problems are easily expressible as MapReduce computations Greatly simplifies large-scale computations at Google

Questions?

Thank You Thank You