By Jeff Dean & Sanjay Ghemawat Google Inc. OSDI 2004 Presented by : Mohit Deopujari

4/9/2013 Map Reduce 2  Cluster Architecture  Distributed File Systems  Coping with Node Failure  Large-Scale Indexing

4/9/2013 Map Reduce 3  A new programming model for processing and generating large data sets  A new abstraction that enables simple computations to be parallelized while hiding away the details  Map Reduce  Map and Reduce are two primitives directly inspired by Functional languages such as LISP  This Functional model enables massive parallelization and fault-tolerance by re-execution

4/9/2013 Map Reduce 4  Input consists of key/value pairs, Final output is also key/value pairs  Computation is expressed as two functions : Map and Reduce  Map :-  Written by user  Input is key/value pairs, Output is intermediate key/value pairs  Groups together all intermediate values associated with same intermediate key I and passes onto Reduce  Reduce :-  Also written by user  Accepts intermediate key I and set of values for that key  Merges together these values to form a possibly smaller set of values  Zero or one output value produced per invocation  An iterator is used to feed the intermediate values to the function thus allowing us to handle lists too large to fit in memory

4/9/2013 Map Reduce 5  map(String key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1");  reduce(String key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result));

4/9/2013 Map Reduce 6  the input keys and values are drawn from a different domain than the output keys and values  the intermediate keys and values are from the same domain as the output keys and values.

4/9/2013 Map Reduce 7 1.MapReduce library in the user program splits the input file into M chunks (16 to 64MB each) and then starts up many copies of the program on a cluster of machines. 2.A special copy, the Master, assigns each of the other copies (workers) a Map or a Reduce task. 3.Each worker assigned with the Map task parses the key/value pair from the corresponding chunk of input and passes it to the Map function. Intermediate key/value pairs are produced and buffered in memory. 4.Periodically, a partitioning function partitions these pairs into R regions and writes to local disk. The locations of these pairs on disk is passed to the master. 5.When a reduce worker is notified of these locations by a master, the worker sorts the intermediate keys and groups the values by key.

4/9/2013 Map Reduce 8 6. Then, the reduce worker iterates over each unique key and passes on the list of corresponding values to the Reduce function. The output of the Reduce function for that partition is appended to the final output. 7. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code.

4/9/2013 Map Reduce 9  Master Failure:  Highly unlikely since there is only a single master  If there is a failure, whole task is aborted and can be restarted  Worker Failure:  If a Map worker fails, it is detected by Master and that node’s map tasks (even if completed) are set to idle and made ready to be rescheduled on other machines  Reduce workers are notified of the new locations from which to pick up data  If a Reduce worker fails, its tasks are set to idle and made ready to be rescheduled on other machines

4/9/2013 Map Reduce 10  Network bandwidth is conserved because of the fact that input data resides on the local disks of the machines that make up the data cluster  The Master knows location of input files. It schedules a map task on a machine that has a copy of the corresponding input data  Thus, on large MapReduce operations, the input data is read locally and bandwidth consumption is minimal

4/9/2013 Map Reduce 11

4/9/2013 Map Reduce 12 distributed grep distributed sort web link-graph reversal term-vector per hostweb access log stats inverted index construction document clusteringmachine learning statistical machine translation...

4/9/2013 Map Reduce 13  Simple and powerful interface enables automatic parallelization and distribution of large scale computations  High performance achieved on large clusters of commodity CPUs (cheap hardware)  Fair fault-tolerance and capability to handle failures and still make progress  Locality of input data conserves bandwidth and helps speed up tasks  Wide variety of applications  Was used to completely rewrite the Google Indexing code in turn making it easier to read, manage while improving performance and efficiency

4/9/2013 Map Reduce 14  Debugging is tricky since actual computation occurs in a distributed environment on thousand of machines with work assignments being made dynamically  In case of Map Node failure, the Map tasks need to be started all over again since the intermediate output lies in the failed machine

4/9/2013 Map Reduce 15  MapReduce has been widely used at Google for many tasks primarily due to these reasons:  The model is easy to use, even for programmers without experience with parallel and distributed systems, since it hides the details of parallelization, fault-tolerance, locality optimization, and load balancing  A large variety of problems are easily expressible as MapReduce computations  Easy and efficient scalability

4/9/2013 Map Reduce 16  Use of a restricted programming model (based on functional model) enables easy parallelization and fault-tolerance through re-execution  Network bandwidth is scarce hence many optimizations need to be built to minimize wastage  redundant execution can be used to reduce the impact of slow machines, and to handle machine failures and data loss