1. The Map-Reduce Framework
Compiled by Mark Silberstein, using slides from
Dan Weld’s class at U. Washington, Yaniv Carmeli and some other

2. Big Data (From Wikipedia)
Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate
[…]
Big data "size" is a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data.

3. Motivation Large-Scale data processing
Want to process and deduct “new” information from a huge, mostly distributed, database
Want to use 1,000s of CPUs
But don’t want hassle of managing things

4. Distributed Computing Frameworks (1)
Hadoop: Basic MapReduce. Intermediate results in disk. Now Yarn and support DAG with Tez
Spark: Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.
Hive: SQL-like queries (HiveQL), which are implicitly converted into MapReduce, Tez, or Spark jobs.
Mahout: Machine learning algs. mostly on top of Hadoop
Tez™: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. Tez is being adopted by Hive™, Pig™ and other frameworks in the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools), to replace Hadoop™ MapReduce as the underlying execution engine.
Pregel: A System for Large-Scale Graph Processing (Google)
Dryad: Distributed Data-parallel Programs from Sequential Building Blocks (Microsoft)
Dremel: Interactive Analysis of Web-Scale Datasets (Google)
PowerDrill: Processing a Trillion Cells per Mouse Click (Google)
Peregrine: A Mapreduce Framework For Iterative And Pipelined Jobs

5. Distributed Computing Frameworks (2)
A Distributed Computing Framework provides:
Automatic parallelization & distribution
Fault tolerance
I/O scheduling
Monitoring & status updates
A MapReduce framework provides the above for any task that match its specific, yet generic enough, pattern

6. The Map-Reduce Paradigm
You are given a phonebook:
How many entries have the last name “Smith”?
Create a list of all last names, with the numbers of entries they appear in
How many entries have the first name “John”?
Create a list of all first names, with the numbers of entries they appear in
How can these tasks be distributed to many computers?

7. Map-Reduce Programming Model
Programming model from Lisp
(and other functional languages)‏
Many problems can be phrased this way
Easy to distribute across nodes
Nice retry/failure semantics

8. Map and Reduce in Lisp (Scheme)‏
Unary operator
(map fu list [list2 list3 …])‏
(reduce fb list [list2 list3 …])‏
(map square ‘( ))‏
( )‏ ⟹ ( )‏
(reduce + ‘( ))‏
(+ 16 (+ 9 (+ 4 1) ) )‏ ⟹ 30
(reduce + (map square (map – L1 L2)))‏
Binary operator

9. Map-Reduce ala Google map(key, val) is run on each item in set
emit (new-key, new-val) pairs
reduce(new-key, lisf of new-vals) is run for each unique new-key emitted by map()‏
emit final output
MapReduce: Simplified Data Processing on Large Clusters
Jeffrey Dean and Sanjay Ghemawat (Google, Inc.)

10. Semantics Assumptions on Map and Reduce operations
Map has no side effects
Reduce is
Associative: (a # b) # c = a # (b # c)
Commutative: a # b = b # a
Idempotent: The same call with the same parameters will produce the same results

11. Count words in docs Inputs: (url, content) pair for each URL
Outputs: (word, count) pair for each word that appears in the documents
map(key=url, val=content):
For each word in content, emit (word, “1”)‏ pair
reduce(key=word, values=unique-counts):
sum all “1”s in unique-counts list
emit (word, sum) pair

12. Count Illustrated see 1 bob 1 bob 1 david 1 run 1 run 1 see bob throw
map(key=url, val=content):
For each word in content, emit (word, “1”)‏ pair
reduce(key=word, values=unique-counts):
sum all “1”s in unique-counts list and emit (word, sum) pair
see 1
bob 1
run 1
bob 1
david 1
run 1
see 2
throw 1
see bob throw
Doc 1:
see david run
Doc 2:
see 1
david 1
throw 1

13. Grep Inputs: (file, regexp)‏ pair for each file and regular expression
Outputs: (regexp, file, #line) for each regexp, all the lines that match the regular expression
map(key=file, val=regexp):
If contents matches regexp, emit (regexp, {file-name, #line}) ‏
reduce(key=regexp, values={file-name, #line}):
emit (regexp, list of {file-name, #line})

14. Reverse Web-Link Graph
Inputs: (source-URL, page-content)‏ for each URL
Outputs: (URL, list of URLs that links to this URL) for each URL
map(key=source-URL, val=page-content):
For each link in page-content:
emit (target-URL, source-URL) pairs
reduce(key=target-URL, values=source-URL):
emit (target-URL, list of source-URLs) pairs

15. Inverted Index Inputs: (source-URL, page-content)‏ for each URL
Outputs: (word, list of pages where it appears) for each word that appears in the documents
map(key=source-URL, val=page-content):
For each word in page-content:
emit (word, source-URL) pair
reduce(key=word, values=source-URL):
emit (word, list of source-URLs) pairs

16. Map-Reduce Framework Input: a collection of keys and their values User
written
Map
function
Each input (k,v) mapped to set of intermediate key-value pairs
Sort all key-value
pairs by key
Each list of values is reduced to a single value (more or less) for that key
User
written
Reduce
function
Compile one list of intermediate
values for each key:(k, [v1,…,vn])‏

17. Parallelization How is this distributed? Map: Shuffle: Reduce:
Partition input key/value pairs into chunks
Run map() tasks in parallel
Shuffle:
After all map()s are complete
Consolidate all emitted values for each unique emitted key
Reduce:
Partition space of output map keys
Run reduce() in parallel
If map() or reduce() fails, re-execute!

18. Shuffle
map
reduce

19. Another example In a credit card company:
Inputs: credit card number -> bargain (cost, date, etc…) for each bargain
Outputs: The average bargain cost per month per card
(card, month, avg-cost)
map(key=card, val=bargain):
emit ({card, bargain.month}, bargain.cost)
reduce(key={card, month}, values=costs):
Find average cost
emit (card, month, average)

20. Implementation
A program forks a master process and many worker processes.
Input is partitioned into some number of splits.
Worker processes are assigned either to perform map on a split or reduce for some set of intermediate keys.

21. Distributed Execution Overview
User
Program
Worker
Master
fork
assign
map
reduce
read
local write
remote
read, sort
Output
File 0
File 1
write
Split 0
Split 1
Split 2
Input Data

22. Distributed Execution (2)
Map Task 1
Map Task 2
Map Task 3
Reduce Task 1
Reduce Task 2

23. Responsibility of the Master
Assign map and reduce tasks to workers
Check that no worker has died (because its processor failed)
Communicate results of map to the reduce tasks

24. Locality Master program divides up tasks based on location of data:
Tries to have map tasks on same machine as physical file data
Or at least same rack
map task inputs are divided into MB blocks
Same size as Google File System chunks

25. Communication from Map to Reduce
Select a number R of reduce tasks
Divide the intermediate keys into R groups, e.g. by hashing
Each map task creates, at its own processor, R files of intermediate key/value pairs, one for each reduce task

26. Fault Tolerance Master detects worker failures
Re-executes failed map tasks
Re-executes failed reduce tasks
Master notices particular input key/values cause crashes in map, and skips those values on re-execution.
Effect: Can work around bugs in third-party libraries

27. Strugglers No reduce can start until map is complete:
A single slow disk controller can rate-limit the whole process
Master redundantly executes “slow” map tasks
Uses results of first correctly-terminating replica

28. Task Granularity & Pipelining
Fine granularity tasks: #map tasks >> #machines
Minimizes time for fault recovery
Can pipeline shuffling with map execution
Hides latency of I/O wait time
Better dynamic load balancing
Often use 200,000 map & 5000 reduce tasks
Running on 2000 machines

29. Distributed Sort (1)
Sorting guarantees within each reduce partition over the order of processing the keys in each reduce task
Note: no order is guaranteed over the values of each key
How do we implement a distributed sort of all the data?

30. Distributed Sort (2)
Inputs: (document-name, document-content) for each document
Outputs: List of documents names ordered by the first word in the document
map(key=document-name, val=document-content):
For the first word in the document:
emit (word, document-name)‏ pair
reduce(key=word, values=document-names):
For each name in document-names:
emit name
Something is missing ....
Shuffle phase must allow distributed sort!

31. Other Refinements Compression of intermediate data Combiner
Useful for saving network bandwidth
Combiner
Perform reduce on local map results before shuffle

32. Yet another example Distributed single source shortest path
Input: (u, list-of-neighbors) for each edge in the graph, and s – source node
Output: (v, minimal-distance)
map(key=u, val={u-distance, neighbours}):
For each v in neighbours:
emit (v, u-distance + 1)‏ pair
reduce(key=u, values=distances):
emit (u, min(distances)) pair
“Graph Algorithms with MapReduce”:

33. Hadoop An open-source implementation of Map-Reduce in Java
Download from:
More about Hadoop in the tutorials...