1. 5/7/2019
Map Reduce
Map reduce

2. Word Count Example We have a large file of words, one word to a line
5/7/2019
Map Reduce
Word Count Example
We have a large file of words, one word to a line
Count the number of times each distinct word appears in the file
Sample application: analyze web server logs to find popular URLs

3. 5/7/2019
Map Reduce
Word Count (2)
Case 1: Entire file fits in memory

4. Word Count (2) Case 1: Entire file fits in memory
5/7/2019
Map Reduce
Word Count (2)
Case 1: Entire file fits in memory
Case 2: File too large for mem, but all <word, count> pairs fit in mem

5. Word Count (2) sort datafile | uniq –c
5/7/2019
Map Reduce
Word Count (2)
Case 1: Entire file fits in memory
Case 2: File too large for mem, but all <word, count> pairs fit in mem
Case 3: File on disk, too many distinct words to fit in memory
sort datafile | uniq –c

6. 5/7/2019
Map Reduce
Word Count (3)
A large corpus of documents, sharded across many disks in many machines
Machines, disks, networks can fail
Motivation for Google’s MapReduce

7. 5/7/2019
Map Reduce
Input: A set of files C N B C F O X …

8. Map: Generate Word Count Per File
5/7/2019
Map Reduce
Map: Generate Word Count Per File C N B C F O X …
C:1
N:2
B:2
C:1
F:1
O:1
X:1

9. Partition (Optional) C N B C F O X … C:1 N:2 B:2 C:1 F:1 O:1 X:1 C:1
5/7/2019
Map Reduce
Partition (Optional) C N B C F O X …
C:1
N:2
B:2
C:1
F:1
O:1
X:1
C:1
B:2
F:1
N:2
O:1
X:1
C:1

10. Reduce C N B C F O X … C:1 N:2 B:2 C:1 F:1 O:1 X:1 C:1 B:2 F:1 N:2 O:1
5/7/2019
Map Reduce
Reduce C N B C F O X …
C:1
N:2
B:2
C:1
F:1
O:1
X:1
C:1
B:2
F:1
N:2
O:1
X:1
C:1
C:2
B:2
F:1
N:2
O:1
X:1

11. MapReduce Input: a set of key/value pairs User supplies two functions:
5/7/2019
Map Reduce
MapReduce
Input: a set of key/value pairs
User supplies two functions:
map(k,v)  list(k1,v1)
reduce(k1, list(v1))  v2
(k1,v1) is an intermediate key/value pair
Output is the set of (k1,v2) pairs

12. Word Count using MapReduce
5/7/2019
Map Reduce
Word Count using MapReduce
map(key, value):
// key: document name; value: text of document
for each word w in value:
emit(w, 1)
reduce(key, values):
// key: a word; value: an iterator over counts
result = 0
for each count v in values:
result += v
emit(result)

13. Map and Reduce vs MapReduce
5/7/2019
Map Reduce
Map and Reduce vs MapReduce
The map and reduce operations in MapReduce are inspired by similar operations in functional programming

14. Map Given a function f : (A) => B A collection a: A[]
5/7/2019
Map Reduce
Map
Given a function f : (A) => B
A collection a: A[]
Generates a collection b: B[], where B[i] = f( A[i] )
Parallel.For, Paralle.ForEach
Where each loop iteration is independent A f B

15. Reduce Given a function f: (A, B) => B A collection a: A[]
5/7/2019
Map Reduce
Reduce
Given a function f: (A, B) => B
A collection a: A[]
An initial value b0: B
Generate a final value b: B
Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) A f b b0

16. Relationship to SQL Implementing word count in SQL
5/7/2019
Map Reduce
Relationship to SQL
Implementing word count in SQL
SELECT word Count(*) as wordCount
FROM files
GROUP BY word;
// where files is a (distributed)
// relation <name, posn, word>

17. Signs of a Good Abstraction
5/7/2019
Map Reduce
Signs of a Good Abstraction
Hides important details
But not too much
Simple for lay programmers to use
Not necessarily general
But not very restricted
Can be application/domain specific
Allows efficient implementations
Automatic optimizations
Manual optimizations (by experts)

18. Distributed Execution Overview
5/7/2019
Map Reduce
Distributed Execution Overview
User
Program
Worker
Master
fork
assign
map
reduce
Split 0
Split 1
Split 2
Input Data
local
write
Output
File 0
File 1
write
read
remote
read,
sort

19. Data flow Input, final output are stored on a distributed file system
5/7/2019
Map Reduce
Data flow
Input, final output are stored on a distributed file system
Scheduler tries to schedule map tasks “close” to physical storage location of input data
Intermediate results are stored on local FS of map and reduce workers
Output is often input to another map reduce task

20. Coordination Master data structures
5/7/2019
Map Reduce
Coordination
Master data structures
Task status: (idle, in-progress, completed)
Idle tasks get scheduled as workers become available
When a map task completes, it sends the master the location and sizes of its R intermediate files, one for each reducer
Master pushes this info to reducers
Master pings workers periodically to detect failures

21. Failures Map worker failure Reduce worker failure Master failure
5/7/2019
Map Reduce
Failures
Map worker failure
Map tasks completed or in-progress at worker are reset to idle
Reduce workers are notified when task is rescheduled on another worker
Reduce worker failure
Only in-progress tasks are reset to idle
Master failure
MapReduce task is aborted and client is notified

22. How many Map and Reduce jobs?
5/7/2019
Map Reduce
How many Map and Reduce jobs?
M map tasks, R reduce tasks
Rule of thumb:
Make M and R much larger than the number of nodes in cluster
One DFS chunk per map is common
Improves dynamic load balancing and speeds recovery from worker failure
Usually R is smaller than M, because output is spread across R files

23. 5/7/2019
Map Reduce
Combiners
Often a map task will produce many pairs of the form (k,v1), (k,v2), … for the same key k
E.g., popular words in Word Count
Can save network time by pre-aggregating at mapper
combine(k1, list(v1))  v2
Usually same as reduce function
Works only if reduce function is commutative and associative

24. 5/7/2019
Map Reduce
Partition Function
Inputs to map tasks are created by contiguous splits of input file
For reduce, we need to ensure that records with the same intermediate key end up at the same worker
System uses a default partition function e.g., hash(key) mod R
Sometimes useful to override
E.g., hash(hostname(URL)) mod R ensures URLs from a host end up in the same output file

25. 5/7/2019
Map Reduce
Avoiding Stragglers
A slow running task (straggler) can prolong overall execution
Overloaded machines
Slow disk
Kill stragglers
Fork redundant tasks and take the first

26. 5/7/2019
Map Reduce
Example: Sorting

27. Example: Database Join
5/7/2019
Map Reduce
Example: Database Join

28. Can Mappers Push instead of Reducers Pulling Data ?
5/7/2019
Map Reduce
Can Mappers Push instead of Reducers Pulling Data ?