MapReduce Algorithm Design

Contents Combiner and in-mapper combining Complex keys and values Relative frequency Secondary Sorting

Contents Combiner and in-mapper combining Complex keys and values Relative frequency Secondary Sorting

Combiner and in-mapper combining Purpose Carry out local aggregation before shuffle and sort phase Reduce the communication volume between map and reduce stages

Combiner example (word count) Use reducer as combiner Integer addition is both associative and commutative

MapReduce with Combiner

MapReduce with combiner map: (k1, v1) -> [(k2, v2)] combine: (k2, [v2]) -> [(k2, v2)] reduce: (k2, [v2]) -> [(k3, v3)] The combiner input and output key-value types must match the mapper output key-value type

Combiner is an optimization, not a requirement Combiner is optional A particular implementation of MapReduce framework may choose to execute the combine method many times or none Calling the combine method zero, one, or many times should produce the same output from the reducer The correctness of the MapReduce program should not rely on the assumption that the combiner is always carried out

In-method combining

Use Java Map to implement AssociativeArray A Map is an object that maps keys to values A map cannot contain duplicate keys Each key can map to at most one value The Java platform contains three general-purpose Map implementations HashMap, TreeMap, and LinkedHashMap Basic operations boolean containsKey(Object key) get(Object key) put(K key, V value) keySet() Returns a Set view of the keys contained in this map entrySet() Returns a Set view of the mappings contained in this map.

In-mapper combining

Demo and Some online resource Demo on in-method and in-mapper combining Online resource https://vangjee.wordpress.com/2012/03/07/the-in-mapper-combining- design-pattern-for-mapreduce-programming codingjunkie.net Blog by Bill Bejeck, who provides a lot of source code on MapReduce in Hadoop

Contents Combiner and in-mapper combining Complex keys and values Relative frequency Secondary Sorting

Complex keys and values Both keys and values can be complex data structures Pairs Stripes Serialization and deserialization for complex structures After map stage, structures need to be serialized to be written to storage Complex keys and values need to be de-serialized at the reducer side

Motivation An example to compute the mean of value

With combiner

Revised mapper

In-mapper combining

A running example Build word co-occurrence matrices for large corpora Build the word co-occurrence matrix for all the works by Shakespeare Co-occurrence within a specific context A sentence A paragraph A document A certain window of m words

Pairs For each map task For each unique pair (a, b), emit [pair (a, b), 1] Use combiner or in-mapper combining to reduce the volume of intermediate pairs Reducers sum up counts associated with these pairs

Pairs approach

Stripes Idea: group together pairs into an associative array Mappers emit [word, associate array] Reducers perform element-wise sum of associative arrays (a, b) → 1 (a, c) → 2 (a, d) → 5 (a, e) → 3 (a, f) → 2 a → { b: 1, c: 2, d: 5, e: 3, f: 2 } a → { b: 1, d: 5, e: 3 } a → { b: 1, c: 2, d: 2, f: 2 } a → { b: 2, c: 2, d: 7, e: 3, f: 2 } +

Stripes approach

Contents Combiner and in-mapper combining Complex keys and values Relative frequency Secondary Sorting

Relative frequency What proportion of the time does B appear in the context A? Whenever there is a co-occurrence of (A,*), for what percentage will * be B The total count of co-occurrence of (A,*) is called marginal (A, *) → 32 Reducer holds this value in memory (a, b1) → 3 (a, b2) → 12 (a, b3) → 7 (a, b4) → 1 (a, b5) → 4 (a, b6) → 5 (a, b1) → 3 / 32 (a, b2) → 12 / 32 (a, b3) → 7 / 32 (a, b4) → 1 / 32 (a, b5) → 4 / 32 (a, b6) → 5 / 32

Relative frequency with stripes It is easy One pass to compute (A, *) Another pass to directly compute f(B|A) May have the scalability issue for really large data The final associative array holds all the neighbors and their co-occurrence counts with the word A

Relative frequency with pairs Must emit extra (A, *) for every B (B can be any word) in mapper Must make sure all pairs of (A, *) and (A, B) get sent to same reducer (use partitioner) Must make sure (A, *) comes first (define sort order) Must hold state in reducer across different key-value pairs This pattern is called “order inversion” For the mapper, the example is as follows. Alice go to the wonderland to have fun For the relative frequency, use the following example ((a, *), [12,4,3]) ((a, b), [1,9]) ((a, c), [2,3]) ((a, d), [4]) ((b, *), [7,12,6]) ((b, b), [3,6]) ((b, c), [2]) ((b, d), [13,1])

Contents Combiner and in-mapper combining Complex keys and values Relative frequency Secondary Sorting

Secondary Sorting A motivating example The readings of m sensors are recorded over the time (m1, t1, r80521) (m1, t2, r21823) …… (m2, t1, r14209) (m2, t2, r66508) (m3, t1, r76042) (m3, t2, r98347) (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) …… (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347)

First approach (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) …… (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347) (m1, (t1, r80521)) (m2, (t1, r14209)) (m3, (t1, r76042)) …… (m1, (t2, r21823)) (m2, (t2, r66508)) (m3, (t2, r98347)) However, Hadoop MapReduce sorts intermediate pairs by key Values can be arbitrarily ordered E.g., (m1, [(t100, r23456), (t2, r21823),…,(t234, r34870)]) Buffer values in memory, then sort Is there issue with this approach?

Second approach Value-to-key conversion Move part of the value into the intermediate key to form a composite key Composite key: (m, t) Intermediate pair: ((m, t), r) Let execution framework do the sorting First sort by the sensor id, i.e., m (the left element in the key) Then sort by the timestamp, i.e., t (the right element in the key) Implement the custom partitioner All pairs with the same sensor shuffled to the same reducer ((m1, t1), r80521) ((m1, t2), r21823) ((m1, t3), r149625) ……