Ch. 3 Lin and Dyer’s text Pages (39-69)

Word count:  Local aggregation as opposed to external combiner that is NOT guaranteed by the Hadoop framework ◦ May not work all the time: what if wanted word “mean” instead of word “count”: may have to adjust types at the output of map Word co-occurrence (matrix) ◦ Very important since many (many) problems are expressed and solved using matrices ◦ Pairs and stripes approaches ◦ And comparison of these two methods P.60 (56)

 First version simplistic counts  Then “relative frequency” instead of counts ◦ What is relative frequency? Instead of absolute counts ◦ f(w i /w j ) = N(w i,w j )/∑ w’ (w i, w’) ◦ For example, if word “dog” co-occurred with “food” 23 times, and “dog” co-occurred with all words 460 times, then relative frequency is 23/460 = 1/20 = 0.05 ◦ Also the 460 could come from many mappers, many documents over the entire corpus. ◦ These co-occurrences from every mapper are delivered to “corresponding reducer” with a special key ◦ This is delivered as special key item as the first pair ◦ The magic is that reducer processes

KeyValueReducer operation/compute Result (dog,*)[200,350,650] One per mapper with combiner Marginal∑ w’ (wi, w’)=1200 (dog, bark)60Relative frequency (dog, cat)12Relative frequency (dog, food)600Relative frequency ….… At the reducer: Blue: reducer1/Orange: reducer 2 KeyValueReducer operation Result (tiger,*)[100,300,600]Compute marginal ∑ w’ (wi, w’)=1000 (tiger, cub)10Compute R.freq (tiger, hunt)100Compute R.freq (tiger, prey)200Compute R.freq ….

4 different reducers

 Emitting a special key-value pair for each co- occurring word pair in the mapper to capture its contribution to the marginal.  Controlling the sort order of the intermediate key so that the key-value pairs representing the marginal contributions are processed by the reducer before any of the pairs representing the joint word co- occurrence counts.  Defining a custom partitioner to ensure that all pairs with the same left word are shuffled to the same reducer.  Preserving state across multiple keys in the reducer to first compute the marginal based on the special key-value pairs and then dividing the joint counts by the marginal to arrive at the relative frequencies.

all delivered to the same reducer.. What can you do with this? Reducer can “middle(left’s value, right’s value) “  Some more: You can do anything you want for function… “KEY.operation” on “VALUE.data” Therein lies the power of MR.

 Text word count  Text co-occurrence  pairs and stripes  Numerical data processing with most math functions  How about sensor data?  Consider m sensors, sending out readings r x at various times t y resulting large volume of data of the format:  (t1;m1; r80521)  (t1;m2; r14209)  (t1;m3; r76042)  :::  (t2;m1; r21823)  (t2;m2; r66508)  (t2;m3; r98347)  Suppose you wanted to know the readings by the sensors, how could process the above to get that info?  Use MR to do that… etc.  But what if wanted that sorted by time t that is a part of the value?

 Solution 1: Let the reducer do the in-memory sorting  memory bottle neck  Solution 2: Move value to be sorted to the key, and modify the shuffler and partitioner  In the later, the “secondary sorting” is left to the framework and it excels in doing this anyway.  So solution 2 is a preferred approach.  Lesson: Let the framework do what it is good at and don’t try to move into your code… in the latter you will be regressing to the “usual” coding practices and ensuing disadvantages

 Reduce-side join is intuitive but inefficient  Map-side join requires simple merge of respective input files and appropriate sort by the MR framework  In-memory joins can be done for smaller data.  We will NOT discuss this in detail since there are other solutions such as Hive, Hbase available for warehouse data. We will look into these later.