1. Distributed and Parallel Processing Technology Chapter2. MapReduce
Sun Jo

2. Introduction MapReduce is a programming model for data processing.
Hadoop can run MapReduce programs written in various languages.
We shall look at the same program expressed in Java, Ruby, Python, and C++.

3. A Weather Dataset Program that mines weather data Data Format
Weather sensors collect data every hour at many locations across the globe
They gather a large volume of log data, which is good candidate for analysis with MapReduce
Data Format
Data from the National Climate Data Center(NCDC)
Stored using a line-oriented ASCII format, in which each line is a record

4. A Weather Dataset Data Format
Data files are organized by date and weather station.
There is a directory for each year from 1901 to 2001, each containing a gzipped file for each weather station with its readings for that year.
The whole dataset is made up of a large number of relatively small files since there are tens of thousands of weather station.
The data was preprocessed so that each year’s readings were concatenated into a single file.

5. Analyzing the Data with Unix Tools
What’s the highest recorded global temperature for each year in the dataset?
Unix Shell script program with awk, the classic tool for processing line-oriented data
Beginning of a run
The complete run for the century took 42 minutes in one run on a single EC2 High-CPU Extra Large Instance.
The scripts loops through the compressed year files
printing the year  processing each file using awk
Awk extracts the air temperature and the quality code from the data.
Temperature value 9999 signifies a missing value in the NCDC dataset.
Maximum temperature is 31.7℃ for 1901.

6. Analyzing the Data with Unix Tools
To speed up the processing, run parts of the program in parallel
Problems for parallel processing
Dividing the work into equal-size pieces isn’t always easy or obvious.
The file size for different years varies
The whole run is dominated by the longest file
A better approach is to split the input into fixed-size chunks and assign each chunk to a process
Combining the results from independent processes may need further processing.
Still limited by the processing capacity of a single machine, handling coordination and reliability for multiple machines
It’s feasible to parallelize the processing, though, it’s messy in practice.

7. Analyzing the Data with Hadoop – Map and Reduce
MapReduce works by breaking the processing into 2 phases: the map and the reduce.
Both map and reduce phases have key-value pairs as input and output.
Programmers have to specify two functions: map and reduce function.
The input to the map phase is the raw NCDC data.
Here, the key is the offset of the beginning of the line and the value is each line of the data set.
The map function pulls out the year and the air temperature from each input value.
The reduce function takes <year, temperature> pairs as input and produces the maximum temperature for each year as the result.

8. Analyzing the Data with Hadoop – Map and Reduce
Original NCDC Format
Input file for the map function, stored in HDFS
Output of the map function, running in parallel for each block
Input for the reduce function & Output of the reduce function

9. Analyzing the Data with Hadoop – Map and Reduce
The whole data flow
Map()
Shuffling
Reduce ()
<1950, 0>
<1950, 22>
<1949,111>
<1949, [111, 78]>
<1950, [0, 22, -11]>
<1949, 111>
<1950, 22>
<1951, 10>
<1952, 22>
<1951, [10, 76,34], 19>
<1952 ,[22, 34]>
<1953, [45]>
<1955, [23]>
<1951, 76>
<1952, 34>
<1953, 45>
<1955,25>
<1954, 0>
<1954, 22>
<1950, -11>
<1949, 78>
<1951, 25>
Input File

10. Analyzing the Data with Hadoop – Java MapReduce
Having run through how the MapReduce program works, express it in code
A map function, a reduce function, and some code to run the job are needed.
Map function

11. Analyzing the Data with Hadoop – Java MapReduce
Reduce function

12. Analyzing the Data with Hadoop – Java MapReduce
Main function for running the MapReduce job

13. Analyzing the Data with Hadoop – Java MapReduce
A test run
The output is written to the output directory, which contains one output file per reducer

14. Analyzing the Data with Hadoop – Java MapReduce
The new Java MapReduce API
The new API, referred to as “Context Objects”, is type-incompatible with the old, so applications need to be rewritten to take advantage of it.
Notable differences
Favors abstract classes over interfaces. The Mapper and Reducer interfaces are abstract classes.
The new API is in the org.apache.hadoop.mapreduce package and subpackages.
The old API can still be found in org.apache.hadoop.mapred
Makes extensive use of context objects that allow the user code to communicate with MapReduce system
i.e.) The MapContext unifies the role of the JobConf, the OutputCollector, and the Reporter
Supports both a ‘push’ and a ‘pull’ style of iteration
Basically key-value record pairs are pushed to the mapper, but in addition, the new API allows a mapper to pull records from within the map() method.
The same goes for the reducer
Configuration has been unified.
The old API has a JobConf object for job configuration, which is an extension of Hadoop’s vanilla Configuration object.
In the new API, job configuration is done through a Configuration.
Job control is performed through the Job class rather than JobClient.
Output files are named slightly differently
part-m-nnnnn for map outputs, part-r-nnnnn for reduce outputs
(nnnnn is an integer designating the part number, starting from 0)

15. Analyzing the Data with Hadoop – Java MapReduce
The new Java MapReduce API
Example 2-6 shows the MaxTemperature application rewritten to use the new API.

16. Scaling Out
To scale out, we need to store the data in a distributed filesystem, HDFS.
Hadoop moves the MapReduce computation to each machine hosting a part of the data.
Data Flow
A MapReduce job consists of the input data, the MapReduce program, and configuration information.
Hadoop runs the job by dividing it into 2 types of tasks, map and reduce tasks.
Two types of nodes, 1 jobtracker and several tasktrackers
Jobtracker : coordinates and schedules tasks to run on tasktrakers.
Tasktrackers : run tasks and send progress report to the jobtracker.
Hadoop divides the input into fixed-size pieces, called input splits, or just splits.
Hadoop creates one map task for each split, which runs the user-defined map function for each record in the split.
The quality of the load balancing increases as the splits become more fine-grained.
Default size : 1 HDFS block, 64MB
Map tasks write their output to the local disk, not to HDFS.
If the node running a map task fails, Hadoop will automatically rerun the map task on another node to re-create the map output.

17. Scaling Out Data Flow – single reduce task
Reduce tasks don’t have the advantage of data locality – the input to a single reduce task is normally the output from all mappers.
All map outputs are merged across the network and passed to the user-defined reduce function.
The output of the reduce is normally stored in HDFS.

18. Scaling Out Data Flow – multiple reduce tasks
The number of reduce tasks is specified independently not governed by the input size.
The map tasks partition their output by keys, each creating one partition for each reduce task.
There can be many keys and their associated values in each partition, but the records for any key are all in a single partition.

19. Scaling Out
Data Flow – zero reduce task

20. Scaling Out Combiner Functions
Many MapReduce jobs are limited by the bandwidth available on the cluster.
It pays to minimize the data transferred between map and reduce tasks.
Hadoop allows the user to specify a combiner function to be run on the map output – the combiner function’s output forms the input to the reduce function.
The contract for the combiner function constrains the type of function that may be used.
Example without a combiner function
Example with a combiner function, finding maximum temperature for a map
<1950, 0>
<1950, 20>
<1950, 10>
Reduce ()
<1950, [0, 20, 10, 25, 15]>
<1950, 25>
<1950, 15>
Map ()
shuffling
<1950, 0>
<1950, 20>
<1950, 10>
Reduce ()
<1950, [20, 25]>
<1950, 25>
<1950, 15>
Map ()
shuffling
combiner

21. Scaling Out Combiner Functions
The function calls on the temperature values can be expressed as follows:
Max(0, 20, 10, 25, 15) = max( max(0, 20, 10), max(25, 15) ) = max(20, 25) = 25
Calculating ‘mean’ temperatures couldn’t use the mean as the combiner function
mean(0, 20, 10, 25, 15) = 14
mean( mean(0, 20, 10), mean(25, 15) ) = mean(10, 20) = 15.
The combiner function doesn’t replace the reduce function.
It can help cut down the amount of data shuffled between the maps and the reduces

22. Scaling Out Combiner Functions Specifying a combiner function
The combiner function is defined using the Reducer interface
It is the same implementation as the reducer function in MaxTemperatureReducer.
The only change is to set the combiner class on the JobConf.

23. Hadoop Streaming Hadoop provides an API to MapReduce Hadoop Streaming
write the map and reduce functions in languages other than Java.
We can use any language in MapReduce program.
Hadoop Streaming
Map input data is passed over standard input to your map function.
The map function processes the data line by line and writes lines to standard output.
A map output key-value pair is written as a single tab-delimited line.
Reduce function reads lines from standard input (sorted by key), and writes its results to standard output.

24. Hadoop Streaming Ruby The map function can be expressed in Ruby.
Simulating the map function in Ruby with a Unix pipeline
The reduce function for maximum temperature in Ruby

25. Hadoop Streaming
Ruby
Simulating the whole MapReduce pipeline with a Unix pipeline
Hadoop command to run the whole MapReduce job
When using the combiner which is coded in any streaming language

26. Hadoop Streaming Python
Streaming supports any programming language that can read from standard input and write to standard output.
The map and reduce script in Python
Test the programs and run the job in the same way we did in Ruby.

27. Hadoop Pipes Hadoop Pipes
The name of the C++ interface to Hadoop MapReduce.
Pipes uses sockets as the channel over which the tasktracker communicates with the process running the C++ map or reduce function.
The source code for the map and reduce functions in C++

28. Hadoop Pipes
The source code for the map and reduce functions in C++

29. Hadoop Pipes Compiling and Running Makefile for C++ MapReduce program
Defining PLATFORM which specifies the operating system, architecture, and data model (e.g., 32- or 64-bit).
To run a Pipes job, we need to run Hadoop (daemon) in pseudo-distributed model.
Next step is to copy the executable code (program) to HDFS.
Next, the sample data is copied from the local filesystem to HDFS.

30. Hadoop Pipes Compiling and Running
Now, we can run the job. For this, we use the Hadoop pipes command, passing URI of the executable in HDFS using the –program argument: