1. Pig, Making Hadoop Easy
Alan F. Gates
Yahoo!

2. Who Am I? Pig committer Hadoop PMC Member
An architect in Yahoo! grid team
Or, as one coworker put it, “the lipstick on the Pig”

3. Who are you?
How many have used Pig? How many have looked at it and have a basic understanding of it?

4. Motivation By Example
Suppose you have user data in one file, website data in another, and you need to find the top 5 most visited pages by users aged
Load Users
Load Pages
Filter by age
Join on name
Group on url
Count clicks
Order by clicks
Take top 5

5. In Map Reduce

6. In Pig Latin
Users = load ‘users’ as (name, age); Fltrd = filter Users by age >= 18 and age <= 25; Pages = load ‘pages’ as (user, url); Jnd = join Fltrd by name, Pages by user; Grpd = group Jnd by url; Smmd = foreach Grpd generate group, COUNT(Jnd) as clicks; Srtd = order Smmd by clicks desc; Top5 = limit Srtd 5; store Top5 into ‘top5sites’;

7. Performance
0.1
0.2
0.3
0.4,
0.5
0.6,
0.7

8. Why not SQL? Data Collection Data Factory Pig Pipelines
Iterative Processing
Research
Data Warehouse
Hive
BI Tools
Analysis

9. Pig Highlights
User defined functions (UDFs) can be written for column transformation (TOUPPER), or aggregation (SUM)
UDFs can be written to take advantage of the combiner
Four join implementations built in: hash, fragment-replicate, merge, skewed
Multi-query: Pig will combine certain types of operations together in a single pipeline to reduce the number of times data is scanned
Order by provides total ordering across reducers in a balanced way
Writing load and store functions is easy once an InputFormat and OutputFormat exist
Piggybank, a collection of user contributed UDFs

10. Who uses Pig for What? 70% of production jobs at Yahoo (10ks per day)
Also used by Twitter, LinkedIn, Ebay, AOL, …
Used to
Process web logs
Build user behavior models
Process images
Build maps of the web
Do research on raw data sets

11. Submit a script directly Grunt, the pig shell
Accessing Pig
Submit a script directly
Grunt, the pig shell
PigServer Java class, a JDBC like interface

12. Components Job executes on cluster Pig resides on user machine
Hadoop Cluster
Pig resides on user machine
User machine
No need to install anything extra on your Hadoop cluster.

13. How It Works Pig Latin pig.jar: parses Execution Plan checks Map:
A = LOAD ‘myfile’
AS (x, y, z);
B = FILTER A by x > 0;
C = GROUP B BY x;
D = FOREACH A GENERATE
x, COUNT(B);
STORE D INTO ‘output’;
pig.jar:
parses
checks
optimizes
plans execution
submits jar to Hadoop
monitors job progress
Execution Plan
Map:
Filter
Count
Combine/Reduce:
Sum

14. Demo s3://hadoopday/pig_tutorial Demo script:
Show group query first, talk about:
load and schema (none, declared, from data)
data types
data sources need not be from HDFS or even from files
parallel clause, how parallelism is determined on maps
how grouping works in Pig Latin
So far what I’ve shown you is a simple join/group query. Now let’s look at something less straight forward in SQL
Often people want to group data a number of different ways. Look at multiquery script:
Note how there’s a branch in the logic now
Often want to operate on the result of each record in a previous statement. Look at top5 query
Note nested foreach allows you to operate on each record coming out of group by
Since result of group by is a bag in each record, can apply operators to that bag
Currently support order, distinct, filter, limit
Use of flatten at the end
Use of positional parameters
There will always be logic you need to write that you can’t get from Pig Latin. This is where rich support of UDFs come in. Look at session query
Note registering UDF
UDF now called like any other Pig builtin function (in fact Pig builtins implemented as UDFs)
Look at SessionAnalysis.java
Class name is UDF name
Input to UDF is always a Tuple, avoids need to declare expected input, means UDF has to check what it gets
Talk about how projection of bags works
Talk about how EvalFunc is templatized on return type
Also easy to write load and store functions to fit your data needs

15. Upcoming Features
In 0.8 (plan to branch end of August, release this fall):
Runtime statistics collection
UDFs in scripting languages (e.g. python)
Ability to specify a custom partitioner
Adding many string and math functions as Pig supported UDFs
Post 0.8
Adding branches, loops, functions, and modules
Usability
Better error messages
Fix ILLUSTRATE
Improved integration with workflow systems

16. Learn More
Read the online documentation:
On line tutorials
From Yahoo,
From Cloudera,
Using Pig on EC2:
A couple of Hadoop books available that include chapters on Pig, search at your favorite bookstore
Join the mailing lists:
for user questions
for developer issues
for Howl