1. Pig Latin - A Not-So-Foreign Language for Data Processing
YING YU
Sep 19th 2016

2. Background what's the problem? Looking for trends →Product improvement
When the data sets are growing to be extremely large, how can we make the ad-hoc data analysis tasks more efficiently?
Looking for trends →Product improvement
→Advertisement

3. Previous Solution-Analysis of Large Data Sets
parallel database products: Teradata, Oracle RAC, etc.
Problem:
Expensive at web scale
SQL:Unnatural, restrictive to write
MapReduce Model
Code: difficult to reuse ad maintain
map and reduce functions : opaque for the optimization and check user's code

4. Yahoo!'s New Solution: Pig Latin
combine the best of SQL and MapReduce:
A sequence of steps , each step carries out a single fairly high-level data transformation
high level declarative querying
low level procedural programming

5. Why Pig Latin? Quick start and interoperability
no need to import data before queries
user chooses the output
operate data in external files
2. Support nested data model
flexible, more natural for us to think
operate on files, reduce time
and space
algebraic language
e.g: To store information of documents
Nested data model :
Map<documentId,Set<positions>>
Databases:
term_info: (termId, termString, ...)
position_info: (termId, documentId, position)

6. Why Pig Latin? 3. Support UDFs-Extensibility
for specialized data processing
top10()is a UDF
In SQL, different clauses can only use the correspondent functions
4. A novel debugging environment
detect errors early and pinpoint the error step.
groups = GROUP urls BY category;
output = FOREACH groups GENERATE
category, top10(urls);
Select url from table where catagory='a'

7. Pig Latin-data model type definition fetch data atom
a string or a number
'a'
tuple
sequence of fields
('a','b')
use tuple.id, tuple.(id0, id1) or $0
bag
collection of tuples
{('a','b'),('a',('c','d'))}
use bag.id, bag.(id0, id1) or $0
map
collection of data items map#’key’
[‘fan of’ →{(‘lakers’)(‘iPod’)}
‘age’ → 20 ]
use $0#’key’ or field_name#’key’
field by position or field by name

8. 1. Load: to specify input data
Commands
1. Load: to specify input data
2. Foreach:
to do the independent processing to every tuple, “flatten” is used to eliminate the nesting.
queries = LOAD ‘query_log.txt’
USING myLoad()
AS (userId, queryString, timestamp);
expanded_queries = FOREACH queries GENERATE
userId, FLATTEN(expandQuery(queryString));
myload ()convert the file into tuples

9. 3. filter : to get the target subset data
Commands
3. filter : to get the target subset data
4. cogroup: to get related data together
group, join,map-reduce
real_queries =
FILTER queries BY NOT isBot(userId);
SELECT queryresult
FROM table
WHERE userId !=”a ”
grouped_data = COGROUP results BY queryString,
revenue BY queryString;

10. 5. Other SQL similar commands: 6.store: store nested data
union, cross, order and distinct
6.store: store nested data
7.nested functions
allow FILTER, ORDER, and DISTINCT to be nested within FOREACH
STORE query_revenues INTO 'myoutput'
USING myStore();
ask for result to be materialized to a file

11. Execution platform: Hadoop Build a logical plan:
Implementation
Execution platform: Hadoop
Build a logical plan:
iterative
independent
Try to avoid the large nested bags
Q: What to do when some operations after the cogroup is not algebraic UDF?

12. e.g: Logical Plan Construction of logical plan
The logical plan of “group_data” includes:
a cogroup command
logical plans for two bags, results and revenue
Interpreter to parse the commands
Compile
When the store command is invoked on a bag:
logical plan→physical plan
grouped_data = COGROUP results BY queryString,
revenue BY queryString;

13. Map-Reduce Plan
convert each COGROUP
command in the logical plan
into a distinct map reduce job
Q: What if there exists more than input data set in a case of a COGROUP command?
map functions appends an extra field to each tuple that identifies where the tuple is from
map and reduce instances are run in parallel
e.g. from Programming Pig

14. Debugging environment- Pig Pen
sandbox data set(Realism,Conciseness,Completeness)
help understand the schema
Q:How to generate
the dataset that
fulfills the three
main objectives

15. Data sets are too big and continuous to be accumulated
use scenarios
Rollup aggregates:
Data sets are too big and continuous to be accumulated
Do the successive aggregations or other commands directly on the files
Temporal analysis
To study the pattern of data change
data in different periods (COGROUP, UDFs)
Session analysis
Analysis of the user activities during a session
use the nested data model represents and manipulates sessions.(ORDER BY)

16. Future Optimizations Database style optimization
Secure optimization without influence on semantics UI
support other languages
unified environment

17. Q: Does Pig have the transaction handling and index ?
Objectives of Pig Latin
What are the main differences between PIG and the RDBMS?
Q: Does Pig have the transaction handling and index ?
A: No
How will the pig latin support other script programming languages such as Perl or Python?