1. Apache Tez : Accelerating Hadoop Data Processing
Bikas Saha
@bikassaha

2. Tez – Introduction
Distributed execution framework targeted towards data- processing applications.
Based on expressing a computation as a dataflow graph.
Highly customizable to meet a broad spectrum of use cases.
Built on top of YARN – the resource management framework for Hadoop.
Open source Apache project and Apache licensed.

3. Tez – Hadoop 1 ---> Hadoop 2
Monolithic
Resource Management – MR
Execution Engine – MR
Layered
Resource Management – YARN
Engines – Hive, Pig, Cascading, Your App!

4. Tez – Empowering Applications
Tez solves hard problems of running on a distributed Hadoop environment
Apps can focus on solving their domain specific problems
This design is important to be a platform for a variety of applications
App
Custom application logic
Custom data format
Custom data transfer technology
Tez
Distributed parallel execution
Negotiating resources from the Hadoop framework
Fault tolerance and recovery
Horizontal scalability
Resource elasticity
Shared library of ready-to-use components
Built-in performance optimizations
Security

5. Tez – End User Benefits Better performance of applications
Built-in performance + Application define optimizations
Better predictability of results
Minimization of overheads and queuing delays
Better utilization of compute capacity
Efficient use of allocated resources
Reduced load on distributed filesystem (HDFS)
Reduce unnecessary replicated writes
Reduced network usage
Better locality and data transfer using new data patterns
Higher application developer productivity
Focus on application business logic rather than Hadoop internals

6. Tez – Design considerations
Look to the Future with an eye on the Past
Don’t solve problems that have already been solved. Or else you will have to solve them again!
Leverage discrete task based compute model for elasticity, scalability and fault tolerance
Leverage several man years of work in Hadoop Map-Reduce data shuffling operations
Leverage proven resource sharing and multi-tenancy model for Hadoop and YARN
Leverage built-in security mechanisms in Hadoop for privacy and isolation

7. Tez – Problems that it addresses
Expressing the computation
Direct and elegant representation of the data processing flow
Interfacing with application code and new technologies
Performance
Late Binding : Make decisions as late as possible using real data from at runtime
Leverage the resources of the cluster efficiently
Just work out of the box!
Customizable to let applications tailor the job to meet their specific requirements
Operation simplicity
Painless to operate, experiment and upgrade

8. Tez – Simplifying Operations
No deployments to do. No side effects. Easy and safe to try it out!
Tez is a completely client side application.
Simply upload to any accessible FileSystem and change local Tez configuration to point to that.
Enables running different versions concurrently. Easy to test new functionality while keeping stable versions for production.
Leverages YARN local resources.
HDFS
Tez Lib 1
Tez Lib 2
Client
Machine
Node
Manager
Node
Manager
Client
Machine
TezTask
TezTask
TezClient
TezClient

9. Tez – Expressing the computation
Distributed data processing jobs typically look like DAGs (Directed Acyclic Graph).
Vertices in the graph represent data transformations
Edges represent data movement from producers to consumers
Preprocessor Stage
Task-1
Task-2
Samples
Sampler
Partition Stage
Ranges
Task-1
Task-2
Distributed Sort
Aggregate Stage
Task-1
Task-2

10. Tez – Expressing the computation
Tez provides the following APIs to define the processing
DAG API
Defines the structure of the data processing and the relationship between producers and consumers
Enable definition of complex data flow pipelines using simple graph connection API’s. Tez expands the logical DAG at runtime
This is how the connection of tasks in the job gets specified
Runtime API
Defines the interfaces using which the framework and app code interact with each other
App code transforms data and moves it between tasks
This is how we specify what actually executes in each task on the cluster nodes

11. Tez – DAG API Defines the global processing flow // Define DAG
DAG dag = DAG.create();
// Define Vertex
Vertex Map1 = Vertex.create(Processor.class);
// Define Edge
Edge edge = Edge.create(Map1, Reduce1, SCATTER_GATHER, PERSISTED, SEQUENTIAL, Output.class, Input.class);
// Connect them
dag.addVertex(Map1).addEdge(edge)…
Map1
Map2
Scatter
Gather
Reduce1
Reduce2
Scatter
Gather
Join

12. Tez – DAG API
Edge properties define the connection between producer and consumer tasks in the DAG
Data movement – Defines routing of data between tasks
One-To-One : Data from the ith producer task routes to the ith consumer task.
Broadcast : Data from a producer task routes to all consumer tasks.
Scatter-Gather : Producer tasks scatter data into shards and consumer tasks gather the data. The ith shard from all producer tasks routes to the ith consumer task.

13. Tez – Logical DAG expansion at Runtime
Map1
Map2
Reduce1
Reduce2
Join

14. Tez – Runtime API Flexible Inputs-Processor-Outputs Model
Thin API layer to wrap around arbitrary application code
Compose inputs, processor and outputs to execute arbitrary processing
Applications decide logical data format and data transfer technology
Customize for performance
Built-in implementations for Hadoop 2.0 data services – HDFS and YARN ShuffleService. Built on the same API. Your implementations are as first class as ours!

15. Tez – Task Composition Logical DAG V-A V-B V-C
Output-1
Output-3
Processor-A
Input-2
Processor-B
Input-4
Processor-C
Task A
Task B
Task C
V-A
Edge AB
Edge AC
V-B
V-C
V-A = { Processor-A.class }
V-B = { Processor-B.class }
V-C = { Processor-C.class }
Edge AB = { V-A, V-B,
Output-1.class, Input-2.class }
Edge AC = { V-A, V-C,
Output-3.class, Input-4.class }

16. Tez – Library of Inputs and Outputs
Classical ‘Map’
Classical ‘Reduce’
Map Processor
HDFS Input
Sorted Output
Reduce Processor
Shuffle Input
HDFS Output
What is built in?
Hadoop InputFormat/OutputFormat
SortedGroupedPartitioned Key-Value Input/Output
UnsortedGroupedPartitioned Key-Value Input/Output
Key-Value Input/Output
Reduce Processor
Shuffle Input
Sorted Output
Intermediate ‘Reduce’ for
Map-Reduce-Reduce

17. Tez – Composable Task Model
Adopt
Evolve
Optimize
HDFS
Input
Remote
File
Server
Input
HDFS
Input
Remote
File
Server
Input
RDMA
Input
Native DB
Input
Hive Processor
Custom Processor
Custom Processor
HDFS
Output
Local
Disk
Output
HDFS
Output
Local
Disk
Output
Kakfa
Pub-Sub
Output
Amazon S3
Output

18. Tez – Performance Benefits of expressing the data processing as a DAG
Reducing overheads and queuing effects
Gives system the global picture for better planning
Efficient use of resources
Re-use resources to maximize utilization
Pre-launch, pre-warm and cache
Locality & resource aware scheduling
Support for application defined DAG modifications at runtime for optimized execution
Change task concurrency
Change task scheduling
Change DAG edges
Change DAG vertices

19. Tez – Benefits of DAG execution
Faster Execution and Higher Predictability
Eliminate replicated write barrier between successive computations.
Eliminate job launch overhead of workflow jobs.
Eliminate extra stage of map reads in every workflow job.
Eliminate queue and resource contention suffered by workflow jobs that are started after a predecessor job completes.
Better locality because the engine has the global picture
Pig/Hive - MR
Pig/Hive - Tez

20. Tez – Container Re-Use Reuse YARN containers/JVMs to launch new tasks
Reduce scheduling and launching delays
Shared in-memory data across tasks
JVM JIT friendly execution
YARN Container
YARN Container / JVM
Tez
Application Master
TezTask Host
Start Task
TezTask1
Task Done
Shared Objects
Start Task
TezTask2

21. Shared Object Registry
Tez – Sessions
Sessions
Standard concepts of pre-launch and pre-warm applied
Key for interactive queries
Represents a connection between the user and the cluster
Multiple DAGs executed in the same session
Containers re-used across queries
Takes care of data locality and releasing resources when idle
Start Session
Submit DAG
Client
Application Master
Task Scheduler
Pre Warmed
JVM
Shared Object Registry
Container Pool

22. Tez – Re-Use in Action
Containers
Time 
Task Execution Timeline

23. Tez – Customizable Core Engine
Start
vertex
Vertex Manager
Determines task parallelism
Determines when tasks in a vertex can start.
DAG Scheduler
Determines priority of task
Task Scheduler
Allocates containers from YARN and assigns them to tasks
Get container
Vertex-1
Get Priority
DAG
Scheduler
Task
Scheduler
Start
vertex
Vertex Manager
Start
tasks
Vertex-2
Get Priority
Get container

24. Tez – Event Based Control Plane
Events used to communicate between the tasks and between task and framework
Data Movement Event used by producer task to inform the consumer task about data location, size etc.
Input Error event sent by task to the engine to inform about errors in reading input. The engine then takes action by re-generating the input
Other events to send task completion notification, data statistics and other control plane information
Map Task 1
Output1
Output2
Map Task 2
Output1
Output2
Output3
Output3
Data Event AM
Router
Scatter-Gather Edge
Error Event
Reduce Task 2
Input1
Input2

25. Tez – Automatic Reduce Parallelism
Event Model
Map tasks send data statistics events to the Reduce Vertex Manager.
Vertex Manager
Pluggable application logic that understands the data statistics and can formulate the correct parallelism. Advises vertex controller on parallelism
App Master
Data Size Statistics
Vertex Manager
Map Vertex
Set Parallelism
Vertex State
Machine
Re-Route
Reduce Vertex
For anyone who has been working on MapReduce, there is this age-old problem around “how do I figure out the correct number of reducers?”. We guess some number at compile-time and usually that turns out to be incorrect at run-time. Let’s see how we can use the Tez model to fix that. So here is this Map Vertex and this Reduce Vertex, which have these tasks running and you have the Vertex Manager running inside the framework …
[CLICK] The Map Tasks can send Data Size Statistics to the Vertex Manager, which can then extrapolate those statistics to figure out “what would be the final size of the data when all of these Maps finish?”. Based on that, it can realize that the data size is actually smaller than expected, and I can actually run two reduce tasks instead of three.
[CLICK] The Vertex Manager sends a Set Paralellism command to the framework which changes the routing information in-between these two tasks and also cancels the last task.
Cancel Task

26. Tez – Reduce Slow Start/Pre-launch
Event Model
Map completion events sent to the Reduce Vertex Manager.
Vertex Manager
Pluggable application logic that understands the data size. Advises the vertex controller to launch the reducers before all maps have completed so that shuffle can start.
App Master
Task Completed
Vertex Manager
Map Vertex
Start Tasks
Vertex State
Machine
Reduce Vertex
So the new plan looks like this … which is how Tez handles the concurrency problem of plan re-configuration. What about “when do you actually want to launch those tasks?”?
In MapReduce, there is an interesting concept of pre-launching the Reducers, in which even though the reducers are going to read the output from 100 mappers, we may want to launch the Reducers when only 50 mappers have finished – because then they can use the capacity in the cluster to start pre-fetching the data from the completed mappers while the rest of their inputs still complete. How can we do this in Tez? Again, the same Vertex Manager comes into the picture.
[CLICK] As the Map Tasks complete they send Task Completed notifications to the Vertex Manager. There it can decide that enough Map tasks have finished that I need to start one of my reducers so that it can start pre-fetching the data.
[CLICK] It sends a Start Task notification to the framework, which results in one of the Reduce Tasks starting.
Start

27. Tez – Theory to Practice
Performance
Scalability

28. Tez – Hive TPC-DS Scale 200GB latency
query 1: SELECT pageURL, pageRank FROM rankings WHERE pageRank > X

29. Tez – Pig performance gains
Demonstrates performance gains from a basic translation to a Tez DAG
Deeper integration in the works for further improvements
1.5x to 3x speedup on some of the Pigmix queries.

30. Tez – Observations on Performance
Number of stages in the DAG
Higher the number of stages in the DAG, performance of Tez (over MR) will be better.
Cluster/queue capacity
More congested a queue is, the performance of Tez (over MR) will be better due to container reuse.
Size of intermediate output
More the size of intermediate output, the performance of Tez (over MR) will be better due to reduced HDFS usage.
Size of data in the job
For smaller data and more stages, the performance of Tez (over MR) will be better as percentage of launch overhead in the total time is high for smaller jobs.
Offload work to the cluster
Move as much work as possible to the cluster by modelling it via the job DAG. Exploit the parallelism and resources of the cluster. E.g. MR split calculation.
Vertex caching
The more re-computation can be avoided the better is the performance.

31. Tez – Data at scale
Hive TPC-DS Scale 10TB

32. Tez – DAG definition at scale
Hive : TPC-DS Query 64 Logical DAG

33. Tez – Container Reuse at Scale
78 vertices tasks on 50 containers (TPC-DS Query 4)

34. Tez – Real World Use Cases for the API

35. Tez – Broadcast Edge Hive : Broadcast Join Hive – MR Hive – Tez
SELECT ss.ss_item_sk, ss.ss_quantity, avg_price, inv.inv_quantity_on_hand
FROM (select avg(ss_sold_price) as avg_price, ss_item_sk, ss_quantity_sk from store_sales
group by ss_item_sk) ss
JOIN inventory inv
ON (inv.inv_item_sk = ss.ss_item_sk);
Hive : Broadcast Join
Hive – MR
Hive – Tez M
HDFS
Store Sales scan. Group by and aggregation reduce size of this input.
Inventory scan and Join
Broadcast edge M
HDFS
Store Sales scan. Group by and aggregation.
Inventory and Store Sales (aggr.) output scan and shuffle join. M M M R R R R
HDFS M R R

36. Split multiplex de-multiplex
Tez – Multiple Outputs
f = LOAD ‘foo’ AS (x, y, z);
g1 = GROUP f BY y;
g2 = GROUP f BY z;
j = JOIN g1 BY group,
g2 BY group;
Group by y Group by z
Load foo
Join
Load g1 and Load g2
Group by y
Group by z
Multiple outputs
Reduce follows
reduce
HDFS
Split multiplex de-multiplex
Pig : Split & Group-by
Pig – MR
Pig – Tez

37. Tez – Multiple Outputs Hive : Multi-insert queries Hive – MR
FROM (SELECT * FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk and d_year = 2000)
INSERT INTO TABLE t1 SELECT distinct ss_item_sk
INSERT INTO TABLE t2 SELECT distinct ss_customer_sk;
Hive : Multi-insert queries
Hive – MR
Hive – Tez M
HDFS
Map join date_dim/store sales
Two MR jobs to do the distinct
Broadcast Join (scan date_dim, join store sales) M M
HDFS M M M
Materialize join on HDFS
Distinct for customer + items R R M M M M M M
HDFS R R
HDFS

38. Partition L and Partition R
Tez – One to One Edge
Aggregate
Sample L
Join
Stage sample map
on distributed cache
l = LOAD ‘left’ AS (x, y);
r = LOAD ‘right’ AS (x, z);
j = JOIN l BY x, r BY x
USING ‘skewed’;
Load &
Sample
Partition L
Pass through input
via 1-1 edge
Partition R
HDFS
Pig : Skewed Join
Pig – MR
Pig – Tez
Broadcast
sample map
Partition L and Partition R

39. Tez – Custom Edge Hive : Dynamically Partitioned Hash Join Hive – MR
SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand
FROM store_sales ss
JOIN inventory inv
ON (inv.inv_item_sk = ss.ss_item_sk);
Hive – MR
Hive – Tez M
HDFS
Inventory scan (Runs as single local map task)
Store Sales scan and Join (Inventory hash table read as side file) M
HDFS
Inventory scan (Runs on cluster potentially more than 1 mapper)
Store Sales scan and Join (Custom vertex reads both inputs – no side file reads)
Custom edge (routes outputs of previous stage to the correct Mappers of the next stage)
HDFS

40. Tez – Bringing it all together
Tez Session populates container pool
Dimension table calculation and HDFS split generation in parallel
Dimension tables broadcasted to Hive MapJoin tasks
Final Reducer pre-launched and fetches completed inputs
TPCDS – Query-27 with Hive on Tez
Architecting the Future of Big Data

41. Tez – Bridging the Data Spectrum
Fact Table
Dimension Table 1
Dimension Table 1
Fact Table
Broadcast
Join
Dimension Table 1
Broadcast join
for small data sets
Result Table 1
Dimension Table 2
Dimension Table 1
Broadcast
Join
Result Table 2
Dimension Table 3
Shuffle
Join
Based on data size, the query optimizer can run either plan as a single Tez job
Typical pattern in a TPC-DS query
Result Table 3

42. Tez – Current status Apache Project Focus on stability
Growing community of contributors and users
Latest release is 0.5.0
Focus on stability
Testing and quality are highest priority
Code ready and deployed on multi-node environments at scale
Support for a vast topology of DAGs
Already functionally equivalent to Map Reduce. Existing Map Reduce jobs can be executed on Tez with few or no changes
Rapid adoption by important open source projects and by vendors

43. Tez – Developer Release
API stability – Intuitive and clean APIs that will be supported and be backwards compatible with future releases
Local Mode Execution – Allows the application to run entirely in the same JVM without any need for a cluster. Enables easy debugging using IDE’s on the developer machine
Performance Debugging – Adds swim-lanes analysis tool to visualize job execution and help identify performance bottlenecks
Documentation and tutorials to learn the APIs

44. Tez – Adoption Apache Hive Apache Pig Cascading Commercial Vendors
Hadoop standard for declarative access via SQL-like interface (Hive 13)
Apache Pig
Hadoop standard for procedural scripting and pipeline processing (Pig 14)
Cascading
Developer friendly Java API and SDK
Scalding (Scala API on Cascading) (3.0)
Commercial Vendors
ETL : Use Tez instead of MR or custom pipelines
Analytics Vendors : Use Tez as a target platform for scaling parallel analytical tools to large data-sets
Hive has written it’s own processor

45. Tez – Adoption Path Pre-requisite : Hadoop 2 with YARN
Tez has zero deployment pain. No side effects or traces left behind on your cluster. Low risk and low effort to try out.
Using Hive, Pig, Cascading, Scalding
Try them with Tez as execution engine
Already have MapReduce based pipeline
Use configuration to change MapReduce to run on Tez by setting ‘mapreduce.framework.name’ to ‘yarn-tez’ in mapred-site.xml
Consolidate MR workflow into MR-DAG on Tez
Change MR-DAG to use more efficient Tez constructs
Have custom pipeline
Wrap custom code/scripts into Tez inputs-processor-outputs
Translate your custom pipeline topology into Tez DAG
Change custom topology to use more efficient Tez constructs
Hive has written it’s own processor

46. Tez – Roadmap Richer DAG support Performance optimizations Usability
Addition of vertices at runtime
Shared edges for shared outputs
Enhance Input/Output library
Performance optimizations
Improve support for high concurrency
Improve locality aware scheduling
Add framework level data statistics
HDFS memory storage integration
Usability
Stability and testability
UI integration with YARN
Tools for performance analysis and debugging

47. Tez – Community Early adopters and code contributors welcome
Adopters to drive more scenarios. Contributors to make them happen.
Tez meetup for developers and users
Technical blog series
Useful links
Work tracking:
Code:
 Developer list:  User list:  Issues list:

48. Tez – Takeaways
Distributed execution framework that models processing as dataflow graphs
Customizable execution architecture designed for extensibility and user defined performance optimizations
Works out of the box with the platform figuring out the hard stuff
Zero install – Safe and Easy to Evaluate and Experiment
Addresses common needs of a broad spectrum of applications and usage patterns
Open source Apache project – your use-cases and code are welcome
It works and is already being used by Apache Hive and Pig

49. Tez Thanks for your time and attention! Video with Deep Dive on Tez
– WordCount code walk through with 0.5 APIs. Debugging in local mode. Execution and profiling in cluster mode. (Starts at ~27 min. Ends at ~37 min)
Questions?
@bikassaha