Yahoo! Research Johns Hopkins University Chris Olston Anish Das Sarma Xiaodan Wang Randal Burns Shared Scan Batch Scheduling in Cloud Computing

Project Goals Eliminate redundant data processing for concurrent workflows that access the same dataset in the Cloud Batch MapReduce workflows to enable scan sharing – Single pass scan of shared data segments – Alleviate contention and improve scalability – Utilize fewer map/reduce slots under load Data-intensive workloads (tens of minutes to hours) – Joins across multiple datasets – User specified rewards for early completion Trade-offs between efficient resource utilization and deadlines

Shared Scan Batch Scheduling in Cloud Computing Data-Driven Batch Scheduling Throughput scales with contention (Astro. & Turbulence) Decompose into sub-queries based on data access Co-schedule sub-queries to amortize I/O Evaluate data atoms based on utility metric – Reordering based on contention vs. arrival order (CIDR’09) – Adaptive starvation resistance – Job-aware (queries with data dependency) (SC’10) Turbulence DB R1R2R3 R2R3R4 R1R2 Q1 Q2 Q3 Decomposition Data Access by Query Q1Q2Q3 Q1Q3 Q1Q2 R2 R1 R3 Q2 R3 Co-schedule by Sub-query Batch Sched. Query Results Query Results

Shared Scan Batch Scheduling in Cloud Computing Application in Cloud Computing Fixed Cloud (fixed resources) – Single pass scan of shared data – Alleviate contention (utilize less map/reduce slots, shared loading and shuffling of data) – Earn rewards for early completion (soft deadlines) – Local improvement w/ simulated annealing, greedy ordering Elastic Cloud – Machine charge = (# of machines) x (# hours) – Speed-up factors w/ more machines (i.e. more parallelism) – Add machines to meet soft deadlines – Aggressive batching to minimize machine charge (efficiency)

Shared Scan Batch Scheduling in Cloud Computing Sample Pig A = load ‘input1' as (a, b, c); B = filter A by a > 5; store B into 'output1'; C = group B by b; store C into 'output2'; Nova Workflow Platform What is Nova? – Content mgmt and workflow scheduling for the Cloud – Leverages existing resources Cloud Data: HDFS/Zebra storage Cloud Computing: Oozie, Pig/MR/Hadoop Users define complex workflows in Oozie that consume the data Storage: HDFS Processing: Hadoop M- R Simple workflow: Oozie Dataflow: Pig Advanced workflow: Nova App 1 App 2 App 3 Oozie Workflow engine for coordinating Map- Reduce/Pig jobs in Hadoop (i.e. Workflow DAG in which nodes are MR tasks and edges are dataflows)

Shared Scan Batch Scheduling in Cloud Computing Sample Nova Workflow cand. entity extractor candidate entity occurrences (url, entity string) crawled pages (url, content) entities (entity id, entity string) validated entity occurrences (url, entity id) entity occurrence counts (entity id, count) crawle r editors join group- wise count output Nova Tasks Nova Task Nova Data

Shared Scan Batch Scheduling in Cloud Computing Shared Scan via Workflow Merging Nova Workflow 1 Nova Workflow 2 c2s0c2s0 c1s0c1s0c3s0c3s0 c4s0c4s0 c2s0c2s0 c5s0c5s0 Workflow Merger Nova Workflow 1.2 (scans c2s0 once) c2s0c2s0 c1s0c1s0c3s0c3s0 c4s0c4s0 c5s0c5s0 Input Data Pig/MR Tasks Output Data Sample Use Cases in Nova – Concurrent research, production, maintenance workflows over same data – Content enrichment workflows (i.e. dedup, clustering) over news content – Webmap workflows consuming same URL table

Shared Scan Batch Scheduling in Cloud Computing Output Data Map 1 Reduce 1 Shuffle Split(Tuple)Split(Tuple) NestedPlanNestedPlan NestedPlanNestedPlan NestedPlanNestedPlan Combine(Tuple)Combine(Tuple) Demux(Tuple)Demux(Tuple) NestedPlanNestedPlan NestedPlanNestedPlan NestedPlanNestedPlan Map 2 Reduce 2 … … Output Data Map n Reduce m Input Data Performance Impact (1) Shared Loading (network, redundant proc.) (2) Consolidated computation (shared startup/tear down) (3) Reducer parallelism (Max/Sum # of reducers) (1) (2) (3)

Shared Scan Batch Scheduling in Cloud Computing Completion Time by Scheduling Strategy Performance in Nova for different enrichment workflows (ie. de-dup) on news content (SIGMOD’11)

Shared Scan Batch Scheduling in Cloud Computing Utilization of Grid Resources (Slot Time)

Shared Scan Batch Scheduling in Cloud Computing PigMix: Load Cost Savings

Shared Scan Batch Scheduling in Cloud Computing PigMix: Estimating Makespan

Shared Scan Batch Scheduling in Cloud Computing Ongoing Work Starvation resistance – Account for heterogeneity in workflow sizes – Provide soft deadline guarantees – Handling cascading failures – Prefer jobs with high load cost (less dilation, high slot time savings, map-only jobs) Predicting workflow runtime and frequency – Robustness to inaccuracies in cost estimates – Conserve or expend Cloud resources based on deadline requirements and system load Jobs that join/scan multiple input sources

Shared Scan Batch Scheduling in Cloud Computing Questions?

Shared Scan Batch Scheduling in Cloud Computing Nova Workflow Platform Nova features – Abstraction for complex workflows that consume data Incrementally arriving data (logs, crawls, feeds,...) Incremental processing of arriving data – Stateless: shingle every newly-crawled page – Stateful: maintain inlink counts as web grows Scheduling processing steps – Periodic: run inlink counter once per week – Triggered: run inlink counter after link extractor – Provides provenance, metadata management, incremental processing (i.e. joins), data replication, transactional guarantees

Shared Scan Batch Scheduling in Cloud Computing PigMix: Reducer Parallelism

Shared Scan Batch Scheduling in Cloud Computing Optimizing for Shared Scan Define a job J (i.e. MapReduce or Pig) – Scans files f(J) = (F 1, …, F i ), scan time per file: s(F i ) – Fixed processing cost c(J) d(J) defines a soft deadline of each job – Step: d defined by n pairs of (t i, p i ) where 0 p i+1 (a job that completes by t i is award p i points) – Linearly decay: enforce eventual completion w/ negative pts Cost of shared scan for Jobs J 1 and J 2 c(J 1 ) + c(J 2 ) + ∑ Fє(f(J1) U f(J2)) s(F) Maximize points and minimize resources – Local improvement w/ simulated annealing, greedy ordering – Aggressive batching when load is high

Shared Scan Batch Scheduling in Cloud Computing Performance Evaluation Experimental Setup – Nova with Shared Scan Module – 200 node Hadoop cluster 128MB HDFS block size 1GB RAM per node 640 mapper and 320 reducer slots – Shingling workflow (offline content enrichment) De-duplication of news Filter and extract features from content Cluster content by feature and pick one per cluster Execution of multiple de-dup workflows using different clustering alg. – Scheduling strategies compared Sequential-NoMerge (slower, conserve Grid resources) Concurrent-NoMerge (fast, elastic Grid resources) Merged (fast, conserve Grid resources)