Active and Accelerated Learning of Cost Models for Optimizing Scientific Applications Piyush Shivam, Shivnath Babu, Jeffrey Chase Duke University

C3C3 C1C1 C2C2 Site A Site B Site C Task scheduler Task workflow A network of clusters or grid sites Networked Computing Utility Each site is a pool of heterogeneous resources Jobs are task workflows Challenge: choose good resource assignments for the jobs

C3C3 C1C1 C2C2 Site A Site B Site C home file server P1 P2 P3 A workflow with a single task Example: Assigning Resources to Run Tasks P1Site A Task input data at Site A Execution plan Ξ Resource assignment P2Site BSite A P3Site B PlanCPUStorage

Plan Selection Problem Choose Best Plan PlansCPUStorage P1Site A P2Site BSite A ……… Task workflow Plan Enumeration Cost T1 T2 … Cost: Plan Execution Time Challenge: Need cost models to estimate plan execution time

Generating Cost Models is Hard Non-declarative –Scientific workflow tasks are usually scripts (matlab, perl) –Such tasks are not database operators like join or select –Hence: task is a black box with no prior knowledge Heterogeneous resources –Computational grid setting –Performance varies a lot across resource assignments Data dependency –Performance can vary significantly based on properties of input data & parameters to scripts

Problem Setting Scientific workflows at DSCR (Duke Shared Cluster Resource) Important scientific workflows are run repeatedly –Opportunity to observe & learn task behavior –Better plan selection for subsequent runs Sequential scientific workflows –Each task runs on a single node –>90% of workflows at DSCR are sequential

NIMO System NonInvasive Modeling for Optimization NIMO learns cost models for task workflows –End-to-end cost models Incorporate properties of tasks, resources, & data –Non-invasive No changes to tasks –Automated and active Automatically collects training data for learning cost models C3C3 C1C1 C2C2 Site A Site B Site C Scheduler NIMO NIMO System NonInvasive Modeling for Optimization

NIMO Fills a Gap WorkFlow Management Systems (WFMSs) –WFMSs use database technology for managing all aspects of scientific workflows [Liu ‘04, Shankar ‘05] Batch scheduling systems –Knowledge of plan execution time is assumed for optimizing resource assignments [Casanova ‘00, Phan ‘05, Kelly ‘03] NIMO generates cost models for these systems

Roadmap Cost models NIMO: active learning of cost models Experimental evaluation Related work Conclusions Future work

Cost Model Task Execution time Resource assignment Cost Model for Task Input data Total workflow execution time can be derived using the cost models for individual tasks Task workflow

O a (compute occupancy) O s (stall occupancy) Task Cost Model compute phase (compute resource busy) stall phase (compute resource stalled on I/O) O d (storage occupancy) O n (network occupancy) ++ ) ( T=D * total data exec. time occupancy: average time spent per unit of data

Cost Model Task Execution time Resource assignment Cost Model Input data T = D * (O a + O n + O d ) Resource profile Data profile Task profile

Learning Cost Models Learning the cost model = Learning profiles + Learning predictors

Independent variables Resource profile ( ) Data profile ( ) Statistical Learning of Predictors Dependent variables Ex: Learn each predictor as a regression model from the training data

Challenges in Learning Cost of sample acquisition Coverage of system operating range Curse of dimensionality –Suppose: 10 profile attributes X 10 values per attribute, and 5 minutes for a task run (sample)  We sample 1% of space and build cost model Passive learning Elapsed Time Accuracy of current best model 951 years! Active & Accelerated Learning Best accuracy possible

Active (and Accelerated) Learning Which predictors are important? Which profile attributes should each predictor have? What values to consider for each profile attribute during training? Resource profile Data profile

WAN emulator (nistnet) NIMO workbench Training set database Active & Accel. learning C3C3 C1C1 C2C2 Site A Site B Site C Scheduler NIMO System Task profiler Resource profiler Run standard benchmarks Data profiler

Active Learning Algorithm Initialization While( ) { }

Relearn predictors with the new set of training samples Compute current prediction error of each predictor –Fixed test set –Cross-validation Active Learning Algorithm Initialization While( ) { } Pick a new assignment Run task on chosen assignment Relearn predictors Relearn Predictors 10ms256M1GHz 1G 512MB 6 8 T4T4 4

Active Learning Algorithm Initialization While( ) { } Run task on chosen assignment Relearn predictors 10ms256M1GHz 1G 512MB 6 8 T4T4 4 Choose a predictor to refine Choose attributes for the predictor Choose attribute values for the run Predictor Choice Predictors – fa, fn, fd, fD Order predictors + Traverse this order –Ex: relevance-based order (Plackett-Burman) –Ex: choose predictor with current max. error

Active Learning Algorithm Initialization While( ) { } Run task on chosen assignment Relearn predictors 10ms256M1GHz 1G 512MB 6 8 T4T4 4 Choose a predictor to refine Choose attributes for the predictor Choose attribute values for the run Attribute Choice Each predictor takes profile attributes as input Not all attributes are equally relevant Order attributes + Traverse this order

Active Learning Algorithm Initialization While( ) { } Run task on chosen assignment Relearn predictors 10ms256M1GHz 1G 512MB 6 8 T4T4 4 Choose a predictor to refine Choose attributes for the predictor Choose attribute values for the run Value Choice Cover the operating range of attributes Expose main interactions with other attributes

Experimental Results Biomedical workflows (from DSCR) –BLAST, fMRI, NAMD, CardioWave –Single task workflows Plan space in the heterogeneous networked utility –5 CPU speeds, 6 Network latencies, 5 Memory sizes –5 X 6 X 5 = 150 resource plans Goal: Converge quickly to a fairly-accurate cost model –We use regression models for the predictors –Model validation details in previous work (ICAC 2005)

Performance Summary Error: Mean absolute % error in predicted execution time A separate test set for evaluating the error

BLAST Application: Predictor Choice

BLAST Application: Attribute Choice

Related Work Workflow Management Systems (WFMSs) –[Shankar ’05, Liu ’04 etc.] Performance prediction in scientific applications –[Carrington ’05, Rosti ’02, etc.] Learning cost models using statistical techniques –[Zhang ’05, Zhu ’96, etc.] NIMO is end-to-end, noninvasive, and active (acquires model learning data automatically)

Conclusions NIMO: –Learns cost models for scientific workflows –Noninvasive and end-to-end –Active and accelerated learning: Learns accurate cost models quickly –Fills a gap in Workflow Management Systems

NIMO + SHIRAKO –A policy-based resource- leasing system that can slice- and-dice virtualized resources NIMO + Fa –Processing system- management queries (e.g., root-cause diagnosis, forecasting performance problems, capacity-planning) C3C3 C1C1 C2C2 Site A Site B Site C Scheduler NIMO Future Work

Backup Slides for Explanation

See Paper for Details of Steps Each algorithm step has sub-algorithms Example: Choosing the predictor to refine in current step –Goal: learn most relevant predictors first –Static Vs. dynamic ordering Static: –Define total order: a priori or using estimates of influence (Plackett-Burman) –Traverse the order: round-robin Vs. improvement-threshold-based Dynamic: choose the predictor with maximum current prediction error

Active and Accelerated Learning

Latency hiding

Saturation