1. June 1, 2015 1 GrenchMark : Towards a Generic Framework for Analyzing, Testing, and Comparing Grids ASCI Conference 2006 A. Iosup, D.H.J. Epema PDS Group, ST/EWI, TU Delft

2. June 1, 2015 2 Outline Introduction and Motivation The GrenchMark Framework Past and Current Experience with GrenchMark A GrenchMark Success Story Future Work Conclusions

3. June 1, 2015 3 The Generic Problem of Analyzing, Testing, and Comparing Grids Use cases for automatically analyzing, testing, and comparing grids Comparisons for system design and procurement Functionality testing and system tuning Performance testing/analysis of grid applications … For grids, this problem is hard ! Testing in real environments is difficult Grids change rapidly Validity of tests … Grid large-scale collection of federated resources, e.g., CPUs, HDDs, …

4. June 1, 2015 4 A Generic Solution to Analyzing, Testing, and Comparing Grids “ Generate and run synthetic grid workloads, based on real and synthetic applications “ Current alternatives (not covering all problems) Benchmarking with real/synthetic applications (representative?) User-defined test management (statistically sound?) Advantages of using synthetic grid workloads Statistically sound composition of benchmarks Statistically sound test management Generic: cover the use cases’ broad spectrum (to be shown)

5. June 1, 2015 5 GrenchMark: Towards a Generic Framework for Analyzing, Testing, and Comparing grids What’s in a name? grid benchmark → working towards a generic tool for the whole community: help standardizing the testing procedures, but benchmarks are too early; we use synthetic grid workloads instead What’s it about? A systematic approach to analyzing, testing, and comparing grid settings, based on synthetic workloads A set of metrics for analyzing grid settings A set of representative grid applications Both real and synthetic Easy-to-use tools to create synthetic grid workloads Flexible, extensible framework

6. June 1, 2015 6 GrenchMark: Iterative Research Roadmap

7. June 1, 2015 7 GrenchMark: Iterative Research Roadmap Simple functional system A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID WS Integrated Research in Grid Computing, Nov 2005.

8. June 1, 2015 8 GrenchMark: Iterative Research Roadmap Open- GrenchMark Community Effort Complex extensible system This work

9. June 1, 2015 9 GrenchMark Overview: Easy to Generate and Run Synthetic Workloads

10. June 1, 2015 10 … but Maybe More Complicated Than You Think Workload structure User-defined and statistical models Dynamic jobs arrival Burstiness and self-similarity Feedback, background load Machine usage assumptions Users, VOs Interactive jobs, malleable, moldable, … Metrics A(W) Run/Wait/Resp. Time Efficiency, MakeSpan Failure rate, waste (Grid) notions Co-allocation, workflow Measurement methods Long workloads Saturated / non-saturated system Start-up, production, and cool-down scenarios Scaling workload to system Applications Synthetic Real Workload definition language Base language layer Extended language layer Other Can use the same workload for both simulations and real environments

11. June 1, 2015 11 GrenchMark Overview: Unitary and Composite Applications Composite applications Bag of tasks Chain of jobs Direct Acyclic Graph-based (Standard Task Graph Archive) Unitary applications sequential, MPI, Java RMI, Ibis, …

12. June 1, 2015 12 GrenchMark Overview: Workload Description Files Format: Combining four workloads into one Number of jobs Composition and application types Co-allocation and number of components Inter-arrival and start time Language extensions

13. June 1, 2015 13 Using GrenchMark: Grid System Analysis Performance testing: test the performance of an application (for sequential, MPI, Java/Ibis applications) Report runtimes, waiting times, grid middleware overhead Automatic results analysis What-if analysis: evaluate potential situations System change Grid inter-operability Special situations: spikes in demand

14. June 1, 2015 14 Using GrenchMark: Functionality Testing in Grid Environments System functionality testing: show the ability of the system to run various types of applications Report failure rates [ arguably, functionality in grids is even more important than performance !  10% job failure rate in a controlled system like the DAS ] Periodic system testing: evaluate the current state of the grid Replay workloads

15. June 1, 2015 15 Using GrenchMark: Comparing Grid Settings Single-site vs. co-allocated jobs: compare the success rate of single-site and co-allocated jobs, in a system without reservation capabilities Single-site jobs 20% better vs. small co-allocated jobs (<32 CPUs), 30% better vs. large co-allocated jobs [setting and workload-dependent !] Unitary vs. composite jobs: compare the success rate of unitary and composite jobs, with and without failure handling mechanisms Both 100% with simple retry mechanism [setting and workload-dependent !]

16. June 1, 2015 16 A GrenchMark Success Story: Releasing the Koala Grid Scheduler on the DAS Koala [ http://www.st.ewi.tudelft.nl/koala/ ] http://www.st.ewi.tudelft.nl/koala/ Grid Scheduler with co-allocation capabilities DAS: The Dutch Grid, ~200 researchers Initially Koala, a tested (!) scheduler, pre-release version Test specifics 3 different job submission modules Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… Evaluate: job success rate, Koala overhead and bottlenecks Results 5,000+ jobs successfully run (all workloads); functionality tests 2 major bugs first day, 10+ bugs overall (all fixed) KOALA is now officially released on the DAS (full credit to KOALA developers, 10x for testing with GrenchMark)

17. June 1, 2015 17 A. Iosup, D.H.J. Epema (TU Delft), C.Franke, A. Papaspyrou, L. Schley, B. Song, R. Yahyapour (U Dortmund), On Grid Performance Evaluation using Synthetic Workloads, 12th Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), held in conjunction with SIGMETRICS, Saint Malo, France, June 2006 (accepted). GrenchMark’s Current Status: pre-”Open-GrenchMark” Already done in Python [http://www.python.org] Workload Generator Generic Workload Submitter (Koala, Globus GRAM, option to extend for JSDL, Condor, PBS, LSF, SGE, …) Applications Unitary, 3 types: sequential, MPI, Ibis (Java) +35 real and synthetic applications Composite applications: DAG-based Extending modeling capabilities

18. June 1, 2015 18 A. Iosup, C. Dumitrescu, D.H.J. Epema (TU Delft), H. Li, L. Wolters (U Leiden), How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications, Tech.Rep. TU Delft/PDS/2006-003, Apr 2006. Towards Open-GrenchMark: Grid traces, Simulators, Benchmarks Distributed testing Integrate with DiPerF (C. Dumitrescu, I. Raicu, M. Ripeanu) Grid traces analysis Automatic tools for grid traces analysis Use in conjunction with simulators Ability to generate workloads which can be used in simulated environments (e.g., GangSim, GridSim, …) Grid benchmarks Analyze the requirements for domain-specific grid benchmarks

19. June 1, 2015 19 Conclusion GrenchMark generates diverse grid workloads easy-to-use, flexible, portable, extensible, … Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to analyze, test, and compare grid settings. 15,000+ jobs generated and run … and counting. (more) advertisement Have specific grid settings you would like to test? Test with GrenchMark!

20. June 1, 2015 20 Thank you! Questions? Remarks? Observations? All welcome! GrenchMark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo] Alexandru IOSUP http://grenchmark.st.ewi.tudelft.nl/ TU Delft A.Iosup@tudelft.nl http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”] Many thanks to Hashim Mohamed (Koala), Jason Maassen and Rob van Nieuwpoort (Ibis).

21. June 1, 2015 21 Representative Grid applications (1/4) Unitary applications Just one scheduling unit (otherwise recursive definition) Examples: Sequential, MPI, Java RMI, Ibis, … Composite applications Composed of several unitary or composite applications Examples: Parameter sweeps, chains of tasks, DAGs, workflows, …

22. June 1, 2015 22 Representative Grid applications (2/4) Unitary: synthetic sequential [I] read I1 data elements from stdin [O] write O1 data elements to stdout for each N steps (i) # superstep [I] read I2 data elements from stdin for each M memory locations (j) # computation step [M] get item j of size S [P] compute C computation units per memory item (fmul), and store results into temp memory location [M] put values from temp to location j [O] write O2 data elements to stdout [O] write O3 data elements to stdout > System reqs Processor (float*N*S*C) Memory (float*(M+1)*S) N+1:I1,N*I2 stdin stdout N+2:O1,N*O2,O3 Also version with computation and I/O

23. June 1, 2015 23 Representative Grid applications (2/4) Unitary: synthetic parallel [I] read I1 data from in$ProcID [O] write O1 data to out$ProcID for each N steps (i) # superstep [B] synchronize with barrier [I] read I2 data from in$ProcID for each M memory locations (j) [M] get item j of size S [P] compute C fmul/mem [M] put values from temp to j [C] communicate to all other processes X values and receive X values from every other processor [O] write O2 data elements to out$ProcID [O] write O3 data elements to out$ProcID > Machine i Processor i k (float*N*S*C) Memory (float*N*2S) N+1: I1,N*I2 app-input.iapp-output.i N+2: O1,N*O2,O3 Process i

24. June 1, 2015 24 Representative Grid applications (3/4) Unitary: Ibis jobs No modeling, just real applications [ http://www.cs.vu.nl/ibis ]http://www.cs.vu.nl/ibis physical simulation parallel rendering computational mathematics state space search bioinformatics data compression grid methods optimization

25. June 1, 2015 25 Representative Grid applications (3/4) Composite: DAG-based DAG-based applications Real DAG Chain of tools Try to model real or predicted (use) cases Input Output User task Linker Identity (one task’s output = other’s input, unmodified) App1 > Linker1 > App2 > Final result > out_1-2.dat param1.in out_1-1.dat huge-data.out perf2.dat param2.insome-list.in > out2.res l1p.dat perf1.dat