1. June 25, 2015 1 GrenchMark: A synthetic workload generator for Grids KOALA Workshop A. Iosup, H. Mohamed, D.H.J. Epema PDS Group, ST/EWI, TU Delft

2. June 25, 2015 2 About productivity and efficiency… We now have a good scheduler for the DAS (Koala) How do we make use of it every day?

3. June 25, 2015 3 Productivity, efficiency 1.Run 1 job run 1 command ET (every time)

4. June 25, 2015 4 Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines script @1 (once), run 1 command @1

5. June 25, 2015 5 Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines script @1 (once), run 1 command @1 3.Run 100 jobs, parameterize runs write 25 lines script @1, run 1 command @1

6. June 25, 2015 6 Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines script @1 (once), run 1 command @1 3.Run 100 jobs, parameterize runs write 25 lines script @1, run 1 command @1 4.(Re-)Run (the same) 100 jobs, parameterize runs write 500 lines script @1, run 1 command @1

7. June 25, 2015 7 Productivity, efficiency 6.Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines script @1, run 1 command @1

8. June 25, 2015 8 Productivity, efficiency 6.Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines script @1, run 1 command @1 7. … and this was per test / project / article

9. June 25, 2015 9 Problem statement (revisited) Synthetic worklods are the basis of grid schedulers comparison [our work], and of... Functionality testing in the DAS (Early) performance testing for application development Comparing alternative settings for the DAS Many others … Need a standard tool for synthetic workloads generation and submission

10. June 25, 2015 10 The GrenchMark framework What’s in a name? grid benchmark → help standardizing the testing procedure, but benchmarks are too early… GrenchMark Easy-to-use tools to create synthetic workloads Easy-to-use tools to submit synthetic workloads Flexible, portable, extensible

11. June 25, 2015 11 The GrenchMark framework

12. June 25, 2015 12 The GrenchMark process: Describe $ cat wl-desc.in # File-type: text/wl-spec #Times UnitType SiteType SiteInfo ArrivalTimeDistrOtherInfo 50 ssert non-fixed*:*U(0,100) - 10 cpi-gm fixed fs1;…Poisson(60) NProc=1;2;4;… 35 cpi-g2 fixed *:*Poisson(60) ExternalFile= my.data wl-desc.in describes the workload to be generated $ cat my.data NComp=10/50; 4/20; 2/10; 5/10; 3 NProc=1/10; 2/10; 4/10; 8/10; 16/10; 32/10; 3;… … my.data describes the characteristics of a part of the workload

13. June 25, 2015 13 $./wl-gen.py wl-desc.in The GrenchMark process: Generate Build directory structure Generate Times x Jobs JDFs 2 Sample run: 95 jobs / 411 components 95 files / 32 directories 300KB data

14. June 25, 2015 14 The GrenchMark process: Submit $./wl-gen.py wl-desc.in $./wl-submit.py out/wl-to-submit.wl 3 3 TODO Sample run: multi-threaded – submission delay +/- 0.01s 95 JDFs

15. June 25, 2015 15 GrenchMark status Already done in Python Generator + Globus, KOALA generators + RSL printer Submitter (Koala, Globus, …) Results analyzer (crude) Support for applications Unitary, 3 types: sequential, MPI, Ibis (Java) +35 different applications Composite applications: DAG-based Ongoing work Automated results analyzer Better modeling capabilities

16. June 25, 2015 16 Extending GrenchMark Motto: Extending GrenchMark is easy! 1.More workload generators a function with a predefined name in a Python module auto-loaded 2.Interface with C/C++, Ruby, Perl, Java, … define your own protocol and write the Python end 3.Interface with other systems (SGE, …) a function with a predefined name in a Python module auto-loaded

17. June 25, 2015 17 GrenchMark for testing K OALA … K OALA Non-fixed, fixed, semi-fixed jobs Test & evaluate Status: pre-release, supposed stable Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… Jobs success rate, K OALA ’s overhead and bottlenecks Results 2 major bugs first day, +10 bugs overall (all fixed) K OALA is officially released (full credit to KOALA developers, 10x for testing with GrenchMark) +5,000 jobs successfully run

18. June 25, 2015 18 … and DAS-2’s functionality … Already done Evaluate for KOALA + Globus + DAS-2 jobs success rate, turnaround time, middleware overhead, types and sources of errors Sample results 5 workloads 500 jobs A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, In Proc. of CoreGRID IW, Nov 2005. +1,000 jobs successfully run

19. June 25, 2015 19 … and for analyzing, testing, and comparing grids Grid system analysis Performance testing / analysis What-if analysis Functionality testing in grid environments System functionality testing Periodic testing Comparing grid settings Single-site vs. co-allocated jobs Unitary vs. composite jobs A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, In CCGrid 2006 (accepted). +2,500 jobs successfully run

20. June 25, 2015 20 Conclusions GrenchMark generates and submits synthetic workloads easy-to-use, flexible, portable, extensible, … Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS/Grid functionality. +5,000 jobs generated and run … and counting. … advertisement Have specific Grid applications types you would like to test? Test with GrenchMark!

21. June 25, 2015 21 Thank you! Questions? Remarks? Observations? All welcome! Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [ “grenchmark”] Alexandru IOSUP http://grenchmark.st.ewi.tudelft.nl/ TU Delft A.Iosup@tudelft.nl [ “iosup”] Also: H. Mohamed and W. Lammers (K OALA ), K. Verstoep and P. Anita (DAS admins), R. van Nieuwpoort and J. Maassen (Ibis)

22. June 25, 2015 22 Outline Introduction Grenchmark Using Grenchmark Customizing Grenchmark Extending Grenchmark Conclusions [done] [here]