1. July 13, 2015 1 GrenchMark: A workload generator for Grids Demo at TU Delft A. Iosup, D.H.J. Epema PDS Group, ST/EWI, TU Delft

2. July 13, 2015 2 Evaluating Grid schedulers performance Performance Qualitative metrics supported application types, advanced fault tolerance, advanced... Quantitative metrics resource consumption, system performance, scheduler functionality Other metrics cost, single number Needs Applications, workloads, more workloads… Benefits Configuration testing and tuning Functionality testing System design and procurement Performance testing Runtime estimation

3. July 13, 2015 3 GrenchMark A systematic approach to testing Grid schedulers A set of metrics for comparing schedulers A set of representative applications Both real and synthetic Sequential tasks, MPI tasks, Java, IBIS/Satin tasks, batch tasks, filters/streaming tasks, chains of tasks, DAGs, general graphs,...(?) Easy tools to create synthetic workloads [TODAY] Testing, testing, testing… [EVERY DAY]

4. July 13, 2015 4 The GrenchMark process

5. July 13, 2015 5 Outline Introduction Grenchmark Using Grenchmark Customizing Grenchmark Extending Grenchmark Conclusions [done] [here]

6. July 13, 2015 6 wl-desc.in Describes the workload to be generated; input file for wl-gen.py Fixed, non-fixed, semi-fixed execution sites Applications inter-arrival time: Poisson, uniform, normal, exp, hyperexp2, Weibull, log normal, gamma, … Over 35 supported applications, in 3 categories: sequential, MPI, Ibis

7. July 13, 2015 7 Grenchmark process $./wl-gen.py wl-desc.in $./wl-submit.py out/wl-to-submit.wl 1 2 1 2 TODO Sample run: 95 jobs / 411 components 95 files / 32 directories 300KB data Sample run: multi-threaded – submission delay +/- 0.01s 95 JDFs

8. July 13, 2015 8 Preliminary notions Job, workload, workload unit Job = Set of components (co-allocation support) [Job = one program execution / the basic scheduling unit / … ] Workload = Set of Jobs Workload Unit = Set of jobs with the same property, generated in the same step (definition useful only for workload generator) Other JDF = Job description file Inter arrival time = time between two consecutive jobs arrive in the system

9. July 13, 2015 9 Describing the workload: Jobs wl-desc.in ID Times UnitType SiteType TotalComps PerSiteDesc [I]ArrivalTimeDistr OtherInfo 0 1 sser single 1 *:? U(0,1000) - Generate Times x Jobs or Times x Sets of Jobs One Job = k Components (here: 1 -- single)

10. July 13, 2015 10 Demo 1: Generating one job Single site, single instance

11. July 13, 2015 11 Describing the workload: Jobs wl-desc.in ID Times UnitType SiteType TotalComps PerSiteDesc [I]ArrivalTimeDistr OtherInfo 0 1 sser single 1 *:? U(0,1000) - Generate Times x Jobs or Times x Sets of Jobs One Job = k Components (here: 1 -- single)

12. July 13, 2015 12 Describing the workload: Units wl-desc.in ID Times UnitType SiteType TotalComps PerSiteDesc ArrivalTimeDistr OtherInfo ? 1 wl:poisson - - - Poisson(500) File=wl.xin wl.xin RandomWorkload=false NComponents=4;8 ComponentSize=8;16 Submitter=drunner Application=pi-g2-gm Generate Times x Jobs or Times x Sets of Jobs wl: prefix – workload unit generator [Hashim’s jobs]

13. July 13, 2015 13 Demo 4: Generating one workload unit Hashim’s sample workload: Poisson-g2-gm (Globus MPICH-G2) Components: 4 and 8 Component Sizes: 8 and 16 IAT Poisson(5s) Run with drunner Total: 4 jobs, 24 components

14. July 13, 2015 14 Demo 5: Generating a full workload Hashim’s full workload, 8 times: Poisson-g2-gm (Globus MPICH-G2) and findpi-gm Components: 4 and 8 Component Sizes: 4/8 and 16 IAT Poisson(5s) Run with drunner (poisson) / krunner (pi) Submit time 1 day Generate: Submit: Total: 3200 jobs, 19200 components / 3k files, 4k dirs Timing: 30s generate / 86,400s submit (1 day, eh?) $ wl-gen.py --duration=86400000 wl-desc.in $ wl-submit.py --onefile out/wl-desc.in

15. July 13, 2015 15 Outline Introduction Grenchmark Using Grenchmark Customizing Grenchmark Extending Grenchmark Conclusions [done] [here] [done]

16. July 13, 2015 16 Customizing the workload generator Command-line parameters wl-gen.py –help … displays a comprehensive help screen wl-gen.py –o outdir … creates the workload in directory outdir wl-gen.py –d 50000 … sets the total submission duration to 50s wl-gen.py –j my-jdf … uses my-jdf to write JDFs wl-gen.py –o out-1 wl-desc.in my favourite command

17. July 13, 2015 17 Customizing the submission tool Command-line parameters wl-submit.py –help … displays a comprehensive help screen wl-submit.py –t 10 … uses 10 threads to submit wl-submit.py --nosubmit … just prints the submission commands wl-submit.py --onefile … output job submit (runner) stderr and stdout to one file each (concatenated) wl-submit.py out/wl-to-submit.wl my favourite command

18. July 13, 2015 18 Outline Introduction Grenchmark Using Grenchmark Customizing Grenchmark Extending Grenchmark Conclusions [done] [here] [done]

19. July 13, 2015 19 Extending the workload generator (1) Write your own Job Generators 1.Create your-job-generator.py in apps/ 2.Place a Generator attribute, organized as a Key=Value list, one per line, with at least Name=your-job-type defined. Example: Generator=""" Name=your-job-type Author=You R. Name Copyright=copyright © 2005 You R. Name. All rights reserved. ""“ 3.Create a function generateWorkloadUnit, which takes in UnitDir, UnitID, WLUnit, SubmitDurationMS, bGenerateRandom 4.Make generateWorkloadUnit return a dictionary with keys ‘info’ and ‘jobs’, with appropriate data each 5.Use the generator in wl-desc.in by putting your-job-type in the UnitType field

20. July 13, 2015 20 Extending the workload generator (2) Write your own Unit Generators 1.Create your-job-generator.py in apps/ 2.Place a WLGenerator attribute, organized as a Key=Value list, one per line, with at least Name=your-job-type defined. Example: WLGenerator=""" Name=your-workload-type Author=You R. Name Copyright=copyright © 2005 You R. Name. All rights reserved. ""“ 3.Create a function generateWorkload, which takes in UnitDir, UnitID, WLUnit, SubmitDurationMS, bGenerateRandom 4.Make generateWorkloadUnit return a dictionary with keys ‘info’ and ‘jobs’, with appropriate data each 5.Use the generator in wl-desc.in by putting your-job-type in the UnitType field

21. July 13, 2015 21 Extending the workload generator (2) Unit Generators in C, Ruby, Perl,... ? 1.Same as previous: generate your-job-generator.py in apps/ 2.Replace step 4 with: 1.Interface your Python code, so that it runs your C, C++, Ruby, Perl, … application (os.exec, os.popen, …) 2.Save your C, C++, Ruby, Perl, … application information to a text file, or use pipes to transfer info 3.Read the saved information.

22. July 13, 2015 22 Extending the workload generator (3) Write your own JDF Writers 1.Create your-jdf-writer.py in jdfprint/ 2.Place a JDFGenerator attribute, organized as a Key=Value list, one per line, with at least Name=your-job-type defined. Example: JDFGenerator=""" Name=your-jdf-writer-type Author=You R. Name Copyright=copyright © 2005 You R. Name. All rights reserved. ""“ 3.Create a function generateJDF, which takes in OutFileName, ListOfComponents 4.Make generateJDF write the components to the specified file, with appropriate data each 5.Use the generator by calling wl-gen.py with the –j parameter

23. July 13, 2015 23 Conclusions and future work Grenchmark easily generates diverse workloads of Grid applications KOALA developers successfully used GrenchMark to test KOALA Future work do you have specific Grid applications types that you would like to test? Try Grenchmark!

24. July 13, 2015 24 Thank you! Questions? Remarks? Observations? All welcome! Grenchmark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo] Alexandru IOSUP http://grenchmark.st.ewi.tudelft.nl/ TU Delft A.Iosup@ewi.tudelft.nl http://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”]