June 25, GrenchMark: A synthetic workload generator for Grids KOALA Workshop A. Iosup, H. Mohamed, D.H.J. Epema PDS Group, ST/EWI, TU Delft

June 25, About productivity and efficiency… We now have a good scheduler for the DAS (Koala) How do we make use of it every day?

June 25, Productivity, efficiency 1.Run 1 job run 1 command ET (every time)

June 25, Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines (once), run 1

June 25, Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines (once), run 1 3.Run 100 jobs, parameterize runs write 25 lines run 1

June 25, Productivity, efficiency 1.Run 1 job run 1 command ET (every time) 2.Run 10 jobs write 10 lines (once), run 1 3.Run 100 jobs, parameterize runs write 25 lines run 1 4.(Re-)Run (the same) 100 jobs, parameterize runs write 500 lines run 1

June 25, Productivity, efficiency 6.Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines run 1

June 25, Productivity, efficiency 6.Run “100 times each of: 100 jobs, parameterized, 1 to 5 components per job, preference for 2 and 4 components, 2 to 128 processors per job, preference for 2, 4, 16 and 32 processors, arrival rate Poisson or Alpha-Beta or mixed, stop runs after 15 or 60 minutes, …” write 5000 lines run 1 7. … and this was per test / project / article

June 25, Problem statement (revisited) Synthetic worklods are the basis of grid schedulers comparison [our work], and of... Functionality testing in the DAS (Early) performance testing for application development Comparing alternative settings for the DAS Many others … Need a standard tool for synthetic workloads generation and submission

June 25, The GrenchMark framework What’s in a name? grid benchmark → help standardizing the testing procedure, but benchmarks are too early… GrenchMark Easy-to-use tools to create synthetic workloads Easy-to-use tools to submit synthetic workloads Flexible, portable, extensible

June 25, The GrenchMark framework

June 25, The GrenchMark process: Describe $ cat wl-desc.in # File-type: text/wl-spec #Times UnitType SiteType SiteInfo ArrivalTimeDistrOtherInfo 50 ssert non-fixed*:*U(0,100) - 10 cpi-gm fixed fs1;…Poisson(60) NProc=1;2;4;… 35 cpi-g2 fixed *:*Poisson(60) ExternalFile= my.data wl-desc.in describes the workload to be generated $ cat my.data NComp=10/50; 4/20; 2/10; 5/10; 3 NProc=1/10; 2/10; 4/10; 8/10; 16/10; 32/10; 3;… … my.data describes the characteristics of a part of the workload

June 25, $./wl-gen.py wl-desc.in The GrenchMark process: Generate Build directory structure Generate Times x Jobs JDFs 2 Sample run: 95 jobs / 411 components 95 files / 32 directories 300KB data

June 25, The GrenchMark process: Submit $./wl-gen.py wl-desc.in $./wl-submit.py out/wl-to-submit.wl 3 3 TODO Sample run: multi-threaded – submission delay +/- 0.01s 95 JDFs

June 25, GrenchMark status Already done in Python Generator + Globus, KOALA generators + RSL printer Submitter (Koala, Globus, …) Results analyzer (crude) Support for applications Unitary, 3 types: sequential, MPI, Ibis (Java) +35 different applications Composite applications: DAG-based Ongoing work Automated results analyzer Better modeling capabilities

June 25, Extending GrenchMark Motto: Extending GrenchMark is easy! 1.More workload generators a function with a predefined name in a Python module auto-loaded 2.Interface with C/C++, Ruby, Perl, Java, … define your own protocol and write the Python end 3.Interface with other systems (SGE, …) a function with a predefined name in a Python module auto-loaded

June 25, GrenchMark for testing K OALA … K OALA Non-fixed, fixed, semi-fixed jobs Test & evaluate Status: pre-release, supposed stable Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… Jobs success rate, K OALA ’s overhead and bottlenecks Results 2 major bugs first day, +10 bugs overall (all fixed) K OALA is officially released (full credit to KOALA developers, 10x for testing with GrenchMark) +5,000 jobs successfully run

June 25, … and DAS-2’s functionality … Already done Evaluate for KOALA + Globus + DAS-2 jobs success rate, turnaround time, middleware overhead, types and sources of errors Sample results 5 workloads 500 jobs A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, In Proc. of CoreGRID IW, Nov ,000 jobs successfully run

June 25, … and for analyzing, testing, and comparing grids Grid system analysis Performance testing / analysis What-if analysis Functionality testing in grid environments System functionality testing Periodic testing Comparing grid settings Single-site vs. co-allocated jobs Unitary vs. composite jobs A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, In CCGrid 2006 (accepted). +2,500 jobs successfully run

June 25, Conclusions GrenchMark generates and submits synthetic workloads easy-to-use, flexible, portable, extensible, … Experience used GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS/Grid functionality. +5,000 jobs generated and run … and counting. … advertisement Have specific Grid applications types you would like to test? Test with GrenchMark!

June 25, Thank you! Questions? Remarks? Observations? All welcome! Grenchmark [ “grenchmark”] Alexandru IOSUP TU Delft [ “iosup”] Also: H. Mohamed and W. Lammers (K OALA ), K. Verstoep and P. Anita (DAS admins), R. van Nieuwpoort and J. Maassen (Ibis)

June 25, Outline Introduction Grenchmark Using Grenchmark Customizing Grenchmark Extending Grenchmark Conclusions [done] [here]