1. VDT 1 The Virtual Data Toolkit Todd Tannenbaum (Alain Roy)

2. VDT 2 What is the VDT? A packaging of software –Grid software (Globus, Condor-G…) –Virtual data software (Chimera) –Utilities An easy installation mechanism Testing and hardening Support

3. VDT 3 Who makes the VDT? Grid Physics Network (GriPhyN) –Constructs the VDT International Virtual Data Grid Laboratory (IVDGL) –Testing and hardening Very tight collaboration between GriPhyN and IVDGL

4. VDT 4 Who makes the VDT? (2) Core VDT Team: –Miron Livny: The boss –Alain Roy –Carey Kireyev VDT Testing –Xin Zhao –Brian Moe Pacman –Saul Youssef

5. VDT 5 Who uses the VDT? GriPhyN collaborators –USCMS:In use today –USAtlas:In use today –LIGO:Will use soon –SDSS:Will use soon European Data Grid –Uses subset of software –Uses just RPMs LCG

6. VDT 6 What exactly is in VDT? VDT 1.1.8: –Globus 2.2.4 + advisories + patches –Condor & Condor-G 6.5.1 –Chimera/Pegasus –RLS –GLUE Schema –CA Certificates –Fault Tolerant Shell –EDG’s Make Gridmap –EDG’s CRL Update –ClassAds –Netlogger

7. VDT 7 What exactly is in VDT? VDT 1.1.8: –Globus 2.2.4 + advisories + patches –Condor & Condor-G 6.5.1 –Chimera/Pegasus –RLS –GLUE Schema –CA Certificates –Fault Tolerant Shell –EDG’s Make Gridmap –EDG’s CRL Update –ClassAds –Netlogger

8. VDT 8 Grid Software Installation Typical Grid Software Installation Experience… VDT Installation Experience!

9. VDT 9 VDT Installation 2 Methods –Pacman –RPM

10. VDT 10 Pacman Installation Goal: –Type a single command –Everything downloads –Everything installs –Everything is configured –No questions asked We’re close: –A few questions if you’re root –Basic configuration, may need changing

11. VDT 11 Pacman Installation (2) Download Pacman –http://physics.bu.edu/~youssef/pacman/ Install VDT –cd –pacman -get VDT-Server –pacman -get VDT-Client –ls condor/ globus/ post-install/ setup.sh edg/ gpt/ replica/ vdt/ ftsh/ perl/ setup.csh vdt-install.log Use

12. VDT 12 Pacman post-installation Post-install directory: –Notes on configuration choices made –Instructions for editing configuration Configuration scripts: –Globus configuration –Condor configuration

13. VDT 13 RPM Installation Subset of whole VDT –Globus –Condor-G Nice RPMs: –We repackage Globus –A dozen Globus RPMs, not hundreds No configuration No post-installation help

14. VDT 14 Testing VDT team is building test suite Interaction with LCG testing group Working with NMI* to leverage: –NMI test suite Stress testing Application testing (CMS pipeline) –NMI test infrastructure *NMI = NSF Middleware Initiative –http://www.nsf-middleware.org

15. VDT 15 Support Send us questions or problems –We will solve them if we can –We will interact with the developers, if necessary

16. VDT 16 Interaction with EDG EDG gets Globus and Condor-G RPMs from VDT We do what we can to solve problems and get changes to Globus and Condor We want to make a great package for you

17. VDT 17 What exactly is in VDT? VDT 1.1.8: –Globus 2.2.4 + advisories + patches –Condor & Condor-G 6.5.1 –Chimera/Pegasus –RLS –GLUE Schema –CA Certificates –Fault Tolerant Shell –EDG’s Make Gridmap –EDG’s CRL Update –ClassAds –Netlogger

18. VDT 18 Chimera Virtual Data System Much scientific data is not obtained from measurements but rather derived from other data by the application of computational procedures Chimera catalog can be used by application environments to describe a set of application programs ("transformations"), and then track all the data files produced by executing those applications ("derivations"). Chimera contains the mechanism to locate the "recipe" to produce a given logical file, in the form of an abstract program execution graph. These abstract graphs are then turned into and executable DAG for the Condor-G DAGMan meta-scheduler by the bundled Pegasus planner. Enables on-demand execution of computation schedules constructed from database queries.

19. VDT 19 NetLogger “Networked Application Logger” API w/ calls you add to existing source code to generate time- stamped monitoring events (sent to a file, network server, syslogd, or RAM) Visualization Tools Storage and Retrieval Tools –Store all events into a database

20. VDT 20 Fault Tolerant Shell (FTSH) The Grid is a hard environment. FTSH –The ease of scripting with very precise error semantics. –Exception-like structure allows scripts to be both succinct and safe. –A focus on timed repetition simplifies the most common form of recovery in a distributed system. –A carefully-vetted set of language features limits the "surprises" that haunt system programmers.

21. VDT 21 Simple Bourne script… #!/bin/sh cd /work/foo rm –rf data cp -r /fresh/data. What if ‘/work/foo’ is unavailable??

22. VDT 22 Getting Grid Ready… #!/bin/sh for attempt in 1 2 3 cd /work/foo if [ ! $? ] then echo "cd failed, trying again..." sleep 5 else break fi done if [ ! $? ] then echo "couldn't cd, giving up..." return 1 fi

23. VDT 23 Or with FTSH #!/usr/bin/ftsh try 5 times cd /work/foo rm -rf bar cp -r /fresh/data. end

24. VDT 24 Or with FTSH #!/usr/bin/ftsh try for 3 days or 100 times cd /work/foo rm -rf bar cp -r /fresh/data. end

25. VDT 25 Or with FTSH #!/usr/bin/ftsh try for 3 days every 1 hour cd /work/foo rm -rf bar cp -r /fresh/data. end

26. VDT 26 Or with FTSH #!/usr/bin/ftsh try for 3 days every 1 hour cd /work/foo rm -rf bar cp -r /fresh/data. end

27. VDT 27 Or with FTSH #!/usr/bin/ftsh try for 3 days every 1 hour cd /work/foo rm -rf bar cp -r /fresh/data. end

28. VDT 28 Or with FTSH hosts="mirror1.wisc.edu mirror2.wisc.edu mirror3.wisc.edu" forany h in ${hosts} echo "Attempting host ${host}" wget http://${h}/some-file end echo "Got file from ${h}"

29. VDT 29 FTSH All the usual constructs –Redirection, loops, conditionals, functions, expressions, nesting, … And more –Logging –Timeouts –Process Cancellation –Complete parsing at startup –File cleanup Used on Linux, Solaris, Irix, Cygwin, … Simplify your life!

30. VDT 30 VDT’s Future Additional Software –MyProxy, Java ClassAds Access to new versions –Globus 3.0 Extra VDT to help early adopters Condor-G will submit to GT2 or GT3 Helping You –What can we do to make life easier for you?

31. VDT 31 Where do you learn more? http://www.griphyn.org/vdt Support: –vdt-support@ivdgl.org –Alain Roy:roy@cs.wisc.edu –Miron Livny:miron@cs.wisc.edu