1. C. Loomis – Title – Date – 1 European DataGrid Charles Loomis Seillac June 26, 2002 Outline EDG Introduction & Goals EDG Architecture Example Job Submission Future Developments Further Information

2. C. Loomis – Title – Date – 2 European DataGrid (EDG) European DataGrid  EU-funded, 3-year project (2001-3)  Goals: —develop grid middleware —deploy onto working testbed —demonstrate grid technology with working applications  Strong application component unique! EDG Organization WP1Workload Mgt. WP2Data Mgt. WP3Info. & Monitoring Sys. WP4Fabric Mgt. WP5Storage Mgt. WP6Testbed WP7Networking WP8HEP Apps. WP9Biomedical Apps. WP10Earth Ob. Apps. WP11Dissemination WP12Project Mgt. 6 Partners; 21 Associates

3. C. Loomis – Title – Date – 3 Computing Grid  Electrical Grid Accessing computing resources as simple as plugging in a toaster!  Fixed resources (CPU)  Mobile resource (Data)  Semi-mobile (Tape) Transparency (I.e. standards)  Simplifies sharing & trading of resources. —Does not imply that all resources are free.  Should be applicable from desktop to supercomputers. Security (Owners retain control!)  System administrators control machine access.  Who can read your data?  Software enforces access control. CPU Cycles Network Bandwidth Disk/tape Storage Data/Databases Software

4. C. Loomis – Title – Date – 4 Transparency for Physicists Single Login:  Access all available local & remote resources w/o a dedicated account. Consistent User Environment:  Same working environment everywhere--same software tools; same configuration. Brokering:  System finds “optimal” resources for task automatically.  Resource discovery, moving job to data, moving data to job. High-Level Selection of Resources:  Ex. choose storage by size/lifetime/reliability/cost/transfer speed; not disk/tape/shared file system/location. Security:  Does not impede authorized access to resources.  Data and computations are (reasonably) safe from tampering.

5. C. Loomis – Title – Date – 5 Interoperability & Interdependency Standards:  Global Grid Forum (like W3C)  Or from dominant implementation Plethora of Projects:  Sharing of code & expertise.  Competing for people, funding, etc.

6. C. Loomis – Title – Date – 6 EDG Architecture Global Batch System:  Centralized Architecture.  Heavy infrastructure.  Other choices  Peer-to-peer Computing Element Storage Element Site X Information Systems submit publish state query retrieve broker chooses optimal site for job Resource Broker User Interface

7. C. Loomis – Title – Date – 7 Hidden Infrastructure Certification Authorities Attest to your identity. Virtual Organizations Maintain membership lists. Used for group authorization. /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr Replica Catalog Location & Directory Info. for data. Maintained by: ALICE ATLAS BaBar CMS LHCb Biomedical Apps. Earth Observation

8. C. Loomis – Title – Date – 8 EDG Testbeds Production Testbed (1.1.4)  For applications to use & stress software in “semi-production” environment.  10 sites (6 countries) Development Testbed (1.2b5)  To facilitate testing and integration of new middleware.  4 sites (4 countries) SiteLocation CataniaCatania (I) CC-IN2P3Lyon (F) CERN*Geneva (CH) CNAF*Bologna (I) Imperial CollegeLondon (UK) LIP**Lisbon (PT) MSUMoscow (Russia) NIKHEF*Amsterdam (NL) PadovaPadova (I) RAL*Rutherford (UK) TorinoTorino (I)

9. C. Loomis – Title – Date – 9 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output #!/bin/sh # Get matched physical file name. filename = `./GetPFN.pl $EDG_WL_RB_BROKERINFO` # Analyze this file. count_evts $filename Find data file and count events.

10. C. Loomis – Title – Date – 10 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output # Executable to run and the arguments. Executable = "ChecksumRC.sh"; Arguments = "none"; # Define file names for standard output and error. StdOutput = "std.out"; StdError = "std.err"; # Files to transfer as input and output. InputSandbox = {"ChecksumRC.sh","GetPFN.pl”,”count_evts”}; OutputSandbox = {"std.out","std.err"}; # Desired file and protocol. InputData = {"LF:utve_v_1_71008101_12902_0609.lv0"}; DataAccessProtocol = {"file"}; # My experiment's "Data Catalog" ReplicaCatalog = "ldap://grid- vo.nikhef.nl:10389/rc=EarthObReplicaCatalog,dc=eu-datagrid,dc=org"; Describe the requirements for the job.

11. C. Loomis – Title – Date – 11 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output > grid-proxy-init Your identity: /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr Enter GRID pass phrase for this identity: ************* Creating proxy.......................................... Done Your proxy is valid until Sun Jun 23 06:13:51 2002 Create a proxy for authorization  like logging in.

12. C. Loomis – Title – Date – 12 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output > dg-job-submit ChecksumRC.jdl Connecting to host testbed011.cern.ch, port 7771 Logging to host testbed011.cern.ch, port 15830 ************************************************************************************ JOB SUBMIT OUTCOME The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is: https://testbed011.cern.ch:7846/137.138.181.253/162422289427581?testbe d011.cern.ch:7771 ************************************************************************************ Submit job. Broker will automatically find appropriate place.

13. C. Loomis – Title – Date – 13 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output >dg-job-status … dg_JobId = https://testbed011.cern.ch:7846/137.138.181.253/… Status = OutputReady Last Update Time (UTC) = Sat Jun 22 16:29:30 2002 Job Destination = gppce05.gridpp.rl.ac.uk:2119/jobmanager-pbs-S Status Reason = terminated Job Owner = /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/… Status Enter Time (UTC) = Sat Jun 22 16:29:30 2002 Get the status of the submitted job.

14. C. Loomis – Title – Date – 14 Example Prerequisites write script job description Job Submission grid-proxy-init dg-job-submit dg-job-status dg-job-get-output >dg-job-get-output … Output sandbox files for the job: - https://testbed011.cern.ch:7846/137.138.181.253/… have been successfully retrieved and stored in the directory: /tmp/162422289427581 Pull the output back to home machine. >cat /tmp/162422289427581/std.out File: /flatfiles/eo/utve_v_1_71008101_12902_0609.lv0 Contains: 1034 events

15. C. Loomis – Title – Date – 15 Future Developments EDG Plans:  more advanced data management (I.e. real “Storage Element”)  job splitting, checkpointing; interactive jobs  more robust, consistent security model OGSA (Open Grid Services Architecture)  Proposal by Globus/IBM based on web-services  Adds concepts like notification and soft-state registration.  Standardizes the protocol and interface descriptions for Grid services. Cross-fertilization of Projects  LAL is ideally situated for this. (EDG, LCG, XtremWeb, CGP2P)

16. C. Loomis – Title – Date – 16 Conclusions EDG has interesting (to a physicist) software & working testbed:  Current release: —Basic features available. —Somewhat painful: reliability, low-level mucking around  Fall release: —Expect big change in functionality, especially data management. —Concerted effort on reliability should make testbed less frustrating.  About the time to get involved… Long-term success:  When you work routinely in a “Grid” environment on your desktop.  Fully automatic: resource discovery, optimization, and task dispatch.  Not there yet, but going in right direction.

17. C. Loomis – Title – Date – 17 Getting Involved & Further Surfing Talk to grid contact:  ATLAS (Laura Perini)  ALICE (Piergiorgio Cerello)  CMS (Andrea Sciaba)  LHCb (Eric van Herwijnin)  others (see me) Register:  “User Registration” @ WP6 website Warnings:  Dev. project  Frequent changes.  No guarantees on availability. Schedule:  EDG 1.2 (shortly)  EDG 2.0 (fall 2002) Interesting Web Sites EDG (Main Website)http://www.eu-datagrid.org EDG (WP6)http://marianne.in2p3.fr CrossGridhttp://www.crossgrid.org DataTaghttp://www.datatag.org GridPPhttp://www.gridpp.ac.uk LCGhttp://lhcgrid.web.cern.ch/LHCgrid NorduGridhttp://www.nordugrid.org Condorhttp://www.cs.wisc.edu/condor Globushttp://www.globus.org GriPhyNhttp://www.griphyn.org iVDGLhttp://www.ivdgl.org PPDGhttp://www.ppdg.net OGSAhttp://www.globus.org/ogsa Global Grid Forumhttp://www.gridforum.org