1. NorduGrid Architecture and tools CHEP2003 – UCSD Anders Wäänänen waananen@nbi.dkwaananen@nbi.dk

2. 2Anders WäänänenNorduGrid Architecture NorduGrid project u Launched in spring of 2001, with the aim of creating a Grid infrastructure in the Nordic countries. u Idea to have a Monarch architecture with a Nordic tier 1 center u Partners from Denmark, Norway, Sweden, and Finland u Initially meant to be the Nordic branch of the EU DataGrid (EDG) project u 3 full-time researchers with few externally funded

3. 3Anders WäänänenNorduGrid Architecture Motivations u NorduGrid was initially meant to be a pure deployment project u One goal was to have the ATLAS data challenge run by May 2002 u Should be based on the the Globus Toolkit™ u Available Grid middleware: n The Globus Toolkit™ s A toolbox – not a complete solution n European DataGrid software s Not mature in the beginning of 2002 s Architecture problems

4. 4Anders WäänänenNorduGrid Architecture Architecture requirements u No single point of failure u Should be scalable u Resource owners should have full control over their resources u As few site requirements as possible: n Local cluster installation details should not be dictated s Method, OS version, configuration, etc… n Compute nodes should not be required to be on the public network n Clusters need not be dedicated to the Grid

5. 5Anders WäänänenNorduGrid Architecture NorduGrid Features at glance u Dynamic Information System, Brokering, Monitoring u Independence on Globus GASS cache (and it’s bug) u Own GridFTP server, pluggable with job submission u Stable and tested Grid testbed u Not Nordic or HEP specific u Tested on RedHat 6.2, 7.2 (also Alpha), Mandrake, Debian, Slackware u Can share resources with non Grid applications u Has been running ATLAS data challenges since May 2002

6. 6Anders WäänänenNorduGrid Architecture NorduGrid components u Grid Manager – Mange Grid jobs in cluster n Job control and data management u Information system n Patched Globus MDS with improved schema u User interface n Job submission and personal broker u Grid monitor n Web based interface to information system u Globus replica catalog

7. 7Anders WäänänenNorduGrid Architecture Grid manager features 1 u Staging of executables and input/output data u Supported protocols: n Local files, gridftp, ftp, http(s), Replica Catalog, Replica Location Services u Data transfer control including retries u Caching of input data n Cache size control n Private (per UNIX user) and shared caches n Data access control based on user’s credentials u Support for runtime environment (eg. Software installations) u Full job information available for auditing, accounting and debugging

8. 8Anders WäänänenNorduGrid Architecture Grid manager features 2 u Globus building blocks used n GridFTP – fast, reliable and secure data access n GASS transfer – http(s) like data access protocol n Replica catalog n Replica Location Service (with EDG) n RSL – expandable Resource Specification Language u Limitations n Data handling is currently only supported at job start and job end when cluster nodes are on a private network

9. 9Anders WäänänenNorduGrid Architecture Grid Manager architecture Cache Job session directory Link or copy submission stagein stageout NorduGrid gridftp server downloader Grid Manager uploader File access Job control Job session directory Computing node LRMS Frontend NFS LRMS

10. 10Anders WäänänenNorduGrid Architecture User interface u The NorduGrid user interface provides a set of commands for interacting with the grid n ngsub – for submitting jobs n ngstat – for states of jobs and clusters n ngcat – to see stdout/stderr of running jobs n ngget – to retrieve the results from finished jobs n ngkill – to kill running jobs n ngclean – to delete finished jobs from the system n ngcopy – to copy files to, from and between file servers and replica catalogs n ngremove – to delete files from file servers and RC’s

11. 11Anders WäänänenNorduGrid Architecture Information system u The nerve system of the Grid - information is a critical resource! u Complications: n Large number of resource -> scalability n Heterogeneous resources -> characterization n Decentralized n Efficient access to dynamic data n Quality and reliability of information n Compromise between: s Up to date data vs. load on the Grid

12. 12Anders WäänänenNorduGrid Architecture NorduGrid information system u Use Globus MDS u Improved schemas with natural representation of resources: n Clusters (queues, jobs and users) n Storage elements n Replica Catalogs u Use efficient providers u Each resource runs a GRIS u GRIS’s are organized into a dynamic country based GIIS hierarchy. u Have enough information to do brokering

13. 13Anders WäänänenNorduGrid Architecture DIT of a cluster cluster queue jobs users job-01 job-02 job-03 user-01 user-02 queue jobs users job-04 job-05 user-02 user-03 user-01

14. 14Anders WäänänenNorduGrid Architecture Cluster entry

15. 15Anders WäänänenNorduGrid Architecture DIT of a cluster cluster queue jobs users job-01 job-02 job-03 user-01 user-02 queue jobs users job-04 job-05 user-02 user-03 user-01

16. 16Anders WäänänenNorduGrid Architecture Queue entry

17. 17Anders WäänänenNorduGrid Architecture DIT of a cluster cluster queue jobs users job-01 job-02 job-03 user-01 user-02 queue jobs users job-04 job-05 user-02 user-03 user-01

18. 18Anders WäänänenNorduGrid Architecture Job entry job status monitoring = information system query

19. 19Anders WäänänenNorduGrid Architecture Another job entry - the job entry is generated on the execution cluster - when the job is completed and the results are retrieved the job disappears from the information system

20. 20Anders WäänänenNorduGrid Architecture DIT of a cluster cluster queue jobs users job-01 job-02 job-03 user-01 user-02 queue jobs users job-04 job-05 user-02 user-03 user-01

21. 21Anders WäänänenNorduGrid Architecture Personalized information user based information is essential on the Grid: n users are not really interested in the total number of cpus of a cluster, but how many of those are available for them! n number of queuing jobs are irrelevant if the submission gets immediately executed n instead of total disk space the user's quota is interesting nordugrid-authuser objectclass n freecpus n diskspace n queuelength

22. 22Anders WäänänenNorduGrid Architecture User entry

23. 23Anders WäänänenNorduGrid Architecture GIIS Hierarchy Hierarchy of GRISes/GIISes

24. 24Anders WäänänenNorduGrid Architecture Grid Montior

25. 25Anders WäänänenNorduGrid Architecture Brokering & job submission ●Searches through the NorduGrid Testbed for available clusters ●Loops through all the clusters and selects those queues (possible targets) where: ●The user is authorized to run ●Job requirements can be satisfied ●Selects a job destination from the matching targets – Randomly selects among the free resources (where user-freecpus>0 ) –In case there are no free matching resources some of the “load” attributes (i.e. user-queuelength) are taken into account

26. 26Anders WäänänenNorduGrid Architecture Implementation u Portability n Support toolkit on variety of platforms as needed by sites n Open for new architectures (already support ia32, ia64 and alpha) u Always choose the best technical workable solution n Not afraid to throw things away n Not tied to solutions because of political reasons n Open for new ideas u Start with something simple that works and proceed from there

27. 27Anders WäänänenNorduGrid Architecture NorduGrid job submission RC RSL MDS Grid Manager Gatekeeper GridFTP RSL

28. 28Anders WäänänenNorduGrid Architecture Quick client installation/job run u As a normal user: retrieve nordugrid-standalone-0.3.17.rh72.i386.tgz tar xfz nordugrid-standalone-0.3.17.rh72.i386.tgz cd nordugrid-standalone-0.3.17 source./setup.sh n Maybe get a certificate grid-cert-request install certificate per instructions grid-proxy-init ngsub '&(executable=/bin/echo)(arguments="Hello World")‘

29. 29Anders WäänänenNorduGrid Architecture Future development or integration u Better Authorization u Accounting u Optimize brokering u More intelligent data management and replication service u Handle network requests from running jobs on “private” networks u Grid portal interface – in testing u Move towards Grid services and improved community compatibility

30. 30Anders WäänänenNorduGrid Architecture Future u The committee of Nordic natural science ministers NOS-N has decided to fund a new common Nordic Grid Project based on the work done by the NorduGrid project. This project should work on a proposal/recommendation for a Nordic DataGrid facility. u Support for the toolkit in the future u This will be supported in each country by local Grid initiatives u Collaboration with the Nordic computing centers have already been initiated with the deployment of the toolkit on several large centers. u Use it for future ATLAS production in the Nordic countries u Move towards OGSA and better community compatibility

31. 31Anders WäänänenNorduGrid Architecture Resources u Documentation and source code are available for download u Main Web site: n http://www.nordugrid.org/ http://www.nordugrid.org/ u Repository n ftp://ftp.nordugrid.org/pub/nordugrid/ ftp://ftp.nordugrid.org/pub/nordugrid/

32. 32Anders WäänänenNorduGrid Architecture The NorduGrid core group u Александр Константинов u Balázs Kónya u Mattias Ellert u Оксана Смирнова u Jakob Langgaard Nielsen u Trond Myklebust u Anders Wäänänen