1. Olof Bärring – WP4 summary- 4/9/2002 - n° 1 Partner Logo WP4 report Plans for testbed 2 Olof.Barring@cern.ch

2. Olof Bärring – WP4 summary- 4/9/2002 - n° 2 Summary u Reminder on how it all fits together u What’s in R1.2 (deployed and not-deployed but integrated) u Piled up software from R1.3, R1.4 u Timeline for R2 developments and beyond u A WP4 problem u Conclusions

3. Olof Bärring – WP4 summary- 4/9/2002 - n° 3 How it all fits together (job management) Farm A (LSF)Farm B (PBS ) Grid User (Mass storage, Disk pools) Local User Monitoring Fabric Gridification Resource Management Grid Info Services (WP3) WP4 subsystems Other Wps Resource Broker (WP1) Data Mgmt (WP2) Grid Data Storage (WP5) - Submit job - Optimized selection of site -Authorize -Map grid  local credentials -Authorize -Map grid  local credentials -Select an optimal batch queue and submit -Return job status and output -Select an optimal batch queue and submit -Return job status and output - publish resource and accounting information

4. Olof Bärring – WP4 summary- 4/9/2002 - n° 4 How it all fits together (system mgmt) WP4 subsystems Other Wps Farm A (LSF)Farm B (PBS ) Installation & Node Mgmt Configuration Management Monitoring & Fault Tolerance Resource Management Information Invocation - Update configuration templates - Node malfunction detected -Remove node from queue -Wait for running jobs(?) -Remove node from queue -Wait for running jobs(?) - Trigger repair - Repair (e.g. restart, reboot, reconfigure, …) - Node OK detected -Put back node in queue Automation

5. Olof Bärring – WP4 summary- 4/9/2002 - n° 5 How it all fits together (node autonomy) Cfg cache Monitoring Buffer Correlation engines Node mgmt components Monitoring Measurement Repository Configuration Data Base Central (distributed) Buffer copy Node profile Local recover if possible (e.g. restarting daemons) Automation

6. Olof Bärring – WP4 summary- 4/9/2002 - n° 6 What’s in R1.2 (and deployed) u Gridification: n Library implementation of LCAS

7. Olof Bärring – WP4 summary- 4/9/2002 - n° 7 What’s in R1.2 but not used/deployed u Resource management n Information provider for Condor (not fully tested because you need a complete testbed including a Condor cluster) u Monitoring n Agent + first prototype repository server + basic linuxproc sensors n No LCFG object  not deployed u Installation mgmt n LCFG light exists in R1.2. Please provide us feedback on any problems you have with it.

8. Olof Bärring – WP4 summary- 4/9/2002 - n° 8 Piled up software from R1.3, R1.4 u Everything mentioned here is ready, unit tested and documented (and rpms are built by autobuild) n Gridification s LCAS with dynamic plug-ins. (already in R1.2.1???) n Resource mgmt s Complete prototype enterprise level batch system management with proxy for PBS (see next slide). Includes LCFG object. n Monitoring s New agent. Production quality. Already used on CERN production clusters sampling some 110 metrics/node. Has also been tested on Solaris. s LCFG object n Installation mgmt s Next generation LCFG: LCFGng for RH6.2 (RH7.2 almost ready)

9. Olof Bärring – WP4 summary- 4/9/2002 - n° 9 queues resources Batch system: PBS, LSF, etc. Scheduler Runtime Control System Grid Local fabric Gatekeeper (Globus or WP4) job 1job 2job n JM 1JM 2JM n scheduled jobs new jobs user queue 2 execution queue stopped, visible for users started, invisible for users submit user queue 1 get job info move move job exec job RMS components PBS-, LSF-Cluster Globus components Enterprise level batch system mgmt prototype (R1.3)

10. Olof Bärring – WP4 summary- 4/9/2002 - n° 10 Timeline for R2 developments u Configuration management: complete central part of framework n High Level Definition Language: 30/9/2002 n PAN compiler: 30/9/2002 n Configuration Database (CDB): 31/10/2002 u Installation mgmt n LCFGng for RH72: 30/9/2002 u Monitoring: Complete final framework n TCP transport: 30/9/2002 n Repository server: 30/9/2002 n Repository API WSDL: 30/9/2002 n Oracle DB support: 31/10/2002 n Alarm display: 30/11/2002 n Open Source DB (MySQL or PostgreSQL): mid-December 2002

11. Olof Bärring – WP4 summary- 4/9/2002 - n° 11 Timeline for R2 developments u Resource mgmt n GLUE info providers: 15/9/2002 n Maintenance support API (e.g. enable/disable a node in the queue): 30/9/2002 n Provide accounting information to WP1 accounting group: 30/9/2002 n Support Maui as scheduler u Fault tolerance framework n Various components already delivered n Complete framework by end of November

12. Olof Bärring – WP4 summary- 4/9/2002 - n° 12 Beyond release 2 u Conclusion from WP4 workshop, June 2002: LCFG is not the future for EDG (see WP4 quarterly report for 2Q02) because: n Inherent LCFG constraints on the configuration schema (per-component config) n LCFG is a project of its own and our objectives do not always coincide n We have learned a lot from LCFG architecture and we continue to collaborate with the LCFG team u EDG future: first release by end-March 2003 n Proposal for a common schema for all fabric configuration information to be stored in the configuration database, implemented using the HLDL. n New configuration client and node management replacing LCFG client (the server side is already delivered in October). n New software package management (replacing updaterpms) split into two modules: an OS independent part and an OS dependent part (packager).

13. Olof Bärring – WP4 summary- 4/9/2002 - n° 13 Global schema tree hardware systemsw CPUharddiskmemory…. sys_nameinterface_typesize…. hostnamearchitecturepartitionsservices…. hda1 sizetypeid hda2…. packagesknown_repositoriesedg_lcas …. versionrepositories…. cluster …. Component specific configuration The population of the global schema is an ongoing activity http://edms.cern.ch/document/352656/1

14. Olof Bärring – WP4 summary- 4/9/2002 - n° 14 SW repository structure (maintained by repository managers): /sw/known_repositories/Arep/url = (host, protocol, prefix dir) /owner = /extras = /directories/dir_name_X/path = (asis) /platform = (i386_rh61) /packages/pck_a/name = (kernel) /version = (2.4.9) /release = 31.1.cern /architecture = (i686) /dir_name_Y /path = (sun_system) /platform = (sun4_58) /packages/pck_b/name = (SUNWcsd) /version = 11.7.0 /release = 1998.09.01.04.16 /architecture = (?) Global schema example

15. Olof Bärring – WP4 summary- 4/9/2002 - n° 15 Problem u Very little of delivered WP4 software is of any interest to EDG application WPs, possibly with the exception of producing nice colour plots of the CPU loads when a job was run… u This is normal, but… n Site administrators do not grow on trees. Because of the lack of good system admin tools, like the ones WP4 tries to develop, the configuration, installation and supervision of the testbed installations require a substantial amount of manual work. u However, thanks to Bob new priority list the need for automated configuration and installation has bubbled up on the required features stack to become absolutely vital for assuring good quality.

16. Olof Bärring – WP4 summary- 4/9/2002 - n° 16 Summary u Substantial amount of s/w piled up from R1.3, R1.4 to be deployed now u R2 also includes two large components: n LCFGng – migration is non-trivial but we already perform as much as the non-trivial part ourselves so TB integration should be smooth n Complete monitoring framework u Beyond R2: LCFG is not future for EDG WP4. First version of new configuration and node management system in March 2003