EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Oliver Keeble SA3 Activity Leader CERN EGEE-III First Review, June, 2009 SA3 Status Report

Enabling Grids for E-sciencE EGEE-III INFSO-RI Activity Overview SA3 - Oliver Keeble - EGEE-III First Review June Country Total PM planned at M24 (1) Total FTE CERN Cyprus120.5 Czech Republic241.0 Finland120.5 Greece301.3 Ireland361.5 Italy964.0 Netherlands241.0 Poland241.0 Russia301.3 Spain321.3 UK361.5 Total PM planned at M24752 Total FTE 31.3

Enabling Grids for E-sciencE EGEE-III INFSO-RI SA3 Objectives Description –SA3 will manage the process of building deployable and documented gLite middleware distributions. Its main objectives are to : –Produce well-tested and documented gLite releases together with associated configuration tools –Improve the multi-platform support of gLite –Increase interoperability of different Grid infrastructures by working towards best practices and established standards and provide input to standardisation bodies In between JRA1 & SA1 in the software process SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI SA3 - Oliver Keeble - EGEE-III First Review June Tasks TSA3.1: Integration, configuration and packaging (186PM)‏ TSA3.2: Testing and certification (319PM)‏ TSA3.3: Support, analysis, debugging, problem resolution (100PM)‏ TSA3.4: Interoperability & Platform support (141PM)‏ TSA3.5: Activity Management (46PM)‏ Distribution of tasks in SA3Software change management SA3/JRA1

Enabling Grids for E-sciencE EGEE-III INFSO-RI Middleware releases Functional highlights –CREAM –HYDRA & MDM –AMGA –glexec/SCAS –Batch sys integration –MPI Updates to gLite 3.1 / SL4 / 32 & 64 bit –Deployed across the infrastructure –22 updates made –Each an aggregation of numerous changes 1556 change requests were opened and 1742 were closed –Includes both bugs and enhancement requests SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Release history SA3 - Oliver Keeble - EGEE-III First Review June Each update represents numerous different changes Changes released together were independent until then

Enabling Grids for E-sciencE EGEE-III INFSO-RI Fixing bugs SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Open & closed change requests SA3 - Oliver Keeble - EGEE-III First Review June EGEE-III inherited a lot of ‘bugs’ Many are in fact fixed, invalid, obsolete, duplicate… The discontinuities represent efforts to clean up different classes of ‘bug’

Enabling Grids for E-sciencE EGEE-III INFSO-RI Certification & Release Release –MSA3.4.1 documented an updated release process  Acceptance criteria –Post-mortem & Rollback  Fixes are not always available in time –Rpm signing Certification –Full documentation for devolution of certification  Ready for product teams –Regression tests –CREAM stress and comparative analysis SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Monitoring of the testbed SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Patch certification SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Patch certification SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Batch system integration Torque/Maui, Condor, Sun Grid Engine and LSF as the main batch systems to support –‘ownership’ of LSF still unclear CREAM support added as it went into production Support for Torque and LSF is in place SGE (done by CESGA and Imperial College) –During the first year or EGEE III, SGE was fully certified as a LRMS for the LCG-CE –CREAM support for SGE is still ongoing Condor (done by IFAE) –integration with the LCG-CE working with known issues –Condor integration with CREAM is ongoing New and updated TWiki pages on LRMS integration and batch system support SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI GLUE 2.0 & InfoSys Information system schema –Will allow a better expression of what is on the grid and therefore more efficient use of resources Work carried out within JRA1 & SA3 Ratified as an OSG standard LDAP rendering is nearly done –Will be packaged and pushed out in a few weeks 3 stage rollout process –Deployment of ‘empty’ schema in parallel with 1.3 –Update of information providers to populate 2.0 –Implementation of support in clients SAGA Service discovery API ‘Scalability & infosys related problems’ SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Activity Coordination Task tracking –Weekly meetings EMT –Cross activity coordination, chaired by SA3 All-hands –Established the principle of joint sessions with JRA1 –CERN & Prague –Expect 2 more during year 2 TMB SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Population of task tracker SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Interoperability & Standards glite-WMS submission to ARC Short term and medium interoperability goals have been achieved with key infrastructures: OSG, ARC Work has moved beyond short term fixes –Now working towards long term sustainability via standards  Pursued in OGF  BES/JSDL workshop  “Production Grid Infrastructures” –UMD  Harmonisation between European middleware stacks Maintaining relationships with other infrastructures NAREGI, Teragrid, DEISA, PRAGMA SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Platform support gLite 3.2 has been released on SL5/64bit –Worker Node and User Interface are available –A full release of all services will be made gLite 3.1 on SL4 (32/64) will be maintained as necessary Retirement of gLite 3.0 and SL3 ‘glite build system’ TMB priority list –SL5, Debian 4, Mac OS X Debian WN ready for certification –Opens the door to other ‘deb’ based platforms Tarball release can be adapted to other Linuxes Client/server split completed SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI The Debian Patch SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Platform Support – the issues Issues are typically not technical –No fundamental incompatibilities Finding effort to deal with multiple platform builds while moving to gLite3.2/SL5 Slow turnaround times on reported problems –Build complexity –Prioritisation Gradual introduction of platform support in ETICS –Client support –Build resources: Late arrival of Debian x86_64 build nodes Poor availability of MacOSX to developers Issues with migration to VDT A new platform requires expertise and resources all along the software lifecycle SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI General Issues Change iteration time –Access to developers –Release reactivity ETICS –Functional freeze to address performance issues –Availability and support of target platforms Effort –CERN SA3 permanently understaffed  CERN central SA3 lost 1 person every 2 months Certification expertise is not fungible Incompatibility of project objectives with local hierarchies Distribution of effort – average outside CERN is 9PM SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Year 2 Continue to deliver gLite updates during a volatile second year –Complete gLite 3.2 / SL5 release  Worker Node on other target platforms Implement ‘Product Teams’ –Overlay this structure on existing teams –Define requirements and constraints –Implement/adopt any new technology required Describe new release process –And all other docs (eg developer’s guide) gLite SDK and gLite 4 planning –Source rpms Fully document certification process SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Summary Continual gLite releases produced during year 1 –On average an update made every 2 weeks  Functional enhancements and bugfixes –Releases made on SL5/64bit –Debian WN in certification Certification –Improved test coverage  In regression tests –Documentation ready for devolution of tasks to product teams Release –Automation –Signed rpms SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI DELETE SLIDE ON SUBMISSION Put your name and presentation title on the first slide –See the meeting agenda Insert your activity and name into the footer Complete the activity overview slide (next slide) –the activity statistics - geographical, budget proportion, effort distribution between partners Structure your slides & presentation time with: –50%: Goals and achievements of the activity:  Pictures showing metrics are better than slides of bullet points  Mention key tasks within the activity i.e. What’s done, how managed, lead partner, involved partners,... –10%: Any deviations from the workplan in year I - if there were any! –20%: Any issues and how they have been addressed –15%: Plans for Y2 – broader EGI transition issues dealt elsewhere –5%: Summary slide highlighting the achievements & proposed solutions to any issues requiring resolution  This slide to be left up during Q & A SA3 - Oliver Keeble - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Other… Security Audits User support MPI Yaim SA3 - Oliver Keeble - EGEE-III First Review June