1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 SA2 – Software Provisioning Michel Drescher Technical Manager EGI.eu SA2 - EGI-InSPIRE Review 20111 30/05/2011

2. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 20112 30/05/2011

3. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 20113 30/05/2011

4. www.egi.eu EGI-InSPIRE RI-261323 SA2 Overview 30/05/2011 SA2 - EGI-InSPIRE Review 20114 8 Countries 10 Beneficiaries 503 PMs 10.49 FTEs WPTaskBeneficiary Total PMs WP5-ETSA2.1EGI.eu36 WP5-ETSA2.2CSIC35 WP5-ETSA2.2LIP35 WP5-ETSA2.3CSIC35 WP5-ETSA2.3LIP35 WP5-ETSA2.4CESNET60 WP5-ETSA2.4GRNET95 WP5-ETSA2.5CESNET47 WP5-ETSA2.5KIT-G24 WP5-ETSA2.5INFN47 WP5-ETSA2.5UCPH24 WP5-ETSA2.5VR-SNIC24 WP5-ETSA2.5NORDUNET6 Spain Portugal Czech Republic Greece Germany Italy Denmark Sweden

5. www.egi.eu EGI-InSPIRE RI-261323 Objectives SA2 - EGI-InSPIRE Review 20115 Taken from the Description of Work TaskObjective TSA2.1 7% O1 Establish agreements with key software providers O2 Maintain the UMD Roadmap TSA2.2 14% O3Define general and component specific quality criteria to be applied to software components TSA2.3 14% O4Verify the software components against these criteria TSA2.4 30% O5Provide a repository for the software components within UMD and the related support tools TSA2.5 35% O6Provide a distributed support unit within the EGI Helpdesk infrastructure with expertise on the deployed middleware in production use. 30/05/2011 TSA2.1: Michel Drescher, EGI.euTSA2.4: Kostas Koumantaros, GRNET TSA2.2: Enol Fernandez, CSICTSA2.5: Michael Gronager, TSA2.3: Carlos Fernandez, FCTSG Henrik Jensen, NORDUNET

6. www.egi.eu EGI-InSPIRE RI-261323 SA2 in the EGI Ecosystem 30/05/2011 6 Public Funding Bodies European Commission National Research Councils Service & Resource Providers EGI.eu Resource Infrastructure Providers Technology Providers Open Source Providers Commercial Providers Requirements + Feedback Policies + Funding Strategic Feedback Requirements + Feedback User Community Services + Support Requirements + Feedback Technology + Support SA2 - EGI-InSPIRE Review 2011 SA2

7. www.egi.eu EGI-InSPIRE RI-261323 A Virtuous Service Cycle SA2 - EGI-InSPIRE Review 2011 Used by Researchers Gathering New Requirements New Technology Assessed Deployed Infrastructure Services 7 EGI.eu National Resource Providers Current User Communities New User Communities MoUs MoUs & OLAs Technology Providers MoUs & SLAs 30/05/2011

8. www.egi.eu EGI-InSPIRE RI-261323 SA2 Virtuous Service cycle SA2 - EGI-InSPIRE Review 20118 Technology Providers MoUs & SLAs 30/05/2011 New Technology Assessed Steering the EGI software evolution –Publish the UMD Roadmap –Collect and prioritise strategic requirements –Engage with Technology Providers Provision software for the EGI community –Ensure the quality of delivered software –Provide a unified middleware repository –Provide 2 nd level support for deployed middleware MoU SLA

9. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 20119 30/05/2011

10. www.egi.eu EGI-InSPIRE RI-261323 10SA2 - EGI-InSPIRE Review 2011 The UMD Roadmap time UMD components Domain-specific components EGI Technology components components 30/05/2011 Defines Three layers of software in EGI General- purpose software Generic Grid platform Specialised software

11. www.egi.eu EGI-InSPIRE RI-261323 The UMD Roadmap 11SA2 - EGI-InSPIRE Review 2011 Describes the future EGI software architecture –UMD = Unified Middleware Distribution –Reflects requirements collected by the EGI community –Main work-package deliverable, see D5.1 and D5.2 Describes a service-oriented architecture (SOA) –Provides clear composition of infrastructure –Promotes consolidation of service provisioning Identifies nine functional areas –Security, Information, Storage, Data, Compute, Operations, Virtualisation, Remote Instrumentation, Clients 30/05/2011

12. www.egi.eu EGI-InSPIRE RI-261323 The UMD Roadmap Capabilities define EGI’s service architecture –Distinct functionality –Technology neutral –Define Interfaces Functional dependencies –Promote interoperability –Facilitate modularity –Ensure infrastructure maintainabilit y SA2 - EGI-InSPIRE Review 201112 30/05/2011

13. www.egi.eu EGI-InSPIRE RI-261323 UMD Repository Distributed service hosted by GRNET –IASA, Athens (Master) –NDC, Athens –AUTH, Thessaloniki Frontend based on Wordpress CMS –RSS based syndication with EGI website –Software distribution based sub-community High Availability of backend, and frontend content –All sites synchronise every 5 minutes –Round-robin list of serving sites updated every 10 minutes –1 Gbit/s GEANT connectivity at each individual site –2*10 Gbit/s + 5 Gbit/s backup GEANT connectivity for GRNET 30/05/2011 SA2 - EGI-InSPIRE Review 201113

14. www.egi.eu EGI-InSPIRE RI-261323 14SA2 - EGI-InSPIRE Review 2011 Steering Middleware evolution Technology Coordination Board Discussion & Prioritisation UMD Roadmap Update & publish UMD Roadmap prioritised Requirements Ingest & implement requirements External Technology Providers External Technology Providers New & improved software New & improved software 30/05/2011 Collect and prioritise requirements –From the EGI community Publish Middleware Roadmap documents –UMD Roadmap –UMD Release Schedule

15. www.egi.eu EGI-InSPIRE RI-261323 Driven by Needs and Expertise –MoU covers intent of collaboration SLA defines –Service scope –Service quality –Problem resolution Engage in cooperation –Active strategic coordination –Regular software delivery SA2 - EGI-InSPIRE Review 201115 Identify Negotiate Agree Technology Coordination Board Technology Coordination Board ExpertiseNeed Engaging with Technology Providers 30/05/2011

16. www.egi.eu EGI-InSPIRE RI-261323 Provisioning software 16 SA2 - EGI-InSPIRE Review 2011 Operations (SA1) Provisioning Infrastructure Staged Rollout Staged Rollout Criteria Verification Criteria Verification Production Criteria Definition Criteria Definition External Technology Providers Deployed Middleware Support Unit Requirements Software 30/05/2011

17. www.egi.eu EGI-InSPIRE RI-261323 Maintaining Quality Criteria SA2 - EGI-InSPIRE Review 201117 30/05/2011 Updated Quality Criteria Event Criteria Definition Criteria Definition Generic criteria apply to all software –e.g. Documentation, Monitoring, Accounting Specific criteria apply to a UMD Capability –e.g. Interface compliance, Parallel Job behaviour Quality Criteria evolve with community needs –Identifiable by name, version number, and validity dates

18. www.egi.eu EGI-InSPIRE RI-261323 Verifying Software Quality Verifies delivered software against criteria –Maintaining, and following verification guidelines Reports capture verification process & result –Accept or reject software –Communicates findings to its customers SA2 - EGI-InSPIRE Review 201118 Quality Criteria Reports New & improved software New & improved software 30/05/2011 Criteria Verification Criteria Verification

19. www.egi.eu EGI-InSPIRE RI-261323 End-to-end semi-automatic provisioning –User interfaces for manual process actions Permanent, public storage of provisioning reports –Both Criteria Verification, and StagedRollout Integrated with processes in –SA1: StagedRollout, Repositories –Technology Providers SA2 - EGI-InSPIRE Review 201119 Support infrastructure and repository 30/05/2011 Repository RT, DocDB, Wiki GGUS Verification StagedRollout

20. www.egi.eu EGI-InSPIRE RI-261323 User1 st level support2 nd level support3 rd level support Deployed Middleware Support Unit 2 nd level support for deployed middleware –Issue advisories, build technology knowledge base Provide consultancy services –Software Best Practices for Operations (SA1) –Strategic middleware evolution (TCB) SA2 - EGI-InSPIRE Review 201120 EGI Helpdesk (GGUS) Support ticket Knowledge Base Knowledge Base Technology Provider Technology Provider DMSU Ticket triage Ticket triage implement 30/05/2011

21. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 201121 30/05/2011

22. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O1 “Establish agreements with key software providers” 4 MoUs signed with: –EMIhttp://go.egi.eu/483http://go.egi.eu/483 –IGEhttp://go.egi.eu/484http://go.egi.eu/484 –SAGAhttp://go.egi.eu/485http://go.egi.eu/485 –StratusLabhttp://go.egi.eu/448http://go.egi.eu/448 3 SLAs signed with: –EMIhttp://go.egi.eu/461http://go.egi.eu/461 –IGEhttp://go.egi.eu/442http://go.egi.eu/442 –SAGAhttp://go.egi.eu/449http://go.egi.eu/449 MS505 provides a template for SLAs: http://go.egi.eu/212 http://go.egi.eu/212 22SA2 - EGI-InSPIRE Review 2011 30/05/2011

23. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O2 “Maintain the UMD Roadmap” UMD Roadmap is published every 6 months –D5.1http://go.egi.eu/UMDRoadmap-1http://go.egi.eu/UMDRoadmap-1 –D5.2http://go.egi.eu/UMDRoadmap-2http://go.egi.eu/UMDRoadmap-2 Requirements are regularly prioritised and tracked –https://wiki.egi.eu/wiki/Track_UMD_Requirementshttps://wiki.egi.eu/wiki/Track_UMD_Requirements –Process draft available: http://go.egi.eu/440http://go.egi.eu/440 6 meetings of the Technology Coordination Board –3 F2F meetings and 3 phone conferences –http://go.egi.eu/TCBhttp://go.egi.eu/TCB 23SA2 - EGI-InSPIRE Review 2011 30/05/2011

24. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O3 “Define general & component-specific quality criteria” 171 Quality Criteria for UMD Roadmap 1.0 defined –http://go.egi.eu/QualityCriteria-1http://go.egi.eu/QualityCriteria-1 –Covers general Quality Criteria (17) –Covers 6 specific UMD Roadmap Criteria areas Security (33), Information (7), Data (18), Storage (16), Compute (49), Operations (31) Next draft document set available –Planned publication: 1 August 2011 –http://go.egi.eu/QualityCriteria-Drafthttp://go.egi.eu/QualityCriteria-Draft Release information available at: –https://wiki.egi.eu/wiki/EGI_Quality_Criteria_Disseminationhttps://wiki.egi.eu/wiki/EGI_Quality_Criteria_Dissemination 24SA2 - EGI-InSPIRE Review 2011 30/05/2011

25. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O4 “Verify delivered software against the EGI Quality Criteria” MS503 describes the Software Provisioning process –http://go.egi.eu/68http://go.egi.eu/68 Instructions, processes and report templates defined in –https://wiki.egi.eu/wiki/EGI_Verification_Testbedhttps://wiki.egi.eu/wiki/EGI_Verification_Testbed –https://wiki.egi.eu/wiki/EGI_Verifier_Guidelinehttps://wiki.egi.eu/wiki/EGI_Verifier_Guideline –http://go.egi.eu/417http://go.egi.eu/417 Technology Providers planned releases for PY2 –Processes were verified collaborating with internal providers –2 EGI Trust Anchors releases –4 Service Availability Monitor (SAM) releases 25SA2 - EGI-InSPIRE Review 2011 30/05/2011

26. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O5 “Provide a repository for the software components within UMD, and the related support tools” EGI Software Repository Architecture –MS501: http://go.egi.eu/46http://go.egi.eu/46 –MS504: http://go.egi.eu/89http://go.egi.eu/89 Provide the UMD Repository and EGI.eu IT infrastructure –Software submission for provisioning: http://ggus.eu/techhttp://ggus.eu/tech Access for Release Managers, and Middleware support unit –RT based provisioning: http://go.egi.eu/SWProvDashboardhttp://go.egi.eu/SWProvDashboard Currently for provisioning teams only! –EGI Software repository: http://repository.egi.eu/http://repository.egi.eu/ 26SA2 - EGI-InSPIRE Review 2011 30/05/2011 Kostas, if you have figures on (increasing) repository popularity, send me an Excel sheet on this!

27. www.egi.eu EGI-InSPIRE RI-261323 Reviewing Objective O6 “Provide a distributed support unit within the EGI Helpdesk infrastructure with expertise on the deployed middleware in production” MS502 (http://go.egi.eu/69) defines scope of DMSUhttp://go.egi.eu/69 328 new tickets in total, 255 solved 27SA2 - EGI-InSPIRE Review 2011 30/05/2011 1.04 days to solve 1.04 days to solve 5.89 days to solve 5.89 days to solve GGUS enforces strict 3-level assignment rules DMSU 55 new tickets 48 solved EMI 268 new tickets 202 solved IGE 5 new tickets 5 tickets solved

28. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 201128 30/05/2011

29. www.egi.eu EGI-InSPIRE RI-261323 Issues in PY1 29SA2 - EGI-InSPIRE Review 2011 30/05/2011 1.External Technology Providers are a new concept requiring explanations –Support for more diverse user communities requires scaling out to diverse technology expertise –4 meetings with Technology Providers 2.External Technology Providers are unfamiliar with formal service agreement, independent software verification and communication –3 dissemination meetings, effects start to show

30. www.egi.eu EGI-InSPIRE RI-261323 Use of Resources 30SA2 - EGI-InSPIRE Review 2011 30/05/2011 In general, no major deviation from plans –All Tasks consumed 75% - 99% of allocated effort GRNET partners consumption non-linear in project –IASA (201%), AUTH (42%), ICCS (33%) –IASA scheduled for high load in PY1, low load later –AUTH, ICCS vice versa Actual usage: UCPH (0%), NORDUNET (33%) –NORDUNET took DMSU leadership for PY1 –3 DMSU support request for ARC components in PY1

31. www.egi.eu EGI-InSPIRE RI-261323 Agenda Introduction WP5 in greater detail Reviewing the objectives Reviewing issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review 201131 30/05/2011

32. www.egi.eu EGI-InSPIRE RI-261323 Plans for next year 32SA2 - EGI-InSPIRE Review 2011 30/05/2011 Continue developing and communicating a sustainable software provisioning model –Streamline installation and configuration of services –Deployment model based on appliances and solutions –First release of UMD 1.0 due for July 2011 Review, identify and implement processes –e.g.: “Routing slip” for new Technology Providers Execute and improve implemented processes –Optimise & automate where necessary

33. www.egi.eu EGI-InSPIRE RI-261323 Summary 33SA2 - EGI-InSPIRE Review 2011 30/05/2011 Definition & implementation of SA2 processes completed –Splitting middleware development, verification & deployment Improve quality of service through experience –Establish and formalise regular service reviews Technology Provider transition to new service and federation model is underway –Requires continued effort and adaptation of SA2 processes

34. www.egi.eu EGI-InSPIRE RI-261323 34SA2 - EGI-InSPIRE Review 2011 Backup slides 30/05/2011

35. www.egi.eu EGI-InSPIRE RI-261323 Technology Helpdesk DMSU EGI-SA2 Technology Provider (EMI / IGE) TPM GGUS RT Technology Helpdesk announce accept/reject Workflow for Bugs found in production Technology release workflow 35SA2 - EGI-InSPIRE Review 2011 30/05/2011

36. www.egi.eu EGI-InSPIRE RI-261323 Doing business Implementing processes, artefacts and interactions Validate feasibility with small, controllable scope –With internal Technology Providers EGI Trust Anchors, SAM (JRA1) Establish and maintain appliance delivery model –With external Technology Providers (EMI) Re-scope and implement distribution based supply chain Verify principal supply chain with two dry runs Scale out to full enactment of the supply chain Review regularly, and implement adjustments 36SA2 - EGI-InSPIRE Review 2011 30/05/2011