EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI SA2 – Software Provisioning Michel Drescher Technical Manager EGI.eu SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI SA2 Overview 30/05/2011 SA2 - EGI-InSPIRE Review Countries 10 Beneficiaries 503 PMs FTEs WPTaskBeneficiary Total PMs WP5-ETSA2.1EGI.eu36 WP5-ETSA2.2CSIC35 WP5-ETSA2.2LIP35 WP5-ETSA2.3CSIC35 WP5-ETSA2.3LIP35 WP5-ETSA2.4CESNET60 WP5-ETSA2.4GRNET95 WP5-ETSA2.5CESNET47 WP5-ETSA2.5KIT-G24 WP5-ETSA2.5INFN47 WP5-ETSA2.5UCPH24 WP5-ETSA2.5VR-SNIC24 WP5-ETSA2.5NORDUNET6 Spain Portugal Czech Republic Greece Germany Italy Denmark Sweden

EGI-InSPIRE RI Objectives SA2 - EGI-InSPIRE Review Taken from the Description of Work TaskObjective TSA2.1 7% O1 Establish agreements with key software providers O2 Maintain the UMD Roadmap TSA2.2 14% O3Define general and component specific quality criteria to be applied to software components TSA2.3 14% O4Verify the software components against these criteria TSA2.4 30% O5Provide a repository for the software components within UMD and the related support tools TSA2.5 35% O6Provide a distributed support unit within the EGI Helpdesk infrastructure with expertise on the deployed middleware in production use. 30/05/2011 TSA2.1: Michel Drescher, EGI.euTSA2.4: Kostas Koumantaros, GRNET TSA2.2: Enol Fernandez, CSICTSA2.5: Michael Gronager, TSA2.3: Carlos Fernandez, FCTSG Henrik Jensen, NORDUNET

EGI-InSPIRE RI SA2 in the EGI Ecosystem 30/05/ Public Funding Bodies European Commission National Research Councils Service & Resource Providers EGI.eu Resource Infrastructure Providers Technology Providers Open Source Providers Commercial Providers Requirements + Feedback Policies + Funding Strategic Feedback Requirements + Feedback User Community Services + Support Requirements + Feedback Technology + Support SA2 - EGI-InSPIRE Review 2011 SA2

EGI-InSPIRE RI A Virtuous Service Cycle SA2 - EGI-InSPIRE Review 2011 Used by Researchers Gathering New Requirements New Technology Assessed Deployed Infrastructure Services 7 EGI.eu National Resource Providers Current User Communities New User Communities MoUs MoUs & OLAs Technology Providers MoUs & SLAs 30/05/2011

EGI-InSPIRE RI SA2 Virtuous Service cycle SA2 - EGI-InSPIRE Review Technology Providers MoUs & SLAs 30/05/2011 New Technology Assessed Steering the EGI software evolution –Publish the UMD Roadmap –Collect and prioritise strategic requirements –Engage with Technology Providers Provision software for the EGI community –Ensure the quality of delivered software –Provide a unified middleware repository –Provide 2 nd level support for deployed middleware MoU SLA

EGI-InSPIRE RI Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI SA2 - EGI-InSPIRE Review 2011 The UMD Roadmap time UMD components Domain-specific components EGI Technology components components 30/05/2011 Defines Three layers of software in EGI General- purpose software Generic Grid platform Specialised software

EGI-InSPIRE RI The UMD Roadmap 11SA2 - EGI-InSPIRE Review 2011 Describes the future EGI software architecture –UMD = Unified Middleware Distribution –Reflects requirements collected by the EGI community –Main work-package deliverable, see D5.1 and D5.2 Describes a service-oriented architecture (SOA) –Provides clear composition of infrastructure –Promotes consolidation of service provisioning Identifies nine functional areas –Security, Information, Storage, Data, Compute, Operations, Virtualisation, Remote Instrumentation, Clients 30/05/2011

EGI-InSPIRE RI The UMD Roadmap Capabilities define EGI’s service architecture –Distinct functionality –Technology neutral –Define Interfaces Functional dependencies –Promote interoperability –Facilitate modularity –Ensure infrastructure maintainabilit y SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI UMD Repository Distributed service hosted by GRNET –IASA, Athens (Master) –NDC, Athens –AUTH, Thessaloniki Frontend based on Wordpress CMS –RSS based syndication with EGI website –Software distribution based sub-community High Availability of backend, and frontend content –All sites synchronise every 5 minutes –Round-robin list of serving sites updated every 10 minutes –1 Gbit/s GEANT connectivity at each individual site –2*10 Gbit/s + 5 Gbit/s backup GEANT connectivity for GRNET 30/05/2011 SA2 - EGI-InSPIRE Review

EGI-InSPIRE RI SA2 - EGI-InSPIRE Review 2011 Steering Middleware evolution Technology Coordination Board Discussion & Prioritisation UMD Roadmap Update & publish UMD Roadmap prioritised Requirements Ingest & implement requirements External Technology Providers External Technology Providers New & improved software New & improved software 30/05/2011 Collect and prioritise requirements –From the EGI community Publish Middleware Roadmap documents –UMD Roadmap –UMD Release Schedule

EGI-InSPIRE RI Driven by Needs and Expertise –MoU covers intent of collaboration SLA defines –Service scope –Service quality –Problem resolution Engage in cooperation –Active strategic coordination –Regular software delivery SA2 - EGI-InSPIRE Review Identify Negotiate Agree Technology Coordination Board Technology Coordination Board ExpertiseNeed Engaging with Technology Providers 30/05/2011

EGI-InSPIRE RI Provisioning software 16 SA2 - EGI-InSPIRE Review 2011 Operations (SA1) Provisioning Infrastructure Staged Rollout Staged Rollout Criteria Verification Criteria Verification Production Criteria Definition Criteria Definition External Technology Providers Deployed Middleware Support Unit Requirements Software 30/05/2011

EGI-InSPIRE RI Maintaining Quality Criteria SA2 - EGI-InSPIRE Review /05/2011 Updated Quality Criteria Event Criteria Definition Criteria Definition Generic criteria apply to all software –e.g. Documentation, Monitoring, Accounting Specific criteria apply to a UMD Capability –e.g. Interface compliance, Parallel Job behaviour Quality Criteria evolve with community needs –Identifiable by name, version number, and validity dates

EGI-InSPIRE RI Verifying Software Quality Verifies delivered software against criteria –Maintaining, and following verification guidelines Reports capture verification process & result –Accept or reject software –Communicates findings to its customers SA2 - EGI-InSPIRE Review Quality Criteria Reports New & improved software New & improved software 30/05/2011 Criteria Verification Criteria Verification

EGI-InSPIRE RI End-to-end semi-automatic provisioning –User interfaces for manual process actions Permanent, public storage of provisioning reports –Both Criteria Verification, and StagedRollout Integrated with processes in –SA1: StagedRollout, Repositories –Technology Providers SA2 - EGI-InSPIRE Review Support infrastructure and repository 30/05/2011 Repository RT, DocDB, Wiki GGUS Verification StagedRollout

EGI-InSPIRE RI User1 st level support2 nd level support3 rd level support Deployed Middleware Support Unit 2 nd level support for deployed middleware –Issue advisories, build technology knowledge base Provide consultancy services –Software Best Practices for Operations (SA1) –Strategic middleware evolution (TCB) SA2 - EGI-InSPIRE Review EGI Helpdesk (GGUS) Support ticket Knowledge Base Knowledge Base Technology Provider Technology Provider DMSU Ticket triage Ticket triage implement 30/05/2011

EGI-InSPIRE RI Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Reviewing Objective O1 “Establish agreements with key software providers” 4 MoUs signed with: –EMIhttp://go.egi.eu/483http://go.egi.eu/483 –IGEhttp://go.egi.eu/484http://go.egi.eu/484 –SAGAhttp://go.egi.eu/485http://go.egi.eu/485 –StratusLabhttp://go.egi.eu/448http://go.egi.eu/448 3 SLAs signed with: –EMIhttp://go.egi.eu/461http://go.egi.eu/461 –IGEhttp://go.egi.eu/442http://go.egi.eu/442 –SAGAhttp://go.egi.eu/449http://go.egi.eu/449 MS505 provides a template for SLAs: SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Reviewing Objective O2 “Maintain the UMD Roadmap” UMD Roadmap is published every 6 months –D5.1http://go.egi.eu/UMDRoadmap-1http://go.egi.eu/UMDRoadmap-1 –D5.2http://go.egi.eu/UMDRoadmap-2http://go.egi.eu/UMDRoadmap-2 Requirements are regularly prioritised and tracked – –Process draft available: 6 meetings of the Technology Coordination Board –3 F2F meetings and 3 phone conferences – 23SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Reviewing Objective O3 “Define general & component-specific quality criteria” 171 Quality Criteria for UMD Roadmap 1.0 defined – –Covers general Quality Criteria (17) –Covers 6 specific UMD Roadmap Criteria areas Security (33), Information (7), Data (18), Storage (16), Compute (49), Operations (31) Next draft document set available –Planned publication: 1 August 2011 – Release information available at: – 24SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Reviewing Objective O4 “Verify delivered software against the EGI Quality Criteria” MS503 describes the Software Provisioning process – Instructions, processes and report templates defined in – – – Technology Providers planned releases for PY2 –Processes were verified collaborating with internal providers –2 EGI Trust Anchors releases –4 Service Availability Monitor (SAM) releases 25SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Reviewing Objective O5 “Provide a repository for the software components within UMD, and the related support tools” EGI Software Repository Architecture –MS501: –MS504: Provide the UMD Repository and EGI.eu IT infrastructure –Software submission for provisioning: Access for Release Managers, and Middleware support unit –RT based provisioning: Currently for provisioning teams only! –EGI Software repository: 26SA2 - EGI-InSPIRE Review /05/2011 Kostas, if you have figures on (increasing) repository popularity, send me an Excel sheet on this!

EGI-InSPIRE RI Reviewing Objective O6 “Provide a distributed support unit within the EGI Helpdesk infrastructure with expertise on the deployed middleware in production” MS502 ( defines scope of DMSUhttp://go.egi.eu/ new tickets in total, 255 solved 27SA2 - EGI-InSPIRE Review /05/ days to solve 1.04 days to solve 5.89 days to solve 5.89 days to solve GGUS enforces strict 3-level assignment rules DMSU 55 new tickets 48 solved EMI 268 new tickets 202 solved IGE 5 new tickets 5 tickets solved

EGI-InSPIRE RI Agenda Introduction Setting up SA2 processes Reviewing objectives: How well did we fare? SA2 issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Issues in PY1 29SA2 - EGI-InSPIRE Review /05/ External Technology Providers are a new concept requiring explanations –Support for more diverse user communities requires scaling out to diverse technology expertise –4 meetings with Technology Providers 2.External Technology Providers are unfamiliar with formal service agreement, independent software verification and communication –3 dissemination meetings, effects start to show

EGI-InSPIRE RI Use of Resources 30SA2 - EGI-InSPIRE Review /05/2011 In general, no major deviation from plans –All Tasks consumed 75% - 99% of allocated effort GRNET partners consumption non-linear in project –IASA (201%), AUTH (42%), ICCS (33%) –IASA scheduled for high load in PY1, low load later –AUTH, ICCS vice versa Actual usage: UCPH (0%), NORDUNET (33%) –NORDUNET took DMSU leadership for PY1 –3 DMSU support request for ARC components in PY1

EGI-InSPIRE RI Agenda Introduction WP5 in greater detail Reviewing the objectives Reviewing issues and resource consumption Wrapping up SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Plans for next year 32SA2 - EGI-InSPIRE Review /05/2011 Continue developing and communicating a sustainable software provisioning model –Streamline installation and configuration of services –Deployment model based on appliances and solutions –First release of UMD 1.0 due for July 2011 Review, identify and implement processes –e.g.: “Routing slip” for new Technology Providers Execute and improve implemented processes –Optimise & automate where necessary

EGI-InSPIRE RI Summary 33SA2 - EGI-InSPIRE Review /05/2011 Definition & implementation of SA2 processes completed –Splitting middleware development, verification & deployment Improve quality of service through experience –Establish and formalise regular service reviews Technology Provider transition to new service and federation model is underway –Requires continued effort and adaptation of SA2 processes

EGI-InSPIRE RI SA2 - EGI-InSPIRE Review 2011 Backup slides 30/05/2011

EGI-InSPIRE RI Technology Helpdesk DMSU EGI-SA2 Technology Provider (EMI / IGE) TPM GGUS RT Technology Helpdesk announce accept/reject Workflow for Bugs found in production Technology release workflow 35SA2 - EGI-InSPIRE Review /05/2011

EGI-InSPIRE RI Doing business Implementing processes, artefacts and interactions Validate feasibility with small, controllable scope –With internal Technology Providers EGI Trust Anchors, SAM (JRA1) Establish and maintain appliance delivery model –With external Technology Providers (EMI) Re-scope and implement distribution based supply chain Verify principal supply chain with two dry runs Scale out to full enactment of the supply chain Review regularly, and implement adjustments 36SA2 - EGI-InSPIRE Review /05/2011