1. EMI INFSO-RI-261611 Technical Overview Balázs Kónya (Lund University) Technical Director 1 st EMI Periodic Review Brussels, 22 June 2011

2. EMI INFSO-RI-261611 Outline EMI is about software, the mission of the project is to deliver software for DCIs The process: how the development is managed – Technical Management roles – Requirements – Development cycle The outcome: EMI software portfolio – EMI on the DCI landscape – EMI products and their functionality – High-level technical roadmap – Year 2 top priorities 22/06/2011 1st EMI Periodic Review 2

3. EMI INFSO-RI-261611 Project Management PTB in EMI 22/06/2011 1st EMI Periodic Review 3 Requirements Technical Plans Development and Test Plans Development Testing Certification Validation Release Maintenance Support NA1, JRA1 JRA1 SA1 SA2 Software & Services Requirements Defines ImplementsCertifies Release Candidate Process definition Process monitoring NA2 NA1, NA2 Collaborations, exploitation, sustainability Training Dissemination & Out-Reach In-Reach NA1

4. EMI INFSO-RI-261611 Managing EMI development Project Technical Board (Technical Director, deputy and four area leaders) Defines the composition of EMI software portfolio Responsible for the overall technical plan and maintains the EMI roadmap Decides upon objectives and priorities for EMI development Approves major technical strategies and plans (e.g. area consolidation) Manages requirements Decides on specific technical matters (e.g. platforms) Technical Area Coordinators (4 areas) Compute, Data, Security, Infrastructure Translate the overall technical strategy to area work plans Responsible for technical strategy of the area Define area specific directions Oversee product team development plans Product Team leaders (26 product teams) Define product level development tasks Manage the development and maintenance of a product 22/06/2011 1st EMI Periodic Review 4

5. EMI INFSO-RI-261611 Technical Areas and PTs 22/06/2011 1st EMI Periodic Review 5 Compute Services Data Services Security Services Infrastructure Services ARC CE, UNICORE Services, gLite MPI, gLite Compute, etc dCache, CERN Data, DGAS, StoRM,etc ARC Container, UNICORE Security, Cesnet Security, Argus, VOMS, etc ARC Infosys, APEL client, DGAS Client, gLite Infosys, EMI Registry, etc Product Teams Dedicate teams of experts Fully responsible for development, maintenance and unit/system testing Product Teams Dedicate teams of experts Fully responsible for development, maintenance and unit/system testing

6. EMI INFSO-RI-261611 Requirement management Sources of requirements – EGI-TCB: formal requests communicated via EGI Tracker – WLCG: informal requests communicated verbally or via email – Users: direct communication with PTs Handling of requirements – After an initial filtering requests are recorded in the EMI Req. Tracker Large ”background noise”: duplicates, out-of-scope, bugreports, etc... – PTB assesses, categorizes and prioritizes requirements – Endorsed requests are translated into objectives, then to development tasks Adjusts and/or creates new objectives – Continous process, though requirements may not have an immediate effect on workplan Special attention to synchronize with EGI/UMD cycles Everything is recorded, monitored and tracked: – https://savannah.cern.ch/task/?group=emi-req 22/06/2011 1st EMI Periodic Review 6

7. EMI INFSO-RI-261611 From requirements to released products SA1 (PTs) Released products JRA1 (PTs) Development tasks Area Leaders Area work plans PTB Tech Objectives PTB Prioritized, assessed requirements Customers Requirements 22/06/2011 1st EMI Periodic Review 7

8. EMI INFSO-RI-261611 EMI on the DCI landscape 22/06/2011 1st EMI Periodic Review 8 EMI DNA1.4 - EMI Roadmap and DCI Collaborations

9. EMI INFSO-RI-261611 EMI software portfolio (1/2) Originates from ARC, dCache, gLite, UNICORE – not a complete union of the four Initial stack was defined by DNA1.3.1 – consisted of 98 components – ”component table” was the first attempt to define ”what EMI is” Early consolidation during first year – iterative process via EMI 0 and EMI 1 release preparations Logical restructuring Phase out (5 components) Current software portfolio (DNA1.3.2) – 58 products of different maturity levels – 54 products are released as part of Kebnekaise Became 51 after further merges – 3 products under certification and 4 in early development – available on SL5/64 22/06/2011 1st EMI Periodic Review 9

10. EMI INFSO-RI-261611 EMI software portfolio (2/2) Consists of services, clients and APIs/libraries Integrated products that work together EMI products contribute to the ”dynamic services” DCI layer and offer the following functionalities: – Compute: job execution, parallel job, job scheduling – Data: file access, file transfer, metadata catalogue – Infrastructure: information publication, accounting sensors, information discovery, monitoring probes, messaging backbone – Security: authentication, authorization, attribute authority, attribute authority, credential management 22/06/2011 1st EMI Periodic Review 10

11. EMI INFSO-RI-261611 Evolution of the software stack Need to manage conflicting requirements: – Product and interface stability – Need for consolidation via portfolio cleanup and adoption of common interfaces and libraries – Need to implement new features Effect on the software stack: – Established code base with little space for ”easy consolidation” emphasis is on hardening, cleanup – Challenging Product phase out transition path is necessary – New products for consolidation common libraries & services for a transitional period will coexist with pre-EMI solution – Feature requests to be implemented on top of existing products avoid development from scratch, yet another set of prototypes 22/06/2011 1st EMI Periodic Review 11

12. EMI INFSO-RI-261611 High-level Technical Roadmap Phase 1 (Kebnekaise): DONE Software integration: created a distribution Agreements: EMI-ES, STAR, GSI-replacement, messaging use cases Design: EMI Authentication Library, EMI Registry Completed tasks: server-side GLUE2, consistent SRM,... Phase 2 (Matterhorn) Functionality integration Agreements: compute accounting, delegation,... Implementation of all the agreements Delivery of the Consolidation plans Design: AAI strategy, Cloud strategy Phase 3 (Monte Bianco) Full realization of the consolidation plans Implementation of AAI and Cloud strategy Finalization of new developments by bringing code to production level 22/06/2011 1st EMI Periodic Review 12

13. EMI INFSO-RI-261611 Year 1 achievements 22/06/2011 1st EMI Periodic Review 13 Find details in JRA1 presentation

14. EMI INFSO-RI-261611 Year 2 top technical priorities (1/2) Compute: – EMI Execution Service: implementation of the agreed common job management methods – GLUE2 support in compute clients Data: – Client-side GLUE2 support implementation – EMI Data Access Library design and implementation – Storage Element and catalogue synchronization Security: – Simplified management of security credentials (AAI) – EMI Authentication Library implementation – EMI delegation agreement 22/06/2011 1st EMI Periodic Review 14

15. EMI INFSO-RI-261611 Year 2 top technical priorities (2/2) Infrastructure: – EMI service registry implementation – Cloud strategy – Delivery of service monitoring via NAGIOS All: – Consolidation plans – Additional platforms Debian family and SL6 – Usability improvements Command line parameters Error messages 22/06/2011 1st EMI Periodic Review 15

16. EMI INFSO-RI-261611 Thank you 22/06/201116 1st EMI Periodic Review EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

17. EMI INFSO-RI-261611 Technical Documents Release planning, management: – https://twiki.cern.ch/twiki/pub/EMI/EMI1Schedule/kebnekaise-v.9.pdf – https://twiki.cern.ch/twiki/pub/EMI/EMI1Updates/EMI_Release_Schedul e_v0.4.pdf Release Tracker: – https://savannah.cern.ch/task/?group=emi-releases RfC (development task on a PT level): – http://bugzilla.nordugrid.org/show_bug.cgi?id=2394 (glue support for ARC CE) – https://savannah.cern.ch/bugs/?77525 (ARGUS) – https://savannah.cern.ch/bugs/?77527 (ARGUS) EMI Execution Service inteface: http://cdsweb.cern.ch/record/1359908/files/EMI-ES-Specification_v1.0.odt Storage Accounting record: – http://cdsweb.cern.ch/record/1352472/files/StAR-EMI-tech-doc-final.doc 22/06/2011 1st EMI Periodic Review 17

18. EMI INFSO-RI-261611 Cloud position (DRAFT!) Presented at the EGI User Virtualization Workshop – https://www.egi.eu/indico/conferenceDispla y.py?confId=415 – https://www.egi.eu/indico/getFile.py/access ?contribId=9&resId=0&materialId=slides&co nfId=415 22/06/2011 1st EMI Periodic Review 18

19. EMI INFSO-RI-261611 Amsterdam slides:What EMI can contribute with EMI develops/operates in the "service level”, not in the VM level some of the EMI services can be useful on the VM-level as well: – BDII as the "info service", storing info about VMMs, so on – a generic service registry which will enable the "hooking up" of all the appliances – ARGUS, VOMS as authorization systems EMI will provide/contribute/build/test grid appliances and “landscape deployments” of EMI services – Landcape deployment example: a complete CE EMI will utilize VMs under/behind EMI services – e.g. Computing Elements with VM-based nodes EMI partners have been the driving force behind some of the “cloud- relevant” standards: – Extensions, profiles for GLUE, accounting Finally, EMI can offer lessons learnt in Grid – Our experience can help not to repeat the same mistakes WP - Presenter Name - EMI First EC Review 22 June 2011 19

20. EMI INFSO-RI-261611 Amsterdam slides: What is NOT in the scope of EMI Virtual Machine Management layer Application catalogues VM Image catalogues Accounting systems (server side) WP - Presenter Name - EMI First EC Review 22 June 2011 20

21. EMI INFSO-RI-261611 Understanding the near future... 12/05/2011 EGI User Virtualization Workshop 21