European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director

Outline Project objectives Administrative structure Technical structure The EMI Middleware Relationships and Collaborations Conclusions EMI Overview

State of the Art and Objectives EMI Overview

A European Vision Tomorrow Today Sustainability Persistence Interoperability Easier Access Today EMI Overview

What is EMI? The European Middleware Initiative (EMI) project represents a close collaboration of the three major middleware providers - ARC, gLite and UNICORE, together with other software providers - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI and other distributed e-Infrastructures EMI Overview

Objectives Consolidate the existing middleware distribution streamlining services and components to meet and exceed the operational requirements of EGI and other distributed computing infrastructures Develop and standardize the middleware services/functionality following the requirement of EGI and other DCIs, fully integrating virtualization, MPI, monitoring interfaces and embracing new computing models like clouds Consolidate Evolve EMI Overview

Objectives Reactively and proactively maintain the middleware distribution to provide users with increasingly user-friendly, maintainable, reliable, stable, and scalable software Promote the EMI achievements in the user communities, defining and implementing standards when needed, contributing to the establishment of a sustainable model Support Promote EMI Overview

Project Governance EMI Overview

Partners (23) EMI Overview

Administrative Structure Collaboration Board (CB) ECB Project Director Technical Director Project Executive Board (PEB) Project Technical Group (PTG) EMI Overview

Project Structure Administrative and Technical Management Dissemination and Communication Quality Assurance Maintenance and Support Integration, Interoperability and Testbeds Development and Evolution EMI Overview

Technical Structure EMI Overview

Technical Areas Information Services Data Management Job Management Security Quality Assurance Maintenance and Support Integration, Interoperabilty and Testbeds Development and Evolution EMI Overview

Product Teams Product teams are the services implementation teams within each area responsible to deliver software releases and all associated material. They perform the required technical tasks from design to release through implementation, testing, certification and 3rd-level support as part of one or more Work Package and according to what specified by the Project Technical Group and the QA policies. Product Teams are flexible, in the sense that they can be formed or closed as the corresponding products are introduced or obsoleted in EMI and allow adding or removing services as needed even from external contributors They provide a transparent and direct method to assign responsibility for a service EMI Overview

Product Teams Job Management Data Management Information Services Security EMI Release x.y.z-r x.y.z-r Quality Assurance Maintenance and Support Integration, Interoperabilty and Testbeds Development and Evolution EMI Overview

Software Engineering PT Release Stress tests Scalability tests Functional tests Regression tests Deployment tests Packages Unit tests Build Develop Design Quality Assurance Maintenance and Support Definition Execution Integration, Interoperabilty and Testbeds Development and Evolution EMI Overview

Quality Assurance Improve Release Stress tests Scalability tests Functional tests Regression tests Deployment tests Packages Unit tests Build Develop Design ISO 12207 (Software Lifecycle Processes) ISO/IEC 14764 (Software Maintenance) ISO/IEC 29119 (Software Testing) ISO 9126 (Software Quality) Quality Assurance Maintenance and Support Integration, Interoperabilty and Testbeds Development and Evolution Monitor EMI Overview

Releases and Release Policies Major releases: once or twice per year, may contain non-backward compatible changes Minor releases: a few times per year, fully backward compatible, may contain new functionality Revisions: every week or two weeks, only bug fixes Emergency: as needed, only specific bug fixes, use emergency release procedures Service Level Agreements EMI Overview

The Middleware EMI Overview

Selection Criteria EMI includes: Middleware services and components from the participating providers All services that are or will be in production by next year EGI, PRACE operations must be fully supported New services that are supposed to replace existing services New services satisfying identified new user needs EMI requirements are user-driven, satisfy the needs of the user communities (Archaeology, Astronomy, Astrophysics, Civil Protection, Computational Chemistry, Earth Sciences, Finance, Fusion, Geophysics, High Energy Physics, Life Sciences, Multimedia, Material Sciences, etc.) EMI Overview

Day-0 Distribution Job Management Data Management Information Services Grid Manager A-REX libarclient ng* job CLI arc* job CLI JURA TSI XNJS UAS-Compute OGSA-BES URC UCC HILA WMS LB CREAM MPI Classic SE libarcdata2 ng* data CLI arc*data CLI Chelonia dCache StoRM UAS-Data DPM lcg_utils DPM/DICOM LFC FTS Hydra AMGA Classic Infoserver Classic Infoindex ALIS ISIS libarcclient (infosys) Grid Monitor CIS Service Registry BDII lcg-info and lcg-infosites Service Discovery API CEMon Messaging APEL HED security API arcproxy Charon Gateway XUUDB XACML Entity UVOS VOMS/VOMS-Admin voms-proxy-* ARGUS SLCS SCAS/LCAS/LCMAPS gLite-Exec (glExec) gLite-Delegation Client Shibboleth SLCS Client Gridsite CGSI_gSoap delegation-Java Trustmanager/Util-Java Proxyrenewal Job Management Data Management Information Services Security EMI Overview

Harmonization Targets Data Handling Unification APIs and Clients Harmonization Standardization (PGI, SRM, SAML/XACML, US/RUS, GLUE,etc) Common Service Registry Common Security Framework EMI Overview

New Development New services or functionality are introduced as needed based on: New user requirements Need to evolve existing services or replace older technologies to follow the growing needs of the infrastructures Special focus on integration of virtualization (grids on clouds), monitoring interfaces, support for easy construction of gateways Trade between new research and stability of the infrastructure operations Large-scale realistic testing, phase-out/migration plans, support policies EMI Overview

Relationships EMI Overview

EGI EGI MCB, SA2 (MU), SA3, SA4 3rd-level support Releases Specs Requirements Repositories, tests, certification tools, etc PTG EMI Overview

User Communities SSCs EGI SSCs MCB, SA2 (MU), SA3, SA4 SSCs 3rd-level support Releases Specs Requirements Repositories, tests, certification tools, etc PTG EMI Overview

Interoperability partners, commercial partners, Collaborations SSCs SSCs EGI MCB, SA2 (MU), SA3, SA4 SSCs 3rd-level support Releases Specs Requirements Repositories, tests, certification tools, etc PTG Works with EMI Interoperability partners, commercial partners, EMI Overview

Exploitation and Sustainability Successful deployment in EGI, PRACE and other infrastructures of the EMI middleware Collaboration with standardization bodies Evolution of existing standards Definition of new standards Inclusion of middleware in OS distributions Packaging requirements, support, conflict management Service Level Agreements Move towards a customer/provider model that could in time become a real commercial relationship EMI Overview

Open Collaborations EMI is not a private club Collaborations with additional software providers are welcome and sought after “Works with EMI” Program Collaborating providers need to follow the same rules as the EMI partners for the software to be included in EMI releases Openness is necessary to converge towards usable common standards and seriously interoperating solutions within and outside Europe Industrial collaborations are welcome EMI Overview

Conclusions EMI Overview

Conclusions EMI is built on the strengths of the major European middleware providers To provide a reliable, usable, scalable, interoperable middleware distribution It is designed to be the major middleware provider for EGI, PRACE and other DCIs Support for reliable operations Evolve the services as the communities grow It’s an important step forward in the establishment of a true pan-European sustainable, persistent research infrastructure EMI Overview

Thank you