Presentation is loading. Please wait.

Presentation is loading. Please wait.

EMI is partially funded by the European Commission under Grant Agreement RI-261611 Development Roadmap of the EMI middleware Balázs Kónya, Lund University,

Published byDustin Atkins Modified over 8 years ago

Similar presentations

Presentation on theme: "EMI is partially funded by the European Commission under Grant Agreement RI-261611 Development Roadmap of the EMI middleware Balázs Kónya, Lund University,"— Presentation transcript:

1 EMI is partially funded by the European Commission under Grant Agreement RI-261611 Development Roadmap of the EMI middleware Balázs Kónya, Lund University, Sweden Project Technical Director and L. Field, J. White, P. Fuhrmann, M. Cecchi, J.K. Nilsen CHEP 2012, 21-25 May 2012, New York

2 EMI INFSO-RI-261611 EMI project in a nutshell Year 2 development through showcases Year 3 development in the pipeline for details see the numerous EMI posters and presentations Roadmap: a high level view 24/5/2012www.eu-emi.eu2 CHEP in Taipei, October 2010 EMI started, May 2010 CHEP 2012, EMI 2 release EMI ends, April 2013

3 EMI INFSO-RI-261611 Project factsheet 24/5/2012www.eu-emi.eu European FP7 project 24 partners with CERN as the coordinator 3 years with April 2013 end 1115 person months 12 M Euro EU funding EMI brings together the major EU middleware developers 3

4 EMI INFSO-RI-261611 3 years of harmony 24/5/2012www.eu-emi.eu 4 Standards, New technologies, Users and Infrastructure Requirements EMI common products 3 yearsBefore EMIAfter EMI Implemented Agreements Repositories

5 EMI INFSO-RI-261611 Main goals 5 Maintenance Support Release Maintenance Support Release Innovation and Development Dissemination Training Exploitation Sustainability Dissemination Training Exploitation Sustainability EMI 1 Kebnekaise Legacy pre-EMI MW EMI 2 Matterhorn Web site, communication channels DCI and other collaborations ScienceSoft Commercial collaborations Software Eng. Quality Improvements 24/5/2012www.eu-emi.eu

6 EMI INFSO-RI-261611 EMI Releases 24/5/2012www.eu-emi.eu6 support period: 12+6+6

7 EMI INFSO-RI-261611 One year maintenance & support 119 component updates during 14 EMI-1 update cycles 1218 bugs fixed 493 incidents closed 24/5/2012www.eu-emi.eu7

8 EMI INFSO-RI-261611 Year of the Development 24/5/2012www.eu-emi.eu EMI 1 Kebnekaise Open Source process adoption Distribution consolidation EMI 2 Matterhorn New functionality New products Agreements and standardization 29 Product Teams 57 products 101 planned development tasks 75 completed 19 prototypes 7 postponed 8

9 EMI INFSO-RI-261611 EMI 2: congratulation! 24/5/2012www.eu-emi.eu9 56 Certified Products

10 EMI INFSO-RI-261611 EMI Ecosystem 24/5/2012www.eu-emi.eu10

11 EMI INFSO-RI-261611 EMI Ecosystem zoomed 24/5/2012www.eu-emi.eu11

12 EMI INFSO-RI-261611 Major new features and services Implementation of a common EMI interface for job management (EMI-ES) Extended back-end support for all EMI CEs to Torque, SGE, LSF, SLURM All the EMI services publish GLUE2 conformant information A common service registry for all EMI services (EMIR) Comprehensive set of Nagios probes for every EMI service A prototype of the common EMI security library (CANL) ARGUS as the official EMI authorization solution via integration with all the SEs and Ces … Technical Agreements and Standardization EMI Delegation agreement Storage Accounting record (StAR) and Compute Accounting record (CAR) A common EMI SAML profile … Year of the Development 24/5/2012www.eu-emi.eu12

13 EMI INFSO-RI-261611 Show-case 1: Data Industry Standards 24/5/2012www.eu-emi.eu13 Industry standard protocols for accessing SEs and the catalog DPM and dCache ready for NFS4.1 HTTPS offered by DPM, StoRM and dCache WebDAV support in DPM and dCache WebDAV support being developed in FTS3 and LFC Vital part of the greater vision for EMI Data

14 EMI INFSO-RI-261611 Show-case 2: EMI service Registry 24/5/2012www.eu-emi.eu14 EMIR: from implementation through performance testing to deployment planning One central registry where all services can be discovered No registry supported by all three middleware providers prior to EMI Unified service discovery Quorum based, replicated Global Service Registry DB ensures HA Service providers register to and push information to Domain Service Registries Deployment hierarchy of DSRs in investigation All services to publish information in EMIR

15 EMI INFSO-RI-261611 Show-case 3: EMI Execution Service 24/5/2012www.eu-emi.eu15 EMI-ES: from specification to implementations EMI-ES interface specification agreed as the common job management interface Web-service interface with Integrated support for data staging Delegation capability Re-engineered state model Revised job description Glue2-based service and activity description Clearly defined Port Types All CEs implemented EMI-ES job management specification at 80-90% completion level Iterative schema definition to protect users from child deceases EMI-ES v1.2 revision Client side development ongoing Central piece in the EMI harmonisation efforts

16 EMI INFSO-RI-261611 Show-case 4: Common Authentication Library 24/5/2012www.eu-emi.eu16 CANL: from implementation to adoption Defined and documented API for common security library Main features Credentials handling Trust store handling Name constraints checking CRL Proxy: verification, generation, proxy CSRs, utilities Partially unified error codes and messages OCSP support (on-line revocation) PKCS 11 (support for smart cards & soft tokens) Implementation available in C, C++ and Java EMI products migrating to CANL VOMS, ARC HED, Trustmanager, L&B, UNICORE, dCache, CREAM, Argus, Pseudonymity, Hydra, STS

17 EMI INFSO-RI-261611 Show-case 5: Car & StAR 24/5/2012www.eu-emi.eu17 EMI agreement on accounting records Compute accounting CAR Storage accounting StAR To be used ”everywhere” accounting sensors APEL Feeding all this back to OGF UR 2.0 (?)

18 EMI INFSO-RI-261611 EMI Compute Accounting Record (CAR) Mainly a profile of the OGF UR1.0 Some differences: – Added support for groups/VOs Using Group+GroupAttribute like in StAR – Definition encompasses both usage records and aggregated records – Aggregated schema follows APEL SSM guidelines Show-case 5: Car & StAR 24/5/2012www.eu-emi.eu18

19 EMI INFSO-RI-261611 StAR: Definition of usage record for storage Inspired by OGF UR1.0 Summarises used space XML-based schema Non-overlapping/non-contiguous records means no space used – ValidDuration configured by sys-admin Supports handling of groups (aka VOs) Scope is limited to consumption of storage space 24/5/2012www.eu-emi.eu19 Show-case 5: Car & StAR

20 EMI INFSO-RI-261611 General strategy: – Complete product developments: – FTS3, GFAL2 – STS – EMI Datalib – Product hardening, focus on usability – Integration and adoption of common EMI solutions (EMIR, CANL) – Migration plans, compatibility The last peak (Y3 development plans) 24/5/2012www.eu-emi.eu20

21 EMI INFSO-RI-261611 See talk later Today In the pipeline: FTS3 (GFAL2) 24/5/2012www.eu-emi.eu21

22 EMI INFSO-RI-261611 Security Token Service – For simplified credential management – STS transforms an existing security token into another security token – SOAP-based Web Service forge.switch.ch/redmine/projects/sts/wiki In the pipeline: STS 24/5/2012www.eu-emi.eu22

23 EMI INFSO-RI-261611 In the pipeline: emi_datalib 24/5/2012www.eu-emi.eu23

24 EMI INFSO-RI-261611 emi_datalib: before 24/5/2012www.eu-emi.eu24 file-based data-moving library – libarcdata2 pluggable architecture libarcdata2 is used by the arc command tools, the ARC CE and external clients to upload and download files written in C++ posix-like data access pluggable architecture several external clients (e.g. experiments software) use GFAL and lcg_util libraries written in C

25 EMI INFSO-RI-261611 Emi_datalib: after 24/5/2012www.eu-emi.eu25 posix-like interface from GFAL2 higher-level file-based interface from libarcdata2 adds transfer interface to GFAL2 for initiation and monitoring of 3rd-party transfers libarcdata2 will use GFAL2 through plugin external clients requiring posix-based byte-wise data access can use GFAL2 directly file-based data moving clients (e.g. lcg_util CLI, ARC CLI and ARC CE) can use libarcdata2 FTS3 and parts of the lcg_utils and ARC CLIs will use the 3rd- party transfer interface of GFAL2 python library will be created to the needed functionality from the lcg_util python API all plugins are moved under GFAL2 though some plugins are file-based and not posix – allows posix access without needing to know physical file locations EMI_Datalib

26 EMI INFSO-RI-261611 Three-year-long EMI project has given an excellent one- time opportunity to work together Most development intensive phase of EMI is just behind us With Matterhorn EMI delivered – Common libraries (CANL) – Common service (EMIR) – Common implemented interfaces (EMI-ES over CEs) – Agreements and adoption plans (CAR-STAR, …) The third last year still to bring some interesting stuff but the focus is now on product hardening – Planned for EMI 3: STS, FTS3, GFAL, emi_datalib Summary 24/5/2012www.eu-emi.eu26

27 EMI INFSO-RI-261611 24/5/2012www.eu-emi.eu EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 27

Download ppt "EMI is partially funded by the European Commission under Grant Agreement RI-261611 Development Roadmap of the EMI middleware Balázs Kónya, Lund University,"

Similar presentations

EMI INFSO-RI-261611 NA2 – Outreach and Collaborations Emidio Giorgio (INFN Catania) NA2 Leader 1 st EMI Periodic Review Brussels, 22 June 2011.

EMI INFSO-RI NA2 – Outreach and Collaborations Emidio Giorgio (INFN Catania) NA2 Leader 1 st EMI Periodic Review Brussels, 22 June 2011.
EMI INFSO-RI-261611 European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.

EMI INFSO-RI European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.
EMI INFSO-RI-261611 European Middleware Initiative (EMI) Standardization and Interoperability Florida Estrella (CERN) Deputy Project Director.

EMI INFSO-RI European Middleware Initiative (EMI) Standardization and Interoperability Florida Estrella (CERN) Deputy Project Director.
Storage Accounting John Gordon, STFC GDB June 2012.

Storage Accounting John Gordon, STFC GDB June 2012.
EMI INFSO-RI-261611 European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.

EMI INFSO-RI European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.
EMI INFSO-RI-261611 EMI roadmap (development plan) Balázs Kónya (Lund University, Sweden) EMI Technical Director 25 th October 2010, Brussels, EGI-TCB.

EMI INFSO-RI EMI roadmap (development plan) Balázs Kónya (Lund University, Sweden) EMI Technical Director 25 th October 2010, Brussels, EGI-TCB.
EMI is partially funded by the European Commission under Grant Agreement RI-261611 Recent ARC developments in the EMI project Andrii Salnikov, Ievgen Sliusar.

EMI is partially funded by the European Commission under Grant Agreement RI Recent ARC developments in the EMI project Andrii Salnikov, Ievgen Sliusar.
EMI INFSO-RI-261611 EMI Quality Assurance Processes (PS06-4-499) Alberto Aimar (CERN) CERN IT-GT-SL Section Leader EMI SA2 QA Activity Leader.

EMI INFSO-RI EMI Quality Assurance Processes (PS ) Alberto Aimar (CERN) CERN IT-GT-SL Section Leader EMI SA2 QA Activity Leader.
EMI is partially funded by the European Commission under Grant Agreement RI-261611 Post EMI Plans and MeDIA Alberto DI MEGLIO, CERN Project Director WLCG.

EMI is partially funded by the European Commission under Grant Agreement RI Post EMI Plans and MeDIA Alberto DI MEGLIO, CERN Project Director WLCG.
EMI is partially funded by the European Commission under Grant Agreement RI-261611 Software stack consolidation Balázs Kónya, Lund University 3rd EMI all-hands,

EMI is partially funded by the European Commission under Grant Agreement RI Software stack consolidation Balázs Kónya, Lund University 3rd EMI all-hands,
EMI is partially funded by the European Commission under Grant Agreement RI-261611 MEDIA: motivation, mandate, scope and organization Balázs Kónya, Lund.

EMI is partially funded by the European Commission under Grant Agreement RI MEDIA: motivation, mandate, scope and organization Balázs Kónya, Lund.
EMI INFSO-RI-261611 ARC on the European Distributed Computing Infrastructure (DCI) landscape Balázs Kónya (Lund University) NorduGrid Technical Workshop.

EMI INFSO-RI ARC on the European Distributed Computing Infrastructure (DCI) landscape Balázs Kónya (Lund University) NorduGrid Technical Workshop.
EMI INFSO-RI-261611 EMI Structure, Plans, Deliverables Alberto Di Meglio (CERN) Project Director ATLAS Software & Computing Week Geneva, 21 July 2011.

EMI INFSO-RI EMI Structure, Plans, Deliverables Alberto Di Meglio (CERN) Project Director ATLAS Software & Computing Week Geneva, 21 July 2011.
European Middleware Initiative (EMI) – Release Process Doina Cristina Aiftimiei (INFN) EGI Technical Forum, Amsterdam 17. Sept.2010.

European Middleware Initiative (EMI) – Release Process Doina Cristina Aiftimiei (INFN) EGI Technical Forum, Amsterdam 17. Sept.2010.
EMI INFSO-RI-261611 NA2 – Outreach and collaborations Status Report Emidio Giorgio (INFN Catania) Work Package Leader EMI First EC Review 22 June 2011,

EMI INFSO-RI NA2 – Outreach and collaborations Status Report Emidio Giorgio (INFN Catania) Work Package Leader EMI First EC Review 22 June 2011,
EMI 1 Release The EMI 1 (Kebnekaise) release features for the first time a complete and consolidated set of middleware components from ARC, dCache, gLite.

EMI 1 Release The EMI 1 (Kebnekaise) release features for the first time a complete and consolidated set of middleware components from ARC, dCache, gLite.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
EMI is partially funded by the European Commission under Grant Agreement RI-261611 Software development, integration and coordination Balázs Kónya, Lund.

EMI is partially funded by the European Commission under Grant Agreement RI Software development, integration and coordination Balázs Kónya, Lund.
EMI is partially funded by the European Commission under Grant Agreement RI-261611 Project Overview Alberto DI MEGLIO, CERN Project Director 2nd EMI Periodic.

EMI is partially funded by the European Commission under Grant Agreement RI Project Overview Alberto DI MEGLIO, CERN Project Director 2nd EMI Periodic.
WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing.

WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing.

Similar presentations

Ads by Google