1. INFN-Grid MW in OMII-EU e MW distribuito su diversi progetti A.Ghiselli INFN-CNAF EB meeting, 29 agosto 2006

2. EU project: RIO31844-OMII-EUROPE mw list VO oriented authentication, authorisation and accounting (OMII-EU) –VOMS/SAML –DGAS/RUS Job submission (OMII-EU) –BES and JSDL Data Management –StoRM: SRM implementation for GPFS (collaborazione INFN ed Egrid (ICTP)) Grid monitoring –GridICE Agreement Service/Advance Reservation –In GridCC con StoRM e Network Services Integration/Interoperability activity Job priorities Integrazione di G-Pbox con DIRAC Glue-schema Infrastructure interoperability (OMII-EU) Interoperable security framework (OMII-EU)

3. EU project: RIO31844-OMII-EUROPE What is OMII-Europe? EU funded FP6 project (RI) –Starting May 2006, initial 2 year duration –16 partners (8 European, 4 USA, 4 Chinese) Open Middleware Infrastructure Institute for Europe –Complimentary to existing national programmes (OMII-UK, NMI, C-OMEGA, OMII-China…) Goal is to provide key software components for building e-infrastructures Project will demonstrate “proof of concept” with expectation for a follow-on project in FP7

4. EU project: RIO31844-OMII-EUROPE What will OMII-Europe do? Initial focus on providing common interfaces and integration of major Grid software infrastructures Common interoperable services: –Database Access, Virtual Organisation Management, Portal, Accounting, Job Submission and Job Monitoring –Capability to add additional services Infrastructure integration –Initial EGEE/UNICORE/Globus interoperability –Interoperable security framework

5. EU project: RIO31844-OMII-EUROPE OMII-Europe guiding principles Committed to standards process –Implementing agreed open standards and working with standards process (GGF/Oasis) Quality Assurance –Published methodology and compliance test –All software components have public QA process and audit trail –Working with similar projects and organisations to agree policies Impartiality –OMII-Europe is “honest broker” providing impartial advice/information on e-infrastructures

6. EU project: RIO31844-OMII-EUROPE What will OMII-Europe deliver? Repository of open-source, quality assured software services for EGEE, Globus, UNICORE and CROWNgrid –Some services bundled with major grid distributions –Initial integration work with EGEE, UNICORE and Globus Public reports on grid infrastructures –Initial benchmark results –Impartial advice and information Evaluation infrastructure to “test” services –User support and training for services

7. EU project: RIO31844-OMII-EUROPE Who benefits from OMII-Europe? E-infrastructure providers –Choice of grid software to deploy can be determined by selecting the most appropriate system to manage resources. –Achieved through common interfaces and interoperability of grid systems Decisions not constrained by membership of a particular VO Not required to deploy and manage multiple grid distributions E-science users –Access to resources beyond the immediate e-infrastructure running a specific grid distribution –Achieved through low level interoperability of Grid distributions Users not restricted to a specific, fixed set of resources E-science application developers –Applications can be deployed and run on multiple grid environments through adherence to common services Not required to develop different solutions for different grids

8. EU project: RIO31844-OMII-EUROPE Project Structure and Effort Allocation Networking activities –Management, Outreach, Training –8% Person Effort Service Activities –Repository, QA, Support –25% Person Effort Joint Research Activities –Re-engineering, new services, integration, benchmarking –67% Person Effort

9. EU project: RIO31844-OMII-EUROPE Effort (Person Years) per Activity

10. EU project: RIO31844-OMII-EUROPE OMII-Europe JRA1 re-engineering activities OGSA DAIBESVOMSRUS Grid Sphere Etc. Identified Components EGEE (GLite) UNICORE Globus Etc. OMII-UK, USA, China

11. EU project: RIO31844-OMII-EUROPE OMII-Europe Project Partners 114 person years over 2 years, 5 million Euro, 4 major Grid infrastructures University of Southampton UK (coordinator)University of Chicago USA Fujitsu Laboratories Europe UKNCSA, University of Illinois USA Forschungszentrum Juelich Germany University of Southern California Los Angeles USA Kungl Tekniska Högskolan SwedenUniversity of Wisconsin-Madison USA Istituto Nazionale di Fisica Nucleare ItalyBeihang University China Poznan Supercomputing & Networking Center Poland China Institute of Computing Technology Beijing China University of Edinburgh UK Computer Network Information Centre Beijing China CERN, European Organisation for Nuclear Research Switzerland Tsinghua University China

12. EU project: RIO31844-OMII-EUROPE OMII-Europe Vision To demonstrate that interoperable Grids can be built from standards- compliant Web Services and to deliver a set of quality-assured services, sourced from open source repositories, able to be used on the principal Grid infrastructures in use in Europe today.

13. EU project: RIO31844-OMII-EUROPE INFN participation JRA! OGSA DAI JRA1BES JRA1 VOMS JRA1R US JRA1 Grid Sphe re JRA2 New service JRA3 Interoperabl e Infrastr. and security framework JRA4 Bench- marking SA1 Repository SA3 Testbed support EGEE (GLite) Giacinto Donvito Giuseppe Fiorentino Moreno Marzolla Sergio Andreozzi V.Venturi F.Stagni A.Gianoli Rosario Piro Dario Russo Annam. Muoio ? Sergio Andreozzi A.Ferraro P.Andretto A.Gianoli E.Luppi Giuseppe Fiorentino Giuseppe Fiorentino Andrea Caltroni Marco Verlato UNICORE Globus Etc. OMII- UK, USA, China

14. EU project: RIO31844-OMII-EUROPE JRA1 – VO management coord. V.Venturi Goal:The definition and implementation of a web service compliant with the SAML Authorization model. July report: The role of VOMS in the overall authorization framework were discussed and better outlined. Semantic and use cases were better defined as those of a Credential Service rather than of an Authorization Service. A document is being produced that describe the authorization framework. Concerns emerged on the version of SAML to use. Similar concerns emerged on OGSI being superseded by WSRF. It will be investigate whether and how properties and notifications could enhance VOMS. A document is being produced defining the interface of the service using SAML that will eventually contains a WSRF interface. Investigations will start on how VOMS credentials could be used in UNICORE's authorization system. UNICORE will use SAML binding of VOMS credentials.

15. EU project: RIO31844-OMII-EUROPE JRA1 – Accounting Coord. R.Piro in attesa di KTH Goal: OMII-Europe will unify accounting information on EGEE, UNICORE and GLOBUS by integrating and supporting a common standardised specification such as RUS on these grid distributions July report FLE, FZJ Input: Investigation of the GGF RUS specification and its integration into UNICORE 6. Establishing the requirements for accounting enhancements to UNICORE 6. KTH: no Input INFN Input: Study of differences between DGAS and RUS functionalities, as well as accounting information stored in DGAS and foreseen in the GGF Usage Record (UR). Active participation in the GGF UR-WG (Usage Record Working Group) for proposing improvements of/additions to the UR format, that currently is very job specific, but should be extended for the purpose of a more generic usage accounting (e.g. storage or other services). UEDIN Input:very little BU Input: No information supplied. Zero effort assumed. CNIC Input: No more progress at CNIC. Resource Usage Specification (RUS)

16. EU project: RIO31844-OMII-EUROPE JRA1 – Job submission coord. M.Marzolla Goal:OMII-Europe will address the extension of the existing job submission systems within GLOBUS, UNICORE and EGEE to meet BES and JSDL standards. In addition to implementing the core functionality, specific extensions necessary for projects such as EGEE will also be defined and fed back to the standardisation process. July report The activity towards the first task milestone was defined –M:JRA1.7, Definition of the required extensions needed to JSDL to satisfy OMII-Europe requirements,M6 We are working towards support of JSDL in the gLite software infrastructure JSDL extensions for gLite and Unicore are being investigated http://grid.pd.infn.it/omii/start (Basic Execution Service (BES))

17. EU project: RIO31844-OMII-EUROPE JRA2 Activity Sergio Andreozzi Objective: –to identify appropriate emerging middleware services from the global Grid initiative –to identify priorities for the placement of such services in the OMII-Europe repository Expected outcome: –report prioritising services for implementation based on user-pull from the potential OMII-Europe user base –report valid to approach potential contributors to the OMII-Europe repository and to lobby at a national or European level for the development of missing components

18. EU project: RIO31844-OMII-EUROPE Prioritization Criteria utility of potential new services to the wider scientific community specifically that in the ERA consolidation of the outcomes of existing Grid activities on a Global scale

19. EU project: RIO31844-OMII-EUROPE Milestones&Deliverables M.JRA2.1: publication of D:JRA2.0 D:JRA2.0: publication of Grid activities report D:JRA2.1, D:JRA2.2: report on the Identification of new services activity Monthly activity report to the Management Report

20. EU project: RIO31844-OMII-EUROPE JRA2 Activity progress for the month From the July report –Definition of the template for comparing different Grid platform –Initial draft of the gLite description –Investigation of other partners’platforms –Update of the JRA2 wiki –http://omii-europe.forge.cnaf.infn.it/jra2/http://omii-europe.forge.cnaf.infn.it/jra2/

21. EU project: RIO31844-OMII-EUROPE Servizi per jra2 Summary Subsystem NameDevelopment StatusAdopted StandardForeseen Standard Security.AuthenticationGT GSImatureIETF RFC3820 VOMSproduction G-PBoxbetaXACML 1.1 XACML for RBAC, SAML 2, XACML for PDP Data.TransferGT GridFTPGridFTP Data.Management.FlatFilesDPMproductionSRM 2.1.1SRM 2.2 Data.Management.FlatFilesStoRMproductionSRM 2.1.1SRM 2.2 AccountingDGASproduction Information.ModelGLUE Schemaproduction Information.Discovery

22. EU project: RIO31844-OMII-EUROPE JRA3 Activity progress for the month Goals are a common security infrastructure and the closer integration of EGEE and UNICORE to facilitate job execution on UNICORE of EGEE submitted jobs. Task1 (security) - initial JRA3 wiki page editing (http://forge.cnaf.infn.it/twiki/bin/view/OMIIEuJRA3) for INFN (maybe, in future, for other JRA3 members) - studying of OGSA, GGF authorization docs - investigation of gLite Cream authorization issues (with gLite gJAF/G-PBox components) - initial editing of the INFN vision about authorization issues with OMII VOMS developers ("An Instantiation of the Authorization Framework) Task2 (integration) - JRA3 wiki page initial editing (http://forge.cnaf.infn.it/twiki/bin/view/OMIIEuJRA3) - initial proposal of a common document template about the main three middleware platforms interoperability (http://forge.cnaf.infn.it/twiki/bin/view/OMIIEuJRA3/InteroperabilityIssu esAndSolutionsDoc)http://forge.cnaf.infn.it/twiki/bin/view/OMIIEuJRA3 http://forge.cnaf.infn.it/twiki/bin/view/OMIIEuJRA3/InteroperabilityIssu esAndSolutionsDoc

23. EU project: RIO31844-OMII-EUROPE Outline of StoRM presentation SRM 2.2 meeting at CERN 29-31 August 1.StoRM Origin and StoRM team Quick overview of architecture 2.Current Status − Functionalities developed 3.Timeline expected –Functionalities incoming

24. EU project: RIO31844-OMII-EUROPE StoRM Origin Result of collaboration between: INFN - Grid.IT Project : to build a disk-based SRM service on top of high performance parallel file systems. + ICTP - EGRID Project : to build a pilot national grid facility for research in Economics and Finance (www.egrid.it)

25. EU project: RIO31844-OMII-EUROPE StoRM : What is’t? StoRM is a disk based Storage Resource Manager which: –implements SRM specification version 2.1.1 –is migrating to the SRM version 2.2 –is designed to support guaranteed space reservation. –supports direct access (native posix I/O calls). Other access protocols remain available (e.g., rfio). –takes advantage of high performance Cluster File System with ACL support, such as GPFS. Other posix file systems are supported (e.g., ext3) –Authentication and Authorization are based on VOMS certificates.

26. EU project: RIO31844-OMII-EUROPE Front end (FE) has responsibilities of: expose a web service interface manage connection with authorized clients store asynchronous request into data base retrieve asynchronous request status co-operate with backend directly for synchronous call manage user authentication co-operate with external authorization service to enforce security policy on service StoRM architecture

27. EU project: RIO31844-OMII-EUROPE Data Base : Store SRM request and status Store application data Back end (BE) has responsibilities of: execute all synchronous (active) action get asynchronous request from data base execute all asynchronous action bind with underlying file systems enforce authorization policy on files manage SRM file and space metadata StoRM architecture

28. EU project: RIO31844-OMII-EUROPE Service configuration ( XML based ) Per VO configuration –Default values (Space size, lifetime, …) –Namespace (relative path names) –Protocols allowed (and other capabilities) –Authorization model adopted –Drivers for wrapping underlying Filesystem Service customization (Property file) –Service properties (port, tuning parameters, …) Properties file that overwrite default service configuration StoRM configurability

29. EU project: RIO31844-OMII-EUROPE We started the migration from the released version of StoRM implementing SRM 2.1.1_modified (begin of July). The strategies adopted is adapting to the new SRM version the functionalities already developed first. Migration Status to SRM v2.2 of StoRM

30. EU project: RIO31844-OMII-EUROPE Status Summary StatusFunctionality Currently available srmPing, srmGetProtocols srmMkdir,srmRmdir, srmRm, srmLs srmReserveSpace, srmGetSpaceMetadata srmStatusOf[*] available in the next week / mid of September srmPrepareToGet, srmPrepareToPut, srmPutDone, srmCopy (push mode) available by mid of September srmBringOnLine, srmReleaseFile, srmMv Available in near future srmReleaseSpace, srmChangeSpaceForFile, srmPurgeFromSpace

31. EU project: RIO31844-OMII-EUROPE Status details (1/4) Directory functionalities srmMkdir : Done. srmRmDir : Done. srmRm : Done. srmLs : Done. –Only synchronous version is provided –Limit on returned entries configurable (admin. config) srmMv : Not provided yet (available by Sept. or before)

32. EU project: RIO31844-OMII-EUROPE Status details (2/4) Data Transfer functionalities srmPrepareToGet : Almost done. (by mid of Sept or before). srmPrepareToPut : Almost done. (by mid of Sept or before). srmPutDone : Almost done. (by mid of Sept or before). srmCopy : Almost done. (by mid of Sept or before). –Push mode only. –The management of SRM_BUSY is not completely tested. srmBringOnLine : Not provided yet (available by Sept.) –We need more analysis for disk only solution ( no “state” or “copies” of SURL within StoRM…) “Other asynch func.” : Not provided yet (no schedule)

33. EU project: RIO31844-OMII-EUROPE Status details (3/4) Status functionality srmStatusOf [PtP/PtG/Copy/Bol] : Done. Discovery functionality srmPing: Done. –otherInfo[] : ? … some example could be useful. srmGetProtocols: Done. –Return all protocols managed by StoRM instance (admin. config) –Protocol ‘file://’ is supported

34. EU project: RIO31844-OMII-EUROPE Status details (4/4) Space Management functionality srmReserveSpace : Done. –Guaranteed space is managed only for GPFS through space allocation mechanism. srmGetSpaceMetadata : Done. srmReleaseSpace : Not provided yet (no scheduled) srmChangeSpaceForFiles : Not provided yet (no scheduled) srmPurgeFromSpace : Not provided yet (no scheduled)

35. EU project: RIO31844-OMII-EUROPE Timeline summary As soon as possible (by the second week of Sept): –StoRM with v2.2 endpoint available for interoperability tests. By the mid of September or before: –All functionalities tagged as “almost done” will be provided. ( PtG, PtP, Copy, PutDone ). By September or before: –BringOnLine and ReleaseFiles Other functionalities: –The schedule will follow the agreement …

36. EU project: RIO31844-OMII-EUROPE Security aspects 1.User perform srmPrepareToPut 2.StoRM verifies if the principal holds a valid proxy certificate and delegates the external policy decision point to validate the request. 3.StoRM then queries the Authorization Sources to verify if the user can perform the specified operation on the SURL 4.StoRM queries LCMAPS to obtain local user account corresponding to the grid identity of the requestor 5.Physical file name derives by SURL and user attributes (Virtual organization name space) 6.The file system wrapper enforces permissions by setting a new ACL on the physical file. 7.The user job can be executed into the worker node 8.The application can perform a standard POSIX call to access the file into/from the storage system.

37. EU project: RIO31844-OMII-EUROPE Links –Hepix Spring meeting 2006 (3-7 April 2006) : SlidesSlides –CHEP’06 (13-17 February 2006) : Poster and PaperPoster and Paper –Fourth EGEE Conference (23-28 October 2005) : SlidesSlides –StoRM web sites : http://grid-it.cnaf.infn.it/storm http://www.egrid.it/sw/storm/ http://forge.cnaf.infn.it/projects/storm/ –Source code http://www.egrid.it/cvs/storm?root=cvs

38. EU project: RIO31844-OMII-EUROPE INFN – CNAF coordinator : A.Ghiselli developers : A.Forti, L.Magnoni, R.Zappi eGRID - ICTP coordinator :A.Nobile, S.Cozzini developers : E.Corso, A.Messina, A.Terpin INFN-Bo testing activity : V.Vagnoni (LHCb) StoRM Team

39. EU project: RIO31844-OMII-EUROPE Integration Activities ECGI Job priorities (presentazione il 25 luglio 06) Partecipanti T1: Alessandro I. e Davide S., F.Rosso G-Pbox: Gianluca, Andrea F., Vincenzo C., Marco C., Francesco G., Salvatore M. Glueschema: Sergio A. Esperimenti: Andrea S. Partecipazione al TCG Job Priorities WG (D.Liko) –Implementation and Exploitation of Fair Shares in EGEE. S.Andreozzi, S.Monforte, G.L. Rubini, J. Templon. Jul 20, 2006 –Status of the discussion of the TCG Job Priority WG. D. Liko Jul 19th, 2006. –http://www.cnaf.infn.it/~andreozzi/wiki/EGEE-JP/HomePage

40. EU project: RIO31844-OMII-EUROPE Problem Statement VO’s want to be able to differentiate the access to the computing resources based on VOMS credentials (i.e., group and role) Optimization on the basis of the job type is also required (e.g., long vs. short) The differentiation is meant to be mainly in term of assigning dedicated shares of the VO resources to sets of VOMS credentials

41. EU project: RIO31844-OMII-EUROPE Workplan Two main phases are envisioned: –Phase 1 (Short term): to be deployed in the current gLite middleware with minimum requirements for changes Small set of VOMS credentials to be considered Static mapping of VOMS credential to shares –Phase 2 (Medium/long term): Full set of VOMS credentials to be considered Dynamic system for mapping VOMS credentials to shares

42. EU project: RIO31844-OMII-EUROPE Phase 1 – Short Term NIKHEF and INFN-T1 provided expertise for performing experiments Considered VOs are CMS and ATLAS NIKHEF proposes a direct and less structured group to share mapping INFN-T1 proposes and indirect and more structured group to share mapping (based on the service class concept) LRMSCE/queue# sharesInfoService CNAFLSFone per VO4 (gold, silver, bronze, express) VO shares are VOViews within the VO CE NIKHEFTorque/ MAUI one2Each share is a VOView; all VOView part of the same CE/queue

43. EU project: RIO31844-OMII-EUROPE Indirect Association via Service Classes Given the resources of each VO, we propose to configure a certain/stable set of service classes Service classes have label and uniform set of characteristics VOMS credendials are mapped into service classes Requirements: –For each VO, the service classes have to be implementated at each site

44. EU project: RIO31844-OMII-EUROPE INFN interest on Phase 2 It is possible to consider dynamic group/role – share mapping (based on G-PBox) Accounting information can help in defining threshold for resource consumption Test nell’INFN della fase 2: cnaf, padova –Experimento CMS –sottomissione di job di prod, di analisi, e job brevi ad alta priorita’. –Utilizzo del WMS+G-Pbox di VO (Glite 3.0) con le policy di accesso alle diverse classi di servizio. –Prossimo step: introduzione dei G-Pbox di sito con la distribuzione delle policy.

45. EU project: RIO31844-OMII-EUROPE G-Pbox / Glite 3.1 Sono state testate le funzionalita’ di WMS (3.0) + PEP+G.Pbox (3.1) –Credenziali VOMS, Voview, e policies per l’accesso alle classi di servizio (4 classi) Preview Glite 3.1 : software per il deployment sul preview –G-Pbox: GUI con wizard per creare Policy di mapping e policy di ACL (in test) –PEP per il WMS (in attesa del funzionamento del WMS 3.1 per fare i test del PEP testato sul WMS 3.0 –Sistema di distribuzione (testato con 4 Gpbox) –PEP del CE di Glite

46. EU project: RIO31844-OMII-EUROPE Architectural Schema VOMS RB VO G-PBox CESite G-PBoxSite HLR VO HLR Distribution system

47. EU project: RIO31844-OMII-EUROPE GlueSchema Attivita’ di Sergio Andreozzi S.A. e’ il riferimento per il gruppo di lavoro sul GlueSchema (invitato al workshop di OSG (23-25 agosto a Seattle). GlueSchema in OMII-Europe –Come raccomandazioni standard in JRA2 –Come base per l’interoperability in JRA3 Futuri sviluppi del GlueSchema –G.S. v 1.3 (richieste di OSG e Nordugrid) –G.S. v 2 (possibile convergenza con CIM) In GGF se ne discute all’interno di GIN

48. EU project: RIO31844-OMII-EUROPE GridICE: Present deployment S.Fantinel Current version: 1.8.3-4 This version is basically the 1.8.3-2 released on the 2nd of December 2005 with some minor patches Installed servers are monitoring Grid resources in the scope of: EGEE EGEE-SWE RDIG EGEE-SEE Grid.it GILDA CMS ATLAS EUMedGrid EUChinaGRID BalticGrid EELA BeGrid The EGEE server (gridice2@cnaf) runs since July 2005; it needs a clean-up procedure (75 Million entries collected) The Grid.it server (gridce4@cnaf) runs since July 2005 without any major intervention and continue to perform very well (24 Million entries collected)

49. EU project: RIO31844-OMII-EUROPE Ongoing release We are working on releasing the v1.9.0 that we hope to finalize by the beginning of September Will be included many new features (not exhaustive list): –Region –GOC DB interfacing / downtime handling –LRMSInfo: preliminary SLA support and basic Site CPU usage efficiency –Usage metering / normalization –New statistic plots with new look & feel (ex: Grid Jobs vs. Local Jobs)

50. EU project: RIO31844-OMII-EUROPE Integrations INFN-T1 –Since April 2006 our Job Monitoring sensor runs on all the INFN-T1 CEs without any major intervention and scalability issue gLite 3.0 –We are working on the sensors integration. We are waiting for the common log file for monitoring and accounting announced some times ago and never released by LCG (nessuna stima). LeMON 2.10.x –The work never started because we were not able to find people really committed to working on it Advanced Notification Service –The development stopped a while ago because of a contract end of the person working on the topic Advanced Storage probes –Not yet ready for production. Need some more development and tests Advanced RB probe –Code is ready for gLite. We need some more time to integrate the info on the GridICE collecting infrastructure

51. EU project: RIO31844-OMII-EUROPE Snapshots

52. EU project: RIO31844-OMII-EUROPE Man power PD: Sergio Fantinel (0.5 fte) Cnaf: G.Misurelli(0.5fte), E.Fattibene (0.5fte), S.Andreozzi (0.2) BA: G.Cuscela, G.Donvito, A.Pierro (tot. 1.5fte)

53. EU project: RIO31844-OMII-EUROPE MW EXP AtlasAliceCMSLHCb Manpower funded WMSXX/ job agent X 8 VOMSXXX2.5 DGASX2.5 GridICE2.2 CEMON4 CREAM???? StoRM?X3+2 (egrid) G-Pbox???2 (.5) AR+AS (GridCC) (0.1) Manpower AR +Grid 4243

54. EU project: RIO31844-OMII-EUROPE ECGI organization Exp. Task forces LCG TCG SA3 Jra1 Osg VDT SA3? JRA1 Testbed: Preview, C.Grandi Certification, ECGI Preproduction, D.Cesini Production, C.Vistoli Computing Coordinators T0,T1 SA1 Mirco Mazzucato Luca dell’Agnello Dario Barberis …… Laura Perini Simone Campana A.Desalvo…… Stefano Belforte Andrea Sciaba…. Umberto Marconi Roberto Santinelli Gianluca Castellani Stefano Bagnasco Piergiorgio Cerello Claudio Grandi Laura Perini Stefano Belforte Daniele Cesini Andrea Sciaba’ C.Grandi F.Prelz E.Molinari A.Gianlle