1. – n° 1 INFN-Grid progress report grid.infn.it Antonia Ghiselli INFN-CNAF Bologna Commissione Scientifica Nazionale 1 22 giugno 2004 Pisa

2. – n° 2 content u INFN-Grid project: introduction u INFN-Grid Production Infrastructure n Sites and resources u INFN Production Grid organization and support u INFN-Grid services n Present release n Next developments u Grid use model (Atlas DC use case) u Conclusions

3. – n° 3 INFN-Grid – goals (started at 2000) 1.To promote computational grid technologies research & development: Middleware 1.Through european and international projects 1. DataGrid, DataTAG, GLUE 2.Internal R&D activities 2.To implement the INFN grid infrastructure 1.National layout: 22 sites 3.To set up the national Grid Infrastructure for the national research community 1.FIRB: Grid.it (open to INAF, INGV, Bioinfo (PD,GE,MI)) 4.To participate to the implementation of the global Grid infrastructure for the LHC community 1.LCG: Tier1 and n*Tier2 5.To set up the eInfrastructure for the European Research Area 1.EU FP6: EGEE : robust grid services for an european infrastructure based on national Grids

4. – n° 4 Organizzazione Executive Board Giunta esecutiva dell’INFN Technical Board Referee groupesperimenti Progetti Europei e nazionali Infrastruttura di produzione Iniziative r&d Project manager mirco.mazzucato Technical coordinators antonia.ghiselli,lamberto.luminari Representative of INFN T1 federico.ruggieri INFN Computing coordinators: ATLAS: laura.perini CMS: paolo.capiluppi LHC-B: domenico.galli ALICE: massimo.masera LCG: alberto.masoni APE: rapuano BABAR: morandin VIRGO: fulvio.ricci CDF: stefano.belforte INFN contacts for other Sciences giorgio.maggi,leonardo.merola Production Italian Grid ROC manager and EGEE ROC Coordinator cristina vistoli Production Grid planning luciano gaido

5. – n° 5 INFN-Grid – collaborations and results u EU – Datagrid (closed March 2004) : middleware development n WMS = job submission to the Grid, s CE and SE selection on the basis of job requirements specification, CPU load, CE-SE network conditions….. s Support for interactive jobs s Job checkpointing s Support for parallel jobs n Virtual Organization authentication and authorization service: VOMS (VO Membership Service, EDG/EDT) u EU – DataTAG (closed March 2004) : inter-grid Interoperability; EU-US collaboration within the GLUE framework n Grid Resources Information modeling: GLUE schema for Computing and Storage Element n Authorization/authentication service : VOMS-VOX integration (EDT-Fnal/CMS coll.) n First WorldGrid demo by nov.2002 within IST2002 and SC2002 events n GridICE, Grid monitoring system based on GLUE schemas extension u Italian Grid.it (on going): Grid management and support infrastructure n First tools in production n R&D on Resource Utilization Policies and Data Management

6. – n° 6 Infn-grid infrastructure u Grid as “ coordinated resource sharing” on a large scale for a multi-institutional and dynamic virtual organisation. u sites 22 n 3 T2 : MI, TO, LNL n 8 T2-T3 : PI, CT, TO, MI, PD, cnaf, Roma1 n 3 sites grid.it (firb) :cnaf, PD, NA + siti INAF (TS, CT) n 14 T3: Bari, Bologna, Cagliari, Ferrara, Genova, Lecce, Legnaro, Napoli, Parma, Pavia, Roma1, Roma2, Roma3, e Trieste n T1 :cnaf n Risorse di esperimento: s CMS: LNL, T1, …. s Atlas: LNF, NA, MI, T1, Roma1…. s Babar : Ferrara, Padova s Virgo : Roma1, NA, Pisa, T1

7. – n° 7 TORINO PADOVA BARI PALERMO FIRENZE PAVIA GENOVA NAPOLI CAGLIARI TRIESTE ROMA PISA L’AQUILA CATANIA BOLOGNA UDINE TRENTO PERUGIA LNF LNGS SASSARI LECCE LNS LNL SALERNO COSENZA S.Piero FERRARA PARMA CNAF ROMA2 INFN CMS T2 T2/3 Atlas T2 T2/3 Alice T2 T2/3 LHCb T2 T2/3 Babar VIRGO T2 (50-100 nodes) T3 (10-15 nodes) T1 Cnaf (~300) grid.it resources INFN (15-25 nodes) INAF (5-10 nodes) INGV (NEC computers), BIO (tbd) general purpose resources (8-15 nodes) Italian – Grid now ( Site/resource map) MILANO National Grid (Internet) Tot. ~ 1000 nodes

8. – n° 8 INFN - Grid infrastructure organization Experiments (VOs) GRID resources Projects/owners Coordination Committee (now EB) Service planning coordination Operations and support coordination VO representatives, Grid technical coord., Operations resp. grid experts Deployment Planning resource Policy application ……. Central management Team Site-man Resource admin GridService support VO admin New VO admin & support VO User support User Application Grid Resource Coordination Experimemt or research org. support release Configuration management Release distribution, documentation and porting Grid Technical coordination Resource availability Resource allocation Policy for shared resources…... VO admin Support for New VO-users From INFN-Grid, LCG, EGEE or National Middleware development projects EGEE/ROC EGEE/CIC

9. – n° 9 Release, Operation & Support organization u Support: 8SA1 + 3cnaf / T1 + 2unf. = 13p n CMS : N.Smirnov(PD), M.Corvo(PD) D.Bonaccorsi(cnaf) n Atlas: G.Negri (MI/cnaf), S.Resconi(MI) n Alice : S.Bagnasco(TO), G.Lore(cnaf) n Servizi generali s Cnaf : D.Cesini, A.Paolini, s PD : M.Verlato s TO: A.Amoroso s NA: G.Tortone s LNL: S.Fantinel u CMT: 1cnaf, 6SA1 man power= 7p n Cnaf: A.Italiano, A.Cavalli, P.Veronesi, M.Donatelli n PD: F.Fanzago n MI L.Vaccarossa (Atlas) n BA, G.Donvito u Release: 4firb + 1sa1 man power= 5p n PD: E.Ferro, A.Caltroni n CNAF: A.Ferraro, R.Zappi, L.Carota(sa1) u Site resource manager 20p n Cnaf : A.Chierici n TO: S.Lusso n MI: L.Vaccarossa (Atlas) n LNL: M.Biasotto n RM1: A.Desalvo(Atlas) n NA: Taurino n CT: G.Andronico n PD: Ferrari n PI: M.Davini n FE: D.Andreotti n PV: C.DeVecchi n PG:M.Mariotti n LNF, ML.Ferrer n RM2, C.Dellefratte n CA: D.Mura n BA: G.Donvito n LE: E. Fasanelli n PR: R.Alfieri n BO : F.Semeria n TS : M.Macorini

10. – n° 10 Present Grid services for users Release infn-grid 2.x based on LCG 2.x General Services u Security based on Globus GSI (Grid Security Infrastructure) n Standard Protocols: X.509 certificates, PKI, GSS-API. n Login once (credential delegation) u VO oriented Authentication/Authorization tools (VOMS with attribute extension of X.509 certificate) u Grid Monitoring System u VO oriented User Support systems Blue: developed by INFN

11. – n° 11 User interacts with CA, VO and Resource Providers 1.Certificates are issued by a set of well-defined Certification Authorities (CAs). 2.Grant authorization at the VO level. 1.Each VO has its own VOMS server. 2.Contains (group / role / capabilities) triples for each member of the VO. 3.RP’s evaluate authorization granted by VO to a user and map into local credentials to access resources Authentication Request C=IT/O=INFN /L=CNAF /CN=Pinco Palla /CN=proxy VOMS pseudo -cert CA’s CERN CESNET CNRS GermanGrid Grid-Ireland INFN NIKHEF NorduGrid LIP Russian DataGrid DATAGRID-ES GridPP US–DOE Root CA US-DOE Sub CA CrossGrid cert-request cert signing cert/crl update Service VO-Manager (administer user membership, roles and Capabilities) Resource Provider (map into Local credential) CAs: Policies and procedures  mutual thrust Grid agreement 1 2 3 4

12. – n° 12 Grid.IT Production Grid: 0perations Portal u User documentation u site managers documentation u Software repository u Monitoring u Trouble tickets system u Knowledge base http://grid-it.cnaf.infn.it

13. – n° 13 Get your personal certificate

14. – n° 14 How to register to a VO

15. – n° 15 Grid groups within the support system Trouble Ticketing System (http://helpdesk.oneorzero.com)

16. – n° 16 GridICE

17. – n° 17 GridICE u A monitoring system for the Grid Infrastructure n Selects grid entities (resources and services) per VOs and sites u Automatic discovery based on Grid Information Service (Globus/MDS2.x) and Glue schema extensions. u Layered architecture: n Measurement service (local monitoring interfaced to the GIS ) n Publisher service n Data collector service with auto-discovery feature n Data Analyzer + Detection and notification service (on going) n Presentation service via web interface u Modularity, flexibility and interoperability u Ongoing activities: n Integration of network resource monitoring n Notification mechanism u Utilizzato in infn-grid, LCG/EGEE, esperimenti (CMS, Atlas..)

18. – n° 18 Grid ICE components Monitoring DB Monitorig Server GIIS GRIS GIIS Server Computing Element/ Storage Element 1 2 3 4 SQL 1: LDAP Query 2: available CE/SE 3: LDAP Query 4: CEIDs, WNs, Steps 3,4 repeated for every CE/SE LDAP Gfx/Presentation WEB

19. – n° 19 <>

20. – n° 20

21. – n° 21

22. – n° 22

23. – n° 23 Data presentation (3)

24. – n° 24 Grid services for scientific applications u Grid job submission n RJS (Remote Job Submission) to specific computers (Globus): s The user submits a job on a WAN computing system providing the address n RJS to Grid domains (INFN) s Via a Grid scheduler without knowing the destination computers u Grid Information System (UK) (GLUE schema) u Data Management (CERN) n Data replica management s Remote Data location via File catalogue and metadata catalogue (RLS Replica Location Service) s Data transfer and access, n GridFTP: Provides high-performance, reliable data transfer u Storage Resource Management (UK) for s Storage allocation s File pinning….. u Grid User Interface :UI, Genius (INFN) u Job Logging & bookkeeping (INFN)

25. – n° 25 New Storage Element INFN-Roma VO server ingv VO server inaf INFN-Milano INFN-Padova VO server cms VO server atlas Grid services (LCG-2 release) INFN-CNAF Computing Element Storage Element GIIS GRIS1 GRIS Information IndexResource Broker User Interface GRAM BDII WorkerNode...WorkerNode INFN-Napoli Computing Element Storage Element GIIS GRIS1 GRIS GRAM WorkerNode...WorkerNode Grid Monitoring (GridICE) RLS Genius SRM interface rfiodcap Castor MSS dCache GRIS1 L&B

26. – n° 26 ‘real’ Grid Job Submission is allowed via the Work load Management Service (INFN within EU- DataGrid) u The user interacts with Grid via a Workload Management System (not directly with GRAM) u The Goal of WMS is the distributed scheduling and resource management in a Grid environment. u What does it allow Grid users to do? To submit their jobs via a “Job description language” To execute them To get information about their status To retrieve their output u The WMS tries to optimize the usage of resources (funzione che sara’ ottimizzata in EGEE) collective

27. – n° 27 u Information to be specified n Job characteristics n Requirements and Preferences of the computing system n Software dependencies n Job Data requirements u It is specified using a Job Description Language (JDL) n It allows to define a set of attributes, the WMS takes into account when making its scheduling decision Job Preparation

28. – n° 28 Job Description Language (JDL) and relevant attributes 2/4 u Based upon Condor’s CLASSified ADvertisement language (ClassAd), that is a fully extensible language and that allows to define easily requirements and preferences to be used by the matchmaking algorithm uExecutable (mandatory) nThe command name uArguments (optional) nJob command line arguments uStdInput, StdOutput, StdErr (optional) nStandard input/output/error of the job uEnvironment (optional) nList of environment settings u InputSandbox (optional) n List of files on the UI local disk needed by the job for running n The listed files will automatically staged to the remote resource u OutputSandbox (optional) n List of files, generated by the job, which have to be retrieved

29. – n° 29 Job Description Language (JDL): relevant attributes 3/4 uRequirements nJob requirements on computing resources nSpecified using attributes of resources published in the Information Service nIf not specified, default value defined in UI configuration file is considered sDefault: other.Active (the resource has to be able to accept jobs) uRank nExpresses preference (how to rank resources that have already met the Requirements expression) nSpecified using attributes of resources published in the Information Service nIf not specified, default value defined in the UI configuration file is considered sDefault: -other.EstimatedTraversalTime (the lowest estimated traversal time)

30. – n° 30 Job Description Language (JDL): “data” attributes 4/4 uInputData (optional) n Refers to data used as input by the job: these data are published in the Replica Catalog and stored in the SEs) n PFNs and/or LFNs u DataAccessProtocol (mandatory if InputData has been specified) n The protocol or the list of protocols which the application is able to speak with for accessing InputData on a given SE u OutputSE (optional) n The Uniform Resource Identifier of the output SE n RB uses it to choose a CE that is compatible with the job and is close to SE

31. – n° 31 GUI & APIs

32. – n° 32 WMS Components u WMS is currently composed of the following parts: 1.User Interface (UI) : access point for the user to the WMS 2.Resource Broker (RB) : the broker of GRID resources, responsible to find the “best” resources where to submit jobs 3.Job Submission Service (JSS) : provides a reliable submission system 4.Information Index (II) : a specialized Globus GIIS (LDAP server) used by the Resource Broker as a filter to the information service (IS) to select resources 5.Logging and Bookkeeping services (LB) : store Job Info available for users to query

33. – n° 33 WMS Architecture in EDG v2.x UI RLS Inform. Service Network Server Job Contr. - CondorG Workload Manager RB node CE characts & status SE characts & status RB storage Match- Maker/ Broker Job Adapter Log Monitor Logging & Bookkeeping

34. – n° 34 Other WMS UI Commands u dg-job-list-match Lists resources matching a job description Performs the matchmaking without submitting the job  dg-job-cancel Cancels a given job  dg-job-status Displays the status of the job u dg-job-get-output Returns the job-output (the OutputSandbox files) to the user  dg-job-get-logging-info Displays logging information about submitted jobs (all the events “pushed” by the various components of the WMS) Very useful for debug purposes  dg-job-id-info A utility for the user to display job info in a formatted style

35. – n° 35 WMS release 2 functionalities u User APIs u GUI u Support for interactive jobs u Job checkpointing u Support for parallel jobs u Support for automatic output data upload and registration u VOMS(Management Service) support u Support for job dependencies (via DAGman) n “Lazy” scheduling: job (node) bound to a resource (by RB) just before that job can be submitted (i.e. when it is free of dependencies)

36. – n° 36 What is a DAG? u Directed Acyclic Graph u Each node represents a job u Each edge represents a (temporal) dependency between two nodes n e.g. NodeC starts only after NodeA has finished Node A Node B Node D Node C Node E

37. – n° 37 WMS: Future activities u New functionalities n Support for job partitioning (available soon)  Use of job checkpointing and DAGMan mechanisms n Original job partitioned in sub-jobs which can be executed in parallel n At the end each sub-job must save a final state, then retrieved by a job aggregator, responsible to collect the results of the sub-jobs and produce the overall output Grid Accounting : DGAS (testing in progress)  Based upon a computational economy model n Users pay in order to execute their jobs on the resources and the owner of the resources earn credits by executing the user jobs  help the Workload Management System in balancing the workload by supporting economic resource brokering n VO-based Resource access Policy support: G-PBOX (development in progress); Grid resource sharing requires: s to deploy VO-wide policies. s to respect local site policies. s to specify policies relating to the behavior of the grid as a whole. s RB can take decision on the basis of VO/user policies

38. – n° 38 VO oriented Policy system: G-PBOX Policy Examples: n Users belonging to group /vo/a may only submit 10 jobs a day. n Users belonging to group /vo/b should have their jobs submitted on the max priority queue. n “some user” is banned from the CNAF site. Requirements. The system should: n Be VO-based and distributed. n Be highly configurable and able to define and enforce previously unknown types of policies. n Leave total control on local sites to local admins. n Be capable of express policies requiring a global view of the grid. n Be compliant to existing protocols and not require their redesign. u Objective: help the Workload Management System in Grid resource selection.

39. – n° 39 Data Management (CERN): requirements In EU-DataGrid and Globus the technique adopted for optimizing data access and providing fault tolerance is data replication. The Requirements are: u Planning, scheduling and monitoring execution of data requests and computations u Management of data replication n Register and query for replicas n Select the best replica for a data transfer u Security n Protect data on storage systems n Support secure data transfers n Protect knowledge about existence of data u Virtual data n Desired data may be stored on a storage system (“materialized”) or created on demand

40. – n° 40 Data Management services u A Replica Location Service (RLS) is a distributed registry service that records the locations of data copies and allows discovery of replicas u Maintains mappings between logical identifiers and target names n Physical targets: Map to exact locations of replicated data n Logical targets: Map to another layer of logical names, allowing storage systems to move data without informing the RLS u RLS was designed and implemented in a collaboration between the Globus project and the DataGrid project u EDG: RM + RLS (problemi di lentezza… altre soluzioni?) u Necessita’ di RLS locali: problema della consistenza delle repliche, che si sta studiando in INFN-Grid.

41. – n° 41 Storage Resource Management adopteb by LCG Data are stored on disk pool servers or Mass Storage Systems based on different technologies. secondarytertiary storage local storage management SRM (Storage Resource Manager) is a Grid Service that takes care of local storage interaction ( to several secondary and tertiary storage systems) and provides a Grid interaface to outside world ( hiding the details of the local storage management from the user). (Original design protocol: LBL, JNL, FNAL, CERN) storage resource management takes into account Transparent access to files (migration to/from disk pool) File pinning n Files can be read from disk caches rather than from tape Space reservation File status notification…. attualmente c’e’ la versione per castor e dcache

42. – n° 42 First version of SE: disk server with GridFTP and NFS server protocols New SE version: u SRM interface n Management and control n SRM (with possible evolution) u Posix-like File I/O n File Access n Open, read, write n Not real posix (like rfio) Grid Storage Element Interfaces SRM interface storage allocation quota management rfiodcapchirp Posix io Castor MSS dCacheNeST Disk /GPFS POSIX API File I/O Storage Management File access SToRM INFN-Grid

43. – n° 43 General User Interface : GENIUS u Based on WEB portal architecture u Support for generic applications u Basic Requirement: Grid transparent access u It must be accessed from everywhere and by “everything” (desktop, laptop, PDA, WAP phone). u It must be redundantly “secure” at all levels: 1) secure for web transactions, 2) secure for user credentials, 3) secure for user authentication, 4) secure at VO level. u All available grid services must be incorporated in a logic way, just “one mouse click away”. u Its layout must be easily understandable and user friendly. u Evolution in EGEE/NA4

44. – n° 44 INFN-Grid, TB : Attivita’ – 2004 coord. A.Ghiselli, L.Luminari Progetti u INFN-Grid u DataGrid (chiuso) u DataTAG (chiuso) u Grid.IT (2005) u EGEE (2006) u Coregrid (2006) Attivita’ Coordinatori u WMS, F.Prelz, M.Sgaravatto u GridICE, G.Tortone, S.Fantinel u VOMS, R.Cecchini, V.Ciaschini u GLUE schema S.Andreozzi u Grid-Policy V.Ciaschini,GL.Rubini u Data Management n SToRM R.Zappi, L.Magnoni n Consistenza delle repliche F.Donno n Grid-database E.Ambrosi u Advance Reservation T.Ferrari, E.Ronchieri u Network Services T.Ferrari u Genius (Portali) R.Barbera u Release E.Ferro u Grid di produzione: ROC-CIC C.Vistoli, L.Gaido

45. – n° 45 INFN-GRID Technical Board Grid di produzione R&D Management Operations DM AA InfoMod Mon WMS portali OGSA Progetti EGEEFirb-Grid GridCC 28pf 43puf (20fte) 20C 9puf 18C 5fte/8puf 8C 6fte/16puf INFN-GRID (LCG) EGEE Appl. & Dissem. 24p 6p 7p6p 2p5p 56fte/116puf 8C usr 43puf Attivita’, Progetti manpower funded e unfunded CoreGrid 4 dottorandi Pbox 4p

46. – n° 46 Applications and WMS-API users Collective services WMS(Grid scheduler), replica consistency Resource or Core services GRAM Storage allocation, Advance reservation, network service Grid Resources and Local Services layer Compute Element, Storage Element, Network, Local Authorization Service, Local policy Service, Local Scheduler General Services CA services VOMS G-Policy and G-Accounting Monitoring Users: WMS-UI, Genius Network Layer Global Grid services view Grid management Operation Support Grid use planning Release&doc

47. – n° 47 Grid use model u ipotesi n Un esperimento e’ interessato a risorse distribuite in diversi domini di grid (risorse proprie, risorse condivise nazionali, risorse condivise internazionali). n I domini di grid hanno proprie organizzazioni di management, operation e support che devono collaborare (INFN-Grid, GridPP, NorduGrid, Grid3, ….). EGEE e’ una interconnessione di Grid (non una grid sola) che oltre alla release software cerca di definire come mettere insieme le diverse strutture organizzative. Cio’ funzionera’ se ogni struttura nazionale mantiene una sua autonomia. n L’esperimento e’ interessato a risorse hardware con particolari requisiti e concorda le condizioni di utilizzo (policy). u approccio n Grid virtuale dell’esperimento n Accordo e Applicazione delle Politiche di uso delle risorse per ogni dominio di Grid n Supporto distribuito e coordinato

48. – n° 48 EGEE INFN-Grid US-Grid same middleware ex: LCG-2 VO-User VO-User (UI) VO-Virtual Grid on top of Multi-Grids same core services RB/II VOMS VO-monitoring VO-RLS RB/II Coordinated VO-support Risorse private

49. – n° 49 Grid use model: esempio di Atlas Data Challege in ambiente omogeneo (INFN-Grid, LCG/EGEE) u Atlas Virtual Grid resources: n INFN-Grid: T1, MI, Roma1, LNF, NA, TO, LNL….. to be defined with CC n LCG/EGEE: Nikhef, RAL, (T1 di Atlas + …..) to be defined with ……. u Atlas Virtual Grid services: to be defined with planning group (and CIC?) n VO-ldap service located at Nikhef and managed by A. De Salvo. n 3 RB: one at CERN, managed by CERN.IT + Miguel Branco + A De Salvo, 1 at CNAF and 1 in Milano, managed by INFN-Grid ROC n GridICE monitoring server located in Milano n RLS…. Managed by the CERN OP n >5 UI in the ATLAS sites (increase number for analisys phase) u Policy: alta priorita’ nell’accesso delle risorse per 1 mese (produzione per la simulazione dei dati, circa altrettanto per la ricostruzione) u Storage allocation: circa 1TB locali ai CE u User distribution: produzione ufficiale da Milano, e successivamente ad hoc production in numerose sedi delle collaborazioni Atlas.

50. – n° 50 Atlas DC u Supporto n Atlas support team: DeSalvo, Resconi, Negri n Site e grid service support di INFN-Grid come parte dell’EGEE support n Uso piu’ intensivo del Supporto per l’analisi a causa di un numero piu’ elevato di utenti. u Monitoring n LCG/EGEE GOC n GridICE specifico di Atlas s Notifica dei down s Monitoring dei job specifici di Atlas s Stato delle risorse e dei servizi

51. – n° 51 Conclusions u Grid services are ready for production Grids and already in use in LCG in EU, INFN-Grid in Italy and in Grid3 in US. u They are still evolving for more functionalities, robustness and security (EGEE).  Standardization is towards: Bringing Grid & Web Services Together ( Dr. Daniel Sabbah, Vice President of Strategy & Technology, IBM Software Group, Globus Wold in January 2004): linee guida per EGEE n Web Service Resource Framework is the new grid model: Modeling Stateful Resources with Web Services s WS-Notification n Provides a publish-subscribe messaging capability for Web Services s WS-Resource framework n A family of Web services specifications that clarify how “state” and Web services combine u Il successo dell’utilizzo della Grid da parte degli esperimenti passa anche dall’interazione con l’organizzazione della grid di produzione ‘nazionale’, per assicurarsi il massimo di sinergia per il funzionamento delle risorse e servizi, e il massimo del supporto per identificare e risolvere velocemente i problemi. Va evitata una eccessiva centralizzazione ed eccessiva nell’uso delle Grid….