EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations Manager European Grid Infrastructure

EGI-InSPIRE RI Outline EGI overview HT data analysis solution User AuthN/AuthZ Data & storage services Compute services 2

EGI-InSPIRE RI European Grid Infrastructure European –Over 35 countries Grid –Secure sharing of IT resources Infrastructure –Compute –Data –Federated operations –User Support –…. and beyond!! 3

EGI-InSPIRE RI Key resource providers in EGI: National Grid Infrastructures 4 Metric Value (March 2013) Sites~330 Nb. of CPU cores~400k Disk (PB)~190 PB Tape (PB)~180 PB

EGI-InSPIRE RI HT Data analysis infrastructure (1) Target: Research Communities Need: Store, analyze and produce large dataset of data Issues addressed: Users communities may have access to resources, but they are distributed and not uniformly accessible Manage big amount of data within a collaboration is time consuming and error prone

EGI-InSPIRE RI HT Data analysis infrastructure (2) Easy access to shared computing and data services from independent resource providers in a uniform way optimizing usage Open standard and open source middleware services Data access based on Virtual Organizations (VO) Opportunistic usage of unused resources

EGI-InSPIRE RI Access the EGI services: AuthN/AuthZ

EGI-InSPIRE RI A multi-disciplinary e-infrastructure User VO Virtual Research Community Members Virtual Organisations Research communities Sites VO GridCloud

EGI-InSPIRE RI User and host authentication in EGI is based on X.509 certificates Certificates are issued by certification authorities part of the EUGridPMA federation –Users must request their credentials to a registration authority –More info: –CAs make sure that the certificate contains the right information about the user All the EGI services accept certificates part of the EUGridPMA distribution Certificates can be stored in the web browser to access web tools and services User authentication

EGI-InSPIRE RI EGI services do not (usually) manage authorization at a user level Sites authorize access to their resources to Virtual Organizations (VO) –Access policies –Resources allocation Finer authorization policies –VO Groups –VO Roles VO membership, groups and roles are managed by the Virtual Organization Membership Service (VOMS) –Privileged VO members (VO Managers) can independently manage the membership and the roles within their organization User Authorization

EGI-InSPIRE RI To execute most of the actions on the infrastructure users must attach their credentials to the request –Proxy certificate: short term credential (24h) signed with the user certificate –Extended with the VO attributes (signed by the VOMS certificate) –Users can generate multiple proxies for different VOs or different roles within the same VO (depending on the task they want to execute) Proxy certificates

EGI-InSPIRE RI Pros –Transparent authentication throughout all the services of the distributed infrastructure –Uniform group based authorization policies Cons –One more credential –F2F Id confirmation required –Works better with command line tools Pros and cons of x509

EGI-InSPIRE RI Storage and data management services

EGI-InSPIRE RI Data can be stored on different storage systems –Common interface for storage access: SRM, gridFTP, WEBDAV Data can be distributed and replicated among different locations –File replica catalog –Metadata catalog –File transfer services Data management on the grid

EGI-InSPIRE RI SRM based Storage Services DPM Users and Applications dCache StoRM SRM File management Space allocation File transfer –gridFTP, http..

EGI-InSPIRE RI Store of Logical File Names –User created alias to refer to a data item Keep track of the data locations and the data replicas Additional access control features File catalog (LFC) SE LFC Entry

EGI-InSPIRE RI EGI Virtual Organisation EGI Grid use case example: data management Computing service Storage service Site X of YOUR VO Information System Query User environment publish state VO Management Service (VOMS) Upload file Download file File Catalog Register file Lookup file File content Metadata 17 Login to your VO (With X509 cert) Your files

EGI-InSPIRE RI File Transfer Service (v3) allows to schedule data transfer between storage services deployed in different sites –Request an monitor multiple data transfers Integrated with VO authorization Command line tool Intensively used by LHC VOs, widely deployed in the infrastructure File transfer Service: FTS3

EGI-InSPIRE RI Globus Online allows users to manage data with a user friendly web interface The tool handles user authentication on the grid services Transfers are performed using the GridFTP protocol Client tool that allows easy access to file from the laptop –Operated by EGCF File transfer service: GlobusOnline

EGI-InSPIRE RI Computing services

EGI-InSPIRE RI Computing resources are usually available through grid interfaces called Computing Elements (CE) –Several implementation of CE: CREAM, ARC-CE, GRAM5, UNICORE/X CEs publish the available resources and the VO supported in the information system Users can directly submit and monitor computing tasks to specific Ces Data input and output are usually staged-in and staged-out to storage services within the Grid Computing resources

EGI-InSPIRE RI Workload management services acts as brokers for the computing resources –EMI-WMS is the most common WMS in EGI –Users submit directly to the WMS their jobs, specifying the requirements –WMS retrieves from the information system the list of CEs compatible with the job requirements Jobs are submitted to the CEs that fit with the requirement and with lower workload –Users can monitor their jobs and retrieve the outputs directly through the WMS Workload management services

EGI-InSPIRE RI EGI Grid use case example: batch processing Computing service Storage Service Site X of YOUR VO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 23 EGI Virtual Organisation job Your files

EGI-InSPIRE RI Science gateways

EGI-InSPIRE RI Science gateways …are community-specific sets of tools, applications, and data collections that are integrated together via a web portal (or a desktop application) Main drivers: –Simple access –Modularity Gateways are domain/community specific, but the enabling technologies are typically not User friendly: –Access with username/password 25

EGI-InSPIRE RI Core services Infrastructure architecture 26 Data and compute services Communit y specific services or workflows

EGI-InSPIRE RI Core services

EGI-InSPIRE RI EGI core services 28 Service registry: GOCDB –Semi static list of production services –Service downtime registry Information system: BDII –Semi static and dynamic information, services, resources, supported Vos Monitoring: SAM –Automatic monitoring system that emulate user’s behavior to test services’ interfaces Operations dashboard –User friendly tool to monitor the status of the services Helpdesk –Centralized helpdesk service, EGI provides 1 st and 2 nd level support –NGIs and sites are accessible through the helpdesk

EGI-InSPIRE RI Conclusions EGI services: HT services for data and compute tasks Uniform access to distributed heterogeneous resources Uniform authentication and authorization Questions?