1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations Manager peter.solagna@egi.eu European Grid Infrastructure

2. www.egi.eu EGI-InSPIRE RI-261323 Outline EGI overview HT data analysis solution User AuthN/AuthZ Data & storage services Compute services 2

3. www.egi.eu EGI-InSPIRE RI-261323 European Grid Infrastructure European –Over 35 countries Grid –Secure sharing of IT resources Infrastructure –Compute –Data –Federated operations –User Support –…. and beyond!! 3

4. www.egi.eu EGI-InSPIRE RI-261323 Key resource providers in EGI: National Grid Infrastructures 4 Metric Value (March 2013) Sites~330 Nb. of CPU cores~400k Disk (PB)~190 PB Tape (PB)~180 PB

5. www.egi.eu EGI-InSPIRE RI-261323 HT Data analysis infrastructure (1) Target: Research Communities Need: Store, analyze and produce large dataset of data Issues addressed: Users communities may have access to resources, but they are distributed and not uniformly accessible Manage big amount of data within a collaboration is time consuming and error prone

6. www.egi.eu EGI-InSPIRE RI-261323 HT Data analysis infrastructure (2) Easy access to shared computing and data services from independent resource providers in a uniform way optimizing usage Open standard and open source middleware services Data access based on Virtual Organizations (VO) Opportunistic usage of unused resources

7. www.egi.eu EGI-InSPIRE RI-261323 Access the EGI services: AuthN/AuthZ

8. www.egi.eu EGI-InSPIRE RI-261323 8 A multi-disciplinary e-infrastructure User VO Virtual Research Community Members Virtual Organisations Research communities Sites VO GridCloud

9. www.egi.eu EGI-InSPIRE RI-261323 User and host authentication in EGI is based on X.509 certificates Certificates are issued by certification authorities part of the EUGridPMA federation –Users must request their credentials to a registration authority –More info: http://www.egi.eu/how-to/get_a_certificate.htmlhttp://www.egi.eu/how-to/get_a_certificate.html –CAs make sure that the certificate contains the right information about the user All the EGI services accept certificates part of the EUGridPMA distribution Certificates can be stored in the web browser to access web tools and services User authentication

10. www.egi.eu EGI-InSPIRE RI-261323 EGI services do not (usually) manage authorization at a user level Sites authorize access to their resources to Virtual Organizations (VO) –Access policies –Resources allocation Finer authorization policies –VO Groups –VO Roles VO membership, groups and roles are managed by the Virtual Organization Membership Service (VOMS) –Privileged VO members (VO Managers) can independently manage the membership and the roles within their organization User Authorization

11. www.egi.eu EGI-InSPIRE RI-261323 To execute most of the actions on the infrastructure users must attach their credentials to the request –Proxy certificate: short term credential (24h) signed with the user certificate –Extended with the VO attributes (signed by the VOMS certificate) –Users can generate multiple proxies for different VOs or different roles within the same VO (depending on the task they want to execute) Proxy certificates

12. www.egi.eu EGI-InSPIRE RI-261323 Pros –Transparent authentication throughout all the services of the distributed infrastructure –Uniform group based authorization policies Cons –One more credential –F2F Id confirmation required –Works better with command line tools Pros and cons of x509

13. www.egi.eu EGI-InSPIRE RI-261323 Storage and data management services

14. www.egi.eu EGI-InSPIRE RI-261323 Data can be stored on different storage systems –Common interface for storage access: SRM, gridFTP, WEBDAV Data can be distributed and replicated among different locations –File replica catalog –Metadata catalog –File transfer services Data management on the grid

15. www.egi.eu EGI-InSPIRE RI-261323 SRM based Storage Services DPM Users and Applications dCache StoRM SRM File management Space allocation File transfer –gridFTP, http..

16. www.egi.eu EGI-InSPIRE RI-261323 Store of Logical File Names –User created alias to refer to a data item Keep track of the data locations and the data replicas Additional access control features File catalog (LFC) SE LFC Entry

17. www.egi.eu EGI-InSPIRE RI-261323 EGI Virtual Organisation EGI Grid use case example: data management Computing service Storage service Site X of YOUR VO Information System Query User environment publish state VO Management Service (VOMS) Upload file Download file File Catalog Register file Lookup file File content Metadata 17 Login to your VO (With X509 cert) Your files

18. www.egi.eu EGI-InSPIRE RI-261323 File Transfer Service (v3) allows to schedule data transfer between storage services deployed in different sites –Request an monitor multiple data transfers Integrated with VO authorization Command line tool Intensively used by LHC VOs, widely deployed in the infrastructure File transfer Service: FTS3

19. www.egi.eu EGI-InSPIRE RI-261323 Globus Online allows users to manage data with a user friendly web interface The tool handles user authentication on the grid services Transfers are performed using the GridFTP protocol Client tool that allows easy access to file from the laptop www.globusonline.eu –Operated by EGCF File transfer service: GlobusOnline

20. www.egi.eu EGI-InSPIRE RI-261323 Computing services

21. www.egi.eu EGI-InSPIRE RI-261323 Computing resources are usually available through grid interfaces called Computing Elements (CE) –Several implementation of CE: CREAM, ARC-CE, GRAM5, UNICORE/X CEs publish the available resources and the VO supported in the information system Users can directly submit and monitor computing tasks to specific Ces Data input and output are usually staged-in and staged-out to storage services within the Grid Computing resources

22. www.egi.eu EGI-InSPIRE RI-261323 Workload management services acts as brokers for the computing resources –EMI-WMS is the most common WMS in EGI –Users submit directly to the WMS their jobs, specifying the requirements –WMS retrieves from the information system the list of CEs compatible with the job requirements Jobs are submitted to the CEs that fit with the requirement and with lower workload –Users can monitor their jobs and retrieve the outputs directly through the WMS Workload management services

23. www.egi.eu EGI-InSPIRE RI-261323 EGI Grid use case example: batch processing Computing service Storage Service Site X of YOUR VO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 23 EGI Virtual Organisation job Your files

24. www.egi.eu EGI-InSPIRE RI-261323 Science gateways

25. www.egi.eu EGI-InSPIRE RI-261323 Science gateways …are community-specific sets of tools, applications, and data collections that are integrated together via a web portal (or a desktop application) Main drivers: –Simple access –Modularity Gateways are domain/community specific, but the enabling technologies are typically not User friendly: –Access with username/password 25

26. www.egi.eu EGI-InSPIRE RI-261323 Core services Infrastructure architecture 26 Data and compute services Communit y specific services or workflows

27. www.egi.eu EGI-InSPIRE RI-261323 Core services

28. www.egi.eu EGI-InSPIRE RI-261323 EGI core services 28 Service registry: GOCDB –Semi static list of production services –Service downtime registry Information system: BDII –Semi static and dynamic information, services, resources, supported Vos Monitoring: SAM –Automatic monitoring system that emulate user’s behavior to test services’ interfaces Operations dashboard –User friendly tool to monitor the status of the services Helpdesk –Centralized helpdesk service, EGI provides 1 st and 2 nd level support –NGIs and sites are accessible through the helpdesk

29. www.egi.eu EGI-InSPIRE RI-261323 Conclusions EGI services: HT services for data and compute tasks Uniform access to distributed heterogeneous resources Uniform authentication and authorization Questions?