1. www.egi.eu EGI-Engage www.egi.eu Data Services and Solutions Part 1: Data in the Grid Vincenzo Spinoso vincenzo.spinoso@egi.eu EGI.eu/INFN Data Services and Solutions Webinar 1

2. www.egi.eu Outline Categorisation of data services in EGI State-of-the-art in the grid data services area: status and future plans Use cases and technical details Plans and next....2

3. www.egi.eu Components Data management is performed by interoperable components Different components address different needs Storage management at site level Transfer between sites Security Catalogue, metadata....3

4. www.egi.eu Storage endpoints....4 How data are managed at site level?

5. www.egi.eu Storage endpoints....5 Distribute data across several disk servers guarantees scalability at site level If tapes are provided, access to tape is transparent A unique namespace is provided to the client Authentication and encryption guarantee confidentiality and integrity Authentication and encryption guarantee confidentiality and integrity Several protocols are supported for file access and transfer

6. www.egi.eu Storage endpoints....6 DPM Lustre or GPFS StoRM

7. www.egi.eu Access, transfers....7 What about interoperability, access, transfers?

8. www.egi.eu Access, transfers....8 DPM StoRM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Applications and users can interact with the endpoints using different protocols SRM offers storage management disk/tape transparent management interface between different transfer protocols standard interface GridFTP offers advanced data transfer Parallel streams Fault tolerance Security (authorization, encryption) Optimization «Storage element»

9. www.egi.eu Access, transfers....9 DPM StoRM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Applications and users can interact with the endpoints using different protocols WebDAV offers a «web-based network file system» Widely supported by many OSes Standard (IETF) NFS4.1 provides «local access» (fast, POSIX) «Storage element»

10. www.egi.eu Access, transfers....10 DPM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Abstraction layer SRM GridFTP WebDAV NFS/pNFS

11. www.egi.eu Data transfer scheduling....11 Can transfers be scheduled?

12. www.egi.eu Data transfer scheduling schedule continuous sustained data transfer across multiple endpoints prioritize inter-VO and intra-VO file transfers Many different clients available towards several protocols (SRM, GridFTP, webdav… ) Useful in the VO management context to control data transfers....12

13. www.egi.eu Catalogue....13 Where are my files? lfn:grid/20150407/store/data/run1312

14. www.egi.eu Catalogue LFC hierarchical view of files to users, with a UNIX-like client interface Logical File Name (LFN) to Storage URL (SURL) mappings authorization on namespace EXAMPLE: lfn:grid/20150407/store/data/run1312  srm://storm-se-01.ba.infn.it:8444/srm/managerv2?SFN=//cms/store/group....14

15. www.egi.eu EGI «whole picture»....15 Really complex infrastructure based on elementary «bricks» each VO chooses its «recipe» of components mature and stable integration in a unified release controls stability of the «off-line» machinery operations control stability of the «on-line» machinery

16. www.egi.eu What is next… Storage Management overview 16 24/09/2014

17. www.egi.eu Dynamic Federations (DynaFeds)  A set of components that can aggregate on-the-fly storage and metadata farms exposing standard protocols, supporting redirections and WAN data access:  Directories are «merged» so that files in the same directory appear inside the same directory even if they come from different sites  Browse and access a huge repository made of many sites without requiring a static index  No “registration”, no maintenance of catalogues  Redirect intelligently clients asking for replica  Automatically detects and avoid sites that go offline  Accommodates client-geography-based redirection choice  stable demo testbed, using HTTP/DAV http://federation.desy.dehttp://federation.desy.de 17

18. www.egi.eu Dynamic Federations (DynaFeds) 18 /voname/docs/file1 /voname/docs/file2 /voname/docs/file3 /voname/software /voname/pub … Aggregation/Abstraction /voname/docs/file1 /voname/docs/file2 /voname/docs/file3

19. www.egi.eu Dynamic Federations (DynaFeds)

20. www.egi.eu Globus Online provides robust and easy to use file transfer capabilities Web interface Transfer management Performance monitoring Retries after failures, autorecover when possible It’s a service, hosted at www.globusonline.eu (US)www.globusonline.eu But the files that the service moves among EGI sites DO NOT LEAVE Europe GridFTP «3rd party transfer» is used Files copied directly between the EGI endpoint.... 20

21. www.egi.eu iRODS Provides high level abstraction layer on top of storage resources Users focus on their data, not on where they are on the data grid Provides native metadata catalogue Multiple authentication plugins (password, PAM, GSI… ) Multiple access protocols (POSIX, S3, RADOS… ) Rule-oriented approach: «policies» can be easily implemented as data management tasks Ongoing integration in the EGI infrastructure Storage Management overview 21 24/09/2014

22. www.egi.eu References EGI http://www.egi.eu https://wiki.egi.eu/wiki/Main_Pagehttp://www.egi.euhttps://wiki.egi.eu/wiki/Main_Page dCache http://www.dcache.org/http://www.dcache.org/ DPM/LFC https://svnweb.cern.ch/trac/lcgdmhttps://svnweb.cern.ch/trac/lcgdm FTS http://fts3-service.web.cern.ch/http://fts3-service.web.cern.ch/ FTS Dashboard http://dashb-fts-transfers.cern.ch/ui/http://dashb-fts-transfers.cern.ch/ui/ Dynamic Federations http://indico.cern.ch/event/287233/session/6/contribution/21/material/slides/ http://indico.cern.ch/event/287233/session/6/contribution/21/material/slides/ iRODS http://irods.org/ Globus Online Cookbook https://wiki.egi.eu/wiki/Globus_Online_cookbook_for_EGI_VOshttps://wiki.egi.eu/wiki/Globus_Online_cookbook_for_EGI_VOs.... 22