Vincenzo Spinoso vincenzo.spinoso@egi.eu EGI.eu/INFN Data Services and Solutions Data Management for Data-Intensive Analysis Vincenzo Spinoso vincenzo.spinoso@egi.eu EGI.eu/INFN Data Services and Solutions - PART I

Categorisation of data services in EGI Outline Categorisation of data services in EGI State-of-the-art in the grid data services area: status and future plans Use cases and technical details Plans and next Data Services and Solutions - PART I

Data management is performed by interoperable components Different components address different needs Storage management at site level Transfer between sites Security Catalogue, metadata Data Services and Solutions - PART I

How data are managed at site level? Storage endpoints How data are managed at site level? Data Services and Solutions - PART I

Storage endpoints A unique namespace is provided to the client Authentication and encryption guarantee confidentiality and integrity Several protocols are supported for file access and transfer Distribute data across several disk servers guarantees scalability at site level If tapes are provided, access to tape is transparent Data Services and Solutions - PART I

Storage endpoints DPM StoRM Lustre or GPFS Data Services and Solutions - PART I

What about interoperability, access, transfers? Data Services and Solutions - PART I

Access, transfers Applications and users can interact with the endpoints using different protocols SRM offers storage management disk/tape transparent management interface between different transfer protocols standard interface GridFTP offers advanced data transfer Parallel streams Fault tolerance Security (authorization, encryption) Optimization «Storage element» DPM StoRM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Data Services and Solutions - PART I

Access, transfers Applications and users can interact with the endpoints using different protocols WebDAV offers a «web-based network file system» Widely supported by many OSes Standard (IETF) NFS4.1 provides «local access» (fast, POSIX) «Storage element» DPM StoRM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Data Services and Solutions - PART I

Access, transfers DPM Abstraction layer SRM GridFTP WebDAV NFS/pNFS Data Services and Solutions - PART I

Data transfer scheduling Can transfers be scheduled? Data Services and Solutions - PART I

Data transfer scheduling schedule continuous sustained data transfer across multiple endpoints prioritize inter-VO and intra-VO file transfers Many different clients available towards several protocols (SRM, GridFTP, webdav… ) Useful in the VO management context to control data transfers Data Services and Solutions - PART I

Catalogue Where are my files? lfn:grid/20150407/store/data/run1312 Data Services and Solutions - PART I

Catalogue LFC hierarchical view of files to users, with a UNIX-like client interface Logical File Name (LFN) to Storage URL (SURL) mappings authorization on namespace EXAMPLE: lfn:grid/20150407/store/data/run1312  srm://storm-se-01.ba.infn.it:8444/srm/managerv2?SFN=//cms/store/group Data Services and Solutions - PART I

EGI «whole picture» Really complex infrastructure based on elementary «bricks» each VO chooses its «recipe» of components mature and stable integration in a unified release controls stability of the «off-line» machinery operations control stability of the «on-line» machinery Data Services and Solutions - PART I

What is next…

Dynamic Federations (DynaFeds) A set of components that can aggregate on-the-fly storage and metadata farms exposing standard protocols, supporting redirections and WAN data access: Directories are «merged» so that files in the same directory appear inside the same directory even if they come from different sites Browse and access a huge repository made of many sites without requiring a static index No “registration”, no maintenance of catalogues Redirect intelligently clients asking for replica Automatically detects and avoid sites that go offline Accommodates client-geography-based redirection choice stable demo testbed, using HTTP/DAV http://federation.desy.de

Dynamic Federations (DynaFeds) /voname/docs/file1 /voname/docs/file2 /voname/docs/file3 /voname/software /voname/pub … Aggregation/Abstraction /voname/docs/file1 /voname/docs/file2 /voname/docs/file3

Dynamic Federations (DynaFeds) Data Services and Solutions - PART I

Globus Online provides robust and easy to use file transfer capabilities Web interface Transfer management Performance monitoring Retries after failures, autorecover when possible It’s a service, hosted at www.globusonline.eu (US) But the files that the service moves among EGI sites DO NOT LEAVE Europe GridFTP «3rd party transfer» is used Files copied directly between the EGI endpoint

iRODS Provides high level abstraction layer on top of storage resources Users focus on their data, not on where they are on the data grid Provides native metadata catalogue Multiple authentication plugins (password, PAM, GSI… ) Multiple access protocols (POSIX, S3, RADOS… ) Rule-oriented approach: «policies» can be easily implemented as data management tasks Ongoing integration in the EGI infrastructure

References EGI http://www.egi.eu https://wiki.egi.eu/wiki/Main_Page dCache http://www.dcache.org/ DPM/LFC https://svnweb.cern.ch/trac/lcgdm FTS http://fts3-service.web.cern.ch/ FTS Dashboard http://dashb-fts-transfers.cern.ch/ui/ Dynamic Federations http://indico.cern.ch/event/287233/session/6/contribution/21/material/slides/ iRODS http://irods.org/ Globus Online Cookbook https://wiki.egi.eu/wiki/Globus_Online_cookbook_for_EGI_VOs