Riccardo Zappi INFN-CNAF SRM Breakout session

February 28, 2012 Ingredients 1. Basic ingredients (Fabric & Conn. level) 2. (Grid) Middleware ingredients  Site level middleware (Resource level)  VO level middleware (Collective level)  Security aspects (Orthogonal level) 3. Experiment Application ingredients (Application level) 2

Fabric and Connectivity 3 January 24-25, 2011, TEG DM & SM, NIKHEF, Amsterdam

February 28, 2012 Basic Ingredients: Hardware  Hardware Solid State drive (SSD), Hard Disk, Tape Storage System, … SATA, SCSI, Fiber Channel, … NAS, DAS, SAN and TAN Hierarchical storage solutions: 2 tiers: SAN and TAN 3 or more tiers. 4

February 28, 2012  File Systems  Lustre, GPFS: commercial products. Great, but the support is not for free.  Hadoop FS: used by big (very big) company; To adapt the usage (Map/Reduce) to exploit the HDFS features …  CEPH FS: not yet in production quality. not used by no-one “Big Company”; sustainability issue? needed a direct involvement Man power issue? Basic Ingredients: File Systems 5

February 28, 2012  File Data Access protocol  Native POSIX  pNFS (NFSv4.1 protocol)  FUSE access  xrootd  RFIO  dCap  … Basic Ingredients: connectivity 6

Resource 7 January 24-25, 2011, TEG DM & SM, NIKHEF, Amsterdam

February 28, 2012 Site level: Storage Resource  Data are ultimately stored in a storage system:  from memory stick to a multi-petabytes tape silo  Different storage resources offer different levels of Quality of Service (QoS).  The Storage Resource is the combination of the storage hardware and the controlling software (eg. file system) 8

February 28, 2012 Site level: Storage Element  Storage Element is the service which allows a user or an application to store data for future retrieval.  Offer two kind of services:  Data Access/ Transfer services  Storage Resource Manager (SRM) services 9

February 28, 2012 Site level: Data Access/Transfer Services  Data Access/Transfer services http(s) standard, widely used and supported WebDav standard, widely used and supported GridFTP OGF standard, extensively used in Grid communities Xrootd non standard, used exclusively by (some) HEP communities RFIO non standard, used exclusively by (some) HEP communities 10

February 28, 2012 Uniform interface to heterogeneous Storage Resources 11

February 28, 2012 Site level: SRM service  Storage Resource Manager (SRM) service  Hide the complexity of storage resource: such as hierarchical storage system (hsm)  Manage storage space and files reserve space, volatile files, move files between tiers  Hide local file namespace SURL (Storage URL): logical URL used to reference the file TURL (Transfer URL): physical URL used to access the file 12

February 28, 2012 Storage Element in a site 13

February 28, 2012 Storage Element solutions (1/2)  Storage Element  Two kinds of SE: All-inclusive (DPM and dCache) Assemble-as-you-need (StoRM and BestMan)  Data Transfer Protocol  GridFTP (obviosly)  http(s), BitTorrent  other (GridTorrent ?) 14

February 28, 2012 Storage Element solutions (2/2)  Data Access Protocol  Standard compliant protocols pNFS (NFSv4.1), WebDav, file  Specific protocol (non standard) xrootd, rfio, dcap, …  SRM service  All-in-one solution: dCache and DPM Doesn’t support file protocol, but they’ll support pNFS  StoRM Enable native file protocol to exploit of storage system solutions based on Lustre and GPFS.  BestMan Similar to StoRM (in features). OSG distribution 15

Collective Level (VO) 16 StoRM-Tutorial for supporter, IGI, Bologna, Italy

February 28, 2012 “VO level” ingredients  File Catalogue Service  Maintain the mapping between LFNs, GUID, SURLs  Maintain System metadata data regarding the file, not about the content of file  Metadata Service  Maintain metadata about (the content of) files  Data Discovery  Enable the discovery of data (LFNs) using queries on Metadata  Data Movement Service  Create replicas of files  Data Placement Service  Create replicas request to Data Movement Service 17

File Naming and Catalog 18

February 28, 2012 “VO level”: File Catalogue (1/2)  Used to locate Replica locations  Queries are simple (CRUD):  "get replica localities“  “create-new/delete replica”  Existing solution:  LFC 19

February 28, 2012 File Transfer Service (FTS)  gLite File Transfer Service (FTS)  Manage the network and the storage at both ends  Define the concept of a CHANNEL: a link between two SEs  File movement is asynchronous : submit a job held in file transfer queue 20

File Transfer Service (FTS) 21

February 28, 2012 Data management clients gfal/LCG-Utils  Data management access libs.  Shield users from complexity  Interacts with information system, catalogue and SRM-SEs  GFAL  Posix like C API for file access  SRMv2.2 support  lcg_util (command line + C API )  Replication, catalogue interaction etc. 22

February 28, 2012 GFAL: open a file Markus Schulz 23

February 28, 2012 Conclusion StoRM-Tutorial for supporter, IGI, Bologna, Italy 24  Storage Element services interact with many other services and client.  Failures can happen at every level:  Fabric & connectivity: hw failures, File system unresponsive, etc.  Resource: AuthN & AuthZ issues, misconfiguration, firewall, etc.  Connectivity: Information System unreachable, network latency and failures, etc.  End user observing failures don’t know where the failure happen, but she knows only that something was wrong and she is willing to bet that it's a StoRM failure.