INFSO-RI Enabling Grids for E-sciencE Enabling Grids for E-sciencE Storage Element Model and Proposal for Glue 1.3 Flavia Donno, Maarten Litmaath CERN, 6 November 2006

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Outline Summary of WLCG view of Storage Element Model Proposed Glue Schema for Glue v1.3 Proposal Status of higher level middleware (GFAL, lcg-utils, FTS) Discussion

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Summary of WLCG view of Storage Element Model A Storage Element (SE) is a Grid service that allows Grid users to store and manage files together with the space assigned to them. The SE Implementation is the software system used to manage the storage devices and servers. Examples of this are: CASTOR, dCache, DPM, StoRM, etc. An SE exposes Total Sizes : an Online for space on disks and Sizes size for space on tape or slow devices. An SE can have multiple Storage Areas.

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Summary of WLCG view : the Storage Component A Storage Component (SC) identifies a specific storage with certain properties:  Retention Policies: CUSTODIAL or REPLICA or OUTPUT  Access Latency: NEARLINE or ONLINE (or OFFLINE)  Access Protocols (examples: rfio, dcap, file, etc.) A multivalued retention policy and a single access latency are associated with an SC. – The concept of a Storage Component was introduced to describe the type of storage that is used to offer a certain quality of storage. For instance a Storage Component is a tape set or a pool of filesystems. Two Storage Components cannot overlap. An SC exposes Sizes and possibly quotas. – a Total Space: is the nominal capacity of the Storage Component subsystem (tape, dvd, disk, etc.) – a Reserved Space: is the size of space reserved but not yet used – a Used Space: is the size occupied by files that are not candidates for garbage collection

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Summary of WLCG view : the Storage Component

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Summary of WLCG view : the Storage Area A Storage Area (SA) is a view on a portion of the total space: – It is created by the System or VO Administrators – It can span different kinds of storage devices within a Storage Element – It exposes a single retention policy and a single access latency (which the underlying storage devices together can support) – In case of WLCG it implements a Storage Class instance. An SA may be dedicated or shared between certain Vos/groups/roles – For WLCG the default SA is typically shared For WLCG the SA implements a Storage Class instance: – It is identified by a Space Description (that can differ per VO).

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Summary of WLCG view : the Storage Area

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Model from WLCG perspective

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Model from WLCG perspective

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Model from WLCG perspective

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Proposed WLCG Glue v.1.3 Schema J. Jensen v.0.8

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Glue schema v1.2

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Glue schema v1.2

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November SE Glue schema v1.2

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Report from Glue Working Group VOSAAssociation rejected because of the many-to-many association. SA->SC association rejected because of the many-to-many association. SC and AccessType too complicated for v1.3. Implementation and Names of Protocols should stay enum. Old sizes should stay. For the proposed ones, users should use GetSpaceMetadata SRM call. OK for proposed GlueSE class changes. Extending SA to include VOSAAssociation: AccessControlBase: cmsFQAN AccessControlBase: atlasFQAN SpaceTokenDescription: atlas?atlasDisk1Tape1 SpaceTokenDescription: cms?cmsDisk1Tape1 Path: atlas?/castor/ads.rl.ac.uk/grid/data/atlas/ Path: cms?/castor/ads.rl.ac.uk/grid/data/cms/ Quality: custodial-online-neverExpire Quality: custodial-nearline-neverExpire Quality: replica-online-releaseWhenExpired Quality: replica-online-warnWhenExpired ??? VO specific ? Why ? Instead of : ExpirationMode not needed for WLCG Can we assume Permanent? Volatile is default

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November New proposal

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Proposal 11th November 2006 proposed date for reaching agreement.Proposal: Use v1.2 with addition of the proposed attributes for GlueSE. GlueForeignKey as GlueService to GlueSE association (check examples to see why it is needed). GlueServiceData Key-Value pairs: to be used for VOSAAssociation in GlueSA (GlueSAData?)

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Higher level middleware: GFAL, lcg-utils, FTS, SAM Under test at the moment an implementation providing SRM v2.2 support Plain Glue v1.2 SE schema used. Some wanted functionalities are missing Use cases: – Find SE type of host (classic or SRM ?)  GlueSEName: CERN-PROD-LHCB:disk (classic)  GlueSEName: CERN-PROD-LHCB:srm_v1  GlueSEName: CERN-PROD-LHCB:srm_v2 ???? – Find SE endpoint for host  GlueServiceURI: httpg://castorgridsc.cern.ch:8443/srm/managerv1  GlueServiceEndpoint: httpg://castorgridsc.cern.ch:8443/srm/managerv1 – Find SE port for host (for classic SE)  GlueSEPort: 2811 – Find SA root for VO on host (for direct access from a WN)  GlueSARoot: lhcb:lhcb – Find SA path for VO on host (needed to find GridFTP path for VO)  GlueSAPath: /storage/atlas

Enabling Grids for E-sciencE INFSO-RI SRM v2.2 Meeting, CERN - 6 November Higher level middleware: GFAL, lcg-utils, FTS, SAM Use cases: – Find SE access protocol for host  GlueSEAccessProtocolType: rfio  GlueSEAccessProtocolPort: 5001 – Find CE access point for host (for “file” protocol and classic SE)  GlueCESEBindCEAccesspoint: /castor/cern.ch/grid/ Use cases not addressed in v1.2 – Find a CE with a close SE that supports ATLAS_RAW – Find a CE that has a close SE with an SA accessible by a given VO FQAN that supports “replica” as a retention policy. – Find an SE close to a specific CE that supports ATLAS_ESD and the protocol gsidcap on WAN – Find the total size of all SAs with User Description ATLAS_RAW – Find an SE accessible by a specific VO/FQAN – Find out status of an SE service – Find out status of an SA