StoRM Architecture and Daemons

Outline SRM standard What is StoRM Deployment Scenario StoRM-Tutorial for supporter, IGI, Bologna, Italy

Storage Resources The storage resource is the combination of the storage system and the controlling software (e.g. file system) Different storage resources offer: different levels of Quality of Service (QoS) different access technologies StoRM-Tutorial for supporter, IGI, Bologna, Italy

Storage Resource Manager (SRM) The Storage Resource Manager (SRM) Interface describes the management service of a Storage Element. SRM services according to the SRM interface hide storage characteristics and allow interoperability. StoRM-Tutorial for supporter, IGI, Bologna, Italy

SRM: key concepts Lifetime of a file File pinning Storage classes volatile with a fixed lifetime, durable or permanent File pinning to ensure a file is not canceled during operation Storage classes to identify different quality of storage resources to enable a separate set Space reservation to ensure the request space is available for the whole life of the application since the beginning StoRM-Tutorial for supporter, IGI, Bologna, Italy

StoRM StoRM is a Storage Resource Manager (SRM) solution designed to leverage the advantages of cluster file systems and standard POSIX file systems in a Grid environment. Current team: Riccardo Zappi, PT leader Elisabetta Ronchieri Michele Dibenedetto StoRM-Tutorial for supporter, IGI, Bologna, Italy

StoRM product factsheet Part of EMI project Sources and binary distributed with EMI releases Developed by INFN Evolving since the late 2005 Used in production in about 50 sites including Italian TIER-1 at CNAF Manages over 20PB di storage space Composed by various sub-components BE, FE, GFTP, GHTTPS, … StoRM-Tutorial for supporter, IGI, Bologna, Italy

StoRM main characteristics Relies on the aggregation functionalities provided by file systems Designed to be independent from the different file system supported Highly scalable and configurable, it can be used at site with different size and requirements Allows to expose in Grid via SRM interface files stored in a standard file system StoRM-Tutorial for supporter, IGI, Bologna, Italy

StoRM and Cluster FS StoRM is designed to take advantage from high performing cluster file system, as GPFS and Lustre, but it supports also every standard POSIX FS (ext3, ext4, xfs, …) . It allows direct access (through the protocol file:// ) to the storage resource, as well as other grid protocol as https, gsiftp and rfio. StoRM-Tutorial for supporter, IGI, Bologna, Italy

StoRM role in a Grid Site StoRM-Tutorial for supporter, IGI, Bologna, Italy

Multi-layer Architecture of StoRM Front-End exposes the web service interface manages user authentication Manages connection with clients Communicates directly with Back-End Stores asynchronous request into the data base Retrieves request status from the data base StoRM-Tutorial for supporter, IGI, Bologna, Italy

Multi-layer Architecture of StoRM Back-End core of StoRM executes all the requests manages user authorization enforces permissions on files and directories interacts with other grid services interacts with file system through a driver mechanism StoRM-Tutorial for supporter, IGI, Bologna, Italy

Stateless components Files properties are stored in Extended Attributes managed by underlying file system. The Database is used to store: SRM request and status Space metadata (e.g. space usage) information FS Namespace is not stored anywhere! Underlying file system provide it very well, so why duplicate it? Space metadata are recreated at service starting, if needed, so: Accidental loss of the DB simply lead to failing ongoing SRM requests.

The other components StoRM-globus-GridFTP Implement the Data Storage Interface (DSI) Seamless integration with Globus GridFTP Calculate Adler32 checksum on the fly Store calculated checksum into an Extended Attribute of the file itself Transparent to the client or remote party.

The other components StoRM-Checksum Calculate the checksum of files already stored on the storage RestFul web service developed as a stand-alone service embedding Jetty container. accept requests containing the file path and the algorithm to be used to compute the checksum elaborates a maximum of concurrent requests depending on how many threads are configured. The result of the elaboration is returned to the client, typically represented by the BE component.

The other components StoRM-GridHTTPs provides http and https protocol as data access protocol able to manage plain requests and requests over SSL providing using a valid x509 certificate Implemented as a Servlet, so it works in any Servlet container Act as a PEP for data access. PDP is currently act by the StoRM-BE.

Deployment scenario Deployment schema: depending on site requirements and load expected. Components replication and distribution: All the components, except for Back-End, are configurable in pool. Notes: Front-End server is CPU intensive due to GSI auth. GridFTP, CheckSum, GridHTTPs servers are CPU and I/O intensive.

Component pooling DNS Aliasing: Internal load balancer: Front-End, GridFTP, CheckSum, GridHTTPs servers. Internal load balancer: Allows a more smart load balancing policies Already in place for GridFTP Coming with next version for GridHTTPs and Checksum.

Simplest deployment All in one host Suggested for small SE One single Front-End One single Back-End One single GridFTP MySQL server Suggested for small SE Low transfer traffic not means with small space size.. Few requests for second

Simplest deployment Scaling the number of FE on the load expected Suggested for: Supporting burst of requests 1 FE = 400 req/min 2 FE = 700 req/min Supporting for fault tolerance DNS alias + Nagios monitoring

All the StoRM components

Most common deployment

Why scaling FE?