1. StoRM Architecture and Daemons

2. Outline SRM standard What is StoRM Deployment Scenario
StoRM-Tutorial for supporter, IGI, Bologna, Italy

3. Storage Resources
The storage resource is the combination of the storage system and the controlling software (e.g. file system)
Different storage resources offer:
different levels of Quality of Service (QoS)
different access technologies
StoRM-Tutorial for supporter, IGI, Bologna, Italy

4. Storage Resource Manager (SRM)
The Storage Resource Manager (SRM) Interface describes the management service of a Storage Element.
SRM services according to the SRM interface hide storage characteristics and allow interoperability.
StoRM-Tutorial for supporter, IGI, Bologna, Italy

5. SRM: key concepts Lifetime of a file File pinning Storage classes
volatile with a fixed lifetime, durable or permanent
File pinning
to ensure a file is not canceled during operation
Storage classes
to identify different quality of storage resources
to enable a separate set
Space reservation
to ensure the request space is available for the whole life of the application since the beginning
StoRM-Tutorial for supporter, IGI, Bologna, Italy

6. StoRM
StoRM is a Storage Resource Manager (SRM) solution designed to leverage the advantages of cluster file systems and standard POSIX file systems in a Grid environment.
Current team:
Riccardo Zappi, PT leader
Elisabetta Ronchieri
Michele Dibenedetto
StoRM-Tutorial for supporter, IGI, Bologna, Italy

7. StoRM product factsheet
Part of EMI project
Sources and binary distributed with EMI releases
Developed by INFN
Evolving since the late 2005
Used in production in about 50 sites
including Italian TIER-1 at CNAF
Manages over 20PB di storage space
Composed by various sub-components
BE, FE, GFTP, GHTTPS, …
StoRM-Tutorial for supporter, IGI, Bologna, Italy

8. StoRM main characteristics
Relies on the aggregation functionalities provided by file systems
Designed to be independent from the different file system supported
Highly scalable and configurable, it can be used at site with different size and requirements
Allows to expose in Grid via SRM interface files stored in a standard file system
StoRM-Tutorial for supporter, IGI, Bologna, Italy

9. StoRM and Cluster FS
StoRM is designed to take advantage from high performing cluster file system, as GPFS and Lustre, but it supports also every standard POSIX FS (ext3, ext4, xfs, …) .
It allows direct access (through the protocol file:// ) to the storage resource, as well as other grid protocol as https, gsiftp and rfio.
StoRM-Tutorial for supporter, IGI, Bologna, Italy

10. StoRM role in a Grid Site
StoRM-Tutorial for supporter, IGI, Bologna, Italy

11. Multi-layer Architecture of StoRM
Front-End
exposes the web service interface
manages user authentication
Manages connection with clients
Communicates directly with Back-End
Stores asynchronous request into the data base
Retrieves request status from the data base
StoRM-Tutorial for supporter, IGI, Bologna, Italy

12. Multi-layer Architecture of StoRM
Back-End
core of StoRM
executes all the requests
manages user authorization
enforces permissions on files and directories
interacts with other grid services
interacts with file system through a driver mechanism
StoRM-Tutorial for supporter, IGI, Bologna, Italy

13. Stateless components
Files properties are stored in Extended Attributes managed by underlying file system.
The Database is used to store:
SRM request and status
Space metadata (e.g. space usage) information
FS Namespace is not stored anywhere!
Underlying file system provide it very well, so why duplicate it?
Space metadata are recreated at service starting, if needed, so:
Accidental loss of the DB simply lead to failing ongoing SRM requests.

14. The other components StoRM-globus-GridFTP
Implement the Data Storage Interface (DSI)
Seamless integration with Globus GridFTP
Calculate Adler32 checksum on the fly
Store calculated checksum into an Extended Attribute of the file itself
Transparent to the client or remote party.

15. The other components StoRM-Checksum
Calculate the checksum of files already stored on the storage
RestFul web service developed as a stand-alone service embedding Jetty container.
accept requests containing the file path and the algorithm to be used to compute the checksum
elaborates a maximum of concurrent requests depending on how many threads are configured.
The result of the elaboration is returned to the client, typically represented by the BE component.

16. The other components StoRM-GridHTTPs
provides http and https protocol as data access protocol
able to manage plain requests and requests over SSL providing using a valid x509 certificate
Implemented as a Servlet, so it works in any Servlet container
Act as a PEP for data access. PDP is currently act by the StoRM-BE.

17. Deployment scenario Deployment schema:
depending on site requirements and load expected.
Components replication and distribution:
All the components, except for Back-End, are configurable in pool.
Notes:
Front-End server is CPU intensive due to GSI auth.
GridFTP, CheckSum, GridHTTPs servers are CPU and I/O intensive.

18. Component pooling DNS Aliasing: Internal load balancer:
Front-End, GridFTP, CheckSum, GridHTTPs servers.
Internal load balancer:
Allows a more smart load balancing policies
Already in place for GridFTP
Coming with next version for GridHTTPs and Checksum.

19. Simplest deployment All in one host Suggested for small SE
One single Front-End
One single Back-End
One single GridFTP
MySQL server
Suggested for small SE
Low transfer traffic
not means with small space size..
Few requests for second

20. Simplest deployment Scaling the number of FE on the load expected
Suggested for:
Supporting burst of requests
1 FE = 400 req/min
2 FE = 700 req/min
Supporting for fault tolerance
DNS alias + Nagios monitoring

21. All the StoRM components

22. Most common deployment

23. Why scaling FE?