T-StoRM: a StoRM testing framework Elisabetta Roncheri, INFN CNAF as member of the StoRM team Munich, 28 March 2012

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Overview Problem Statement T-StoRM solution Tests details T-StoRM usage Future work Conclusions 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany PROBLEM STATEMENT 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany What we considered StoRM is an implementation of the SRM interface It is characterized by Being usable on any posix file systems, such as GPFS and Lustre; Supporting several transfer protocols, such as gsiftp, file, https and http; Supporting access to tape by using GEMSS; Publishing information by using the GLUE standard; Supporting VOMS and GSI for authentication and authorization. Its deployment can be both easy and extremely complex according to site requirements. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

Stand Alone StoRM deployment FrontEnd (FE) Exposes the Web service interface Manages user authentication Sends the request to the BackEnd DataBase (DB) Stores SRM request and status Stores file and space information BackEnd (BE) Binds with the underlying file systems Enforces authorization policy on files Manages SRM file and space metadata 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

Common StoRM deployment 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

Complex StoRM deployment 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany What we looked for StoRM is a multi-service software subject to intense testing, validation and verification activities. It needed something to Optimize the deployment of a new StoRM software version to be certified Simplify the reproduction of environments where error events raised to react as quick as possible Provide users that experience criticalities with suitable support in reasonable time Delegate validation and verification of remote sites to their administrators Reduce time and effort to support users Improve its software development life cycle 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany T-storm solution 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

What we thought: T-StoRM T-StoRM is a StoRM testing framework that orchestrates tests by using a proper deployment and a test engine fed with a pre-built configuration file. It is designed to simplify the integration of new test suites. It verifies the StoRM implementation of the SRM interface by using all the SRM clients. It tests SRM calls one by one, providing atomic test that can be arbitrarily arranged to build more complex tests. It produces a report log for all the executed tests. It will be included in EMI 2. This development is ongoing in collaboration with IGI. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany T-StoRM deployment 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Tests details 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Testing Levels Levels Descriptions Area Atomic testing verifies any SRM operation Any SRM Functional testing verifies any SRM functional specification Sanity testing verifies the correctness of the StoRM installation and configuration StoRM Specific Regression testing verifies the correctness of each bugfix in StoRM Integration testing verifies the behavirour between different SRM implementations Performance testing determines how StoRM performs in terms of responsiveness and stability under a particular workload Conformance testing determines whether the SRM implementation agrees with the SRM specification and the SRM memorandum of understanding 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Atomic Testing This level verifies any SRM operation This level contains a set of tests that are also used in functional, regression, and integration levels. SRM Operations SRM Clients ping ClientSRM srmping arcsrmping ptp ptg rm srmrm lcg-del rmdir srmrmdir mkdir srmmkdir ... 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Functional Testing This level verifies any SRM functional specification. Each test reproduces a certain use case in order to verify the SRM functional specification. Each use case can be implemented by using different SRM clients, such as dCache client, lcg-utils, and StoRM client. Some examples are: Move a file from the local node to the storage element by using the gsiftp protocol Move a file between the local node and the storage element by using the gsiftp protocol 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Sanity Testing This level verifies the correctness of the StoRM installation and configuration. It contains tests that are specific to StoRM. Each test of this level needs to be executed before those of the other levels. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Regression Testing This level verifies the correctness of each StoRM bugfix. It contains tests that are specific to StoRM. Each test verifies bugfixes for the current X.Y.Z-W and previous versions. If the StoRM version under verification is lower than X.Y.Z-W, the test should show the issue. Some tests can be only executed in a given version range. The test is executed on top of a test environment similar to the one that showed the bug. These tests cover atomic, functional, and sanity levels. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Integration Testing A new testing level has added to satisfy a request coming from EMI JRA1 This level verifies the transfer functionality between different SRM implementations. The designed and implemented tests verify the files transfer between the EMI storage elements by using the gsiftp protocols. Even if it is out of the T-StoRM original scope, it well fits in its architecture. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

Integration Testing: Use Case 1 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

Integration Testing: Use Case 2 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Performance Testing This level determines how StoRM performs in terms of responsiveness and stability under a particular workload. The tests are not specific to StoRM. Load and stress tests are currently supported. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Load and Stress Tests Load tests are executed to understand the behaviour of the system under a specific load. Metrics to be gathered are for example: Response time of the service by using an SRM client on a user interface Service resource usage such as CPU and memory. Stress tests are executed to understand the upper limits within the StoRM system, and its robusteness in terms of load. They are also useful in production deployment to determine if the system satisfies expected bust load. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Conformance Testing This level determines if the SRM implementation agrees with the SRM specification (https://sdm.lbl.gov/srm-wg/doc/SRM.v2.2.html) the SRM Memorandum of Understanding. The more significant tests will be developed. One of the examples is: Verify the srmPING behaviour: having authorizationID as Input returning as Ouput: versionInfo of the SRM specification that is implemented extraInfo of the SRM implementation such as the StoRM version 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany T-storm usage 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany The Current Usage Who: the StoRM team the IGI-RTC (Release, Testing, Certification) group Why: To certify the StoRM version To verify the correctness of a StoRM installation and configuration in the StoRM testbed To verify new functionalities and fixed bugs 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

The Test Production Procedure The production of a new T-StoRM test follows the 5 steps described below: the test is identified with the pair (id, name): id is a unique identifier of 6 chairs; it is defined and described in the Test Plan document where test id is included it is developed in the correct test level it is included in the test suite finally, it is released in T-StoRM 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany The Near Future Usage Who: Site-administrators SRM certifiers Other software certifiers Why To validate a StoRM installation and configuration To certify SRM software products To certify other software products containing tests suitable for other software To monitor SRM instances 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Future work 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Future Work Web-based Reporting To simplify the relation among the test plan, the tests and the report logs T-StoRM integration with a Continuos Integration Framework To automate tests execution To immediate tests results reporting Taking advantage from virtual technologies To automate deployment and configuration of StoRM 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany conclusions 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany

E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany Conclusions T-StoRM is making faster and simpler the StoRM certification process. It has shown to be able to cover several test levels even not considered at the design time. It is designed to be easily extendible in order to be used for the certification of other software. 02/08/2018 E. Ronchieri, EGI UF, 26-30 March 2012, Munich, Germany