1. M Gallas CERN EP-SFT LCG-SPI: SW-Testing1 LCG-SPI: SW-Testing LCG Applications Area GridPP 7 th Collaboration Meeting Manuel.Gallas@cern.ch LCG/SPI LCG Software Process & Infrastructure

2. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 2 Outline and Context SPI SW-Testing Overview Test FrameWorks User support Test policies Test documents LCG Infrastructure SPI project http://spi.cern.ch/ LHC grid software applications (LHC experiments, projects, etc) LCG Application Area POOL PI SIMU SEAL SW-Testing

3. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 3 Overview Software testing will be an integral part of the software development process. All level of software testing should be run as part of an automatic process. GOAL Software testing Integration test System test Acceptance test Unit test Sw-testing team Integration Tests System Tests Examples LHC experiments Use in Exp. Test FrameWorks SW-testing doc SW- testing support SPI SW-Testing SW-testing polices Code Documentation Software developer CODE Work Package Test Automated testing Nightly Building System SW-Testing

4. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 4 Test FrameWorks Aim: to help developers: to produce code for testing to run tests in automatic way Our inputs: Contacts within HEP-community. What is available as free open source code. Our constrains: Avoid commercial software and licensing problems. Avoid “do it yourself solutions” Try to adopt commonly used open-source software. SW-Testing Requirements: Work with different languages (C++, Python, …) Allow two ways of testing: “test the output of the code” and “test inside the testing-code”. Easy way to integrate existing tests. Allow the creation of dependencies among tests Organize the tests by components or packages. Regression testing Easy integration with the Nightly Building System. Provide also a graphical interface.

5. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 5 Scope: Unit –testing The same “assertion style” in different languages. Provides:- The name of the test case that failed. -The name of the source file that contains the test. -The line number where the failure occurred Different platforms/compilers: (Linux/Solaris/Windows) Test FrameWorks: X-Unit family X-Unit family JUnit QtUnit CppUnit PyUnit A simple test: 1.Subclass the TestCase CppUnit or PyUnit class 2.Override the method runTest(). 3.When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds SW-Testing

6. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 6 Test FrameWorks: Oval Oval: validation and regression used in CMS Can be use for Unit-test to Validation tests. Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) It is possible to set different run environments. Can run external scripts and external binaries. Ovalfile (configuration) Executable Test Source Code Reference file Log file Oval run Oval diff Oval build SW-Testing

7. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 7 Test FrameWorks: QMTest QMTest Uses a graphical interface for creating and running tests Runs tests in parallel Organizes tests hierarchically Supports execution of a single test or many at once Records dependencies between tests Can be run in batch mode -> easy integration with the Nightly-Building systems SW-Testing

8. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 8 Test FrameWorks: Global picture Test Scripts Valgrind Aim: to help developers: to produce code for testing to run tests in automatic way OvalFile Oval L og file Ref file ? Automated tests Regression test CppUnit PyUnit X-Unit family JUnit QtUnit QMTest Uses a graphical interface for creating and running tests Runs tests in parallel Organizes tests hierarchically Supports execution of a single test or many at once Records dependencies between tests In LCG AppArea runs what is in the OvalFile and uses oval. Oval SW-Testing

9. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 9 SPI SW-Testing Overview Test FrameWorks User support Test policies Test documents SW-Testing

10. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 10 User support: Web information httphttp://spi.cern.ch/ Test FrameWorks HowTo Test doc Policies SW-Testing

11. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 11 User support: Tools supported CppUnit PyUnit Provided as free software by the SPI External Software Service Different platforms: (Linux/Solaris/Windows) Different compilers Integrated with SCRAM and ready to be used Installed in AFS/DFS Instructions to install them in a local machine How to use them ? Oval QMTest SW-Testing

12. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 12 1 User support: How-To 23 Follow sw-testing policies * * * * * SW-Testing

13. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 13 SPI SW-Testing Overview Test FrameWorks User support Test policies Test documents SW-Testing

14. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 14 Sw-Testing policies httphttp://spi.cern.ch/software_development.html For the projects For the developers For the testing responsible Sw-testing policies are needed to ensure: Software is created together with the tests. Test directory structure and naming should be homogeneous and this will imply easy life to have automatic tools. Common and well integrated tools SW-Testing

15. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 15 SPI SW-Testing Overview Test FrameWorks User support Test policies Test documents SW-Testing

16. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 16 Test Documents For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. We use a templatetemplate Test Case template collects information about the existing tests and how they cover the expected functionality of the software. SW-Testing

17. M Gallas CERN EP-SFTLCG-SPI: SW-Testing 17 Thanks to: PPARC LHC experiments LCG Feedback and interaction are always welcome!! SPI SW-Testing Overview Test FrameWorks User support Test policies Test documents SW-Testing Conclusions: Software testing is an important task and we have tools in place to do it.