Automated Tests in NICOS Nightly Control System Alexander Undrus Brookhaven National Laboratory, Upton, NY Software testing is a difficult, time-consuming process that requires technical sophistication and proper planning. This is especially true for the large-scale software projects of High Energy Physics where constant modifications and enhancements are typical. The automated nightly testing is the important component of NICOS, NIghtly COntrol System, that manages the multi-platform nightly builds based on the recent versions of software packages. It facilitates collective work in collaborative environment and provides four benefits to developers: repeatability (tests can be executed more than once), accumulation (results are stored and reflected on NICOS web pages), feedback (automatic notifications about test failures), user friendly setup (configuration parameters can be encrypted in the body of test scripts). The modular structure of NICOS allows plugging in other validation and organization tools, such as QMTest and CppUNIT. NICOS classifies tests according to their granularity level and purpose. The low level structural tests reveal compilation problems, inconsistencies in package configuration, such as circular dependencies, and simple isolated bugs. The results for these three groups of tests are published for each package of the software project. The integrated (or behavioral) tests find bugs at levels of users scenarios and NICOS generates the special web page with their results. The NICOS tool is currently used to coordinate the efforts of more than 100 developers for the ATLAS project at CERN and included in the tool library of the LHC computing project. Abstract CHEP 2004

NICOS - NIghtly COntrol System NICOS Control System [1] has a modular structure. Each module is responsible for a certain step (such as code checkout, release build, testing, error analysis) and independently described in the NICOS configuration file. This organization allows to plug in build, test, validation, and other external tools. NICOS currently manages the nightly releases of ATLAS software comprising about 1000 packages. It creates the nightly build framework that also uses: ATLAS Tag Collector [2] – web interfaced database application. Developers are able to interactively select the tags from ATLAS CVS repository for the nightly releases CMT [3] configuration management tool that defines the conventions for structuring software releases and describes the package properties, constituents, and dependencies Packages Database Packages Database Build Results Tag Collector Automatic s Release Build Code Checkout Testing Error Analysis NICOS Control System NICOS Control System CMT CVS Repository CHEP Alexander Undrus - Automated Tests in NICOS Nightly Control System - page 2

Test Levels and Tools in ATLAS Nightly Builds RELEASE SYSTEM PACKAGE COMPONENT/UNIT Verification of compilation/linking Unit Tests Integration Tests QA (Configuration, Coding Standards) NICOS Control Tool provides a framework for tests of different manners and levels of focus. The tests are automatically applied to each nightly release. The compilation and linking results are verified automatically and published for each package. Quality Assurance tests verify the descriptions of package properties (in CMT requirements files). The code checking tools are under development and will be added to the nightly tests in the future. Unit tests check individual software components (such as C++ classes) or a collection of components (such as the packages of the ATLAS software release). A helper tool for unit testing of C++ programs, CppUNIT [4], is integrated in CMT environment of ATLAS releases. For testing individual packages the special make target, make check, is provided. It allows to run the test jobs focused on the package functionalities in Athena, ATLAS control framework. Integration tests show that the major systems of the software release work well and communicate with other systems successfully. QMTest [5] tool is used for the ATLAS integration tests organization and validation. It supports regression tests that compare the output of the current test with previous (or known) values. The tests are arranged in suites. The suite usually includes the tests related to the specific system of a release. QMTest saves the test results for further inspection with the graphical or command-line interface. CHEP Alexander Undrus - Automated Tests in NICOS Nightly Control System - page 3

Delivering Test Results to Developers Precise sw release info in header Integrated tests statistics Sorting options Addresses for notifications Build, QA and Unit tests results Problems highlighted NICOS web pages show the results of QA and unit tests for each package of a release. There are two levels of problem seriousness: warning and error. The results for integrated tests and suites are published on a separate web page. References 1.The NICOS home page is 2.S. Albrand “The Tag Collector – A Tool for Atlas Code Release Management”, CHEP04 contribution 3.CMT home page is 4.CppUNIT home page is 5.QMTest home page is CHEP Alexander Undrus - Automated Tests in NICOS Nightly Control System - page 4