EGEE is a project funded by the European Union under contract IST Testing processes Leanne Guy Testing activity manager JRA1 All hands meeting, 17 June

JRA1 All hands meeting, June Contents The situation today - testing the prototype How we intend to work with gLite Discussion of current issues

JRA1 All hands meeting, June Testing the prototype today Almost non existent documentation for installation and configuration of services Only a small number of people understand how to configure the system, Information extraction is via lengthy private exchanges – we still don’t understand 100% how to deploy a fully functional prototype testbed Updated rpms are often not in synchronization with bug fixes and updates on the prototype testbed – so we install something that is known to be broken! Ad-hoc testing process Time consuming and frustrating

JRA1 All hands meeting, June What have we achieved ! Understand the deployment procedures for the prototype and produced almost complete installation and configuration notes for the prototype. Have managed to get the prototype installed and working  But not always ! RAL and NIKHEF installing components of the prototype Can perform some basic tests on testing testbed Lots of testing being done on the prototype testbed, many bugs submitted already Adopting the process of tracking and managing bugs using savannah  Fixed bugs in “Ready to test” state are being tested by test team on the prototype testbed

JRA1 All hands meeting, June How should we proceed? 1 month ago it was decided that the test team would start to use the prototype, try to install and configure the system and contribute to the testing of it, Has this been useful or are we just placing more demands on already overworked people ? Propose to continue only testing installation and configuration of the prototype releases on our testbed when rpms are made available and reporting bugs, In parallel we will start system testing and developing testcases on the prototype testbed and report functionality bugs based this. Comments?

JRA1 All hands meeting, June How we plan to work with gLite Continuous cycle for release installation and testing,  Cycle frequency will depend on success rate of builds, bugs found by testing, rate of bug fixes, stability of the system, etc,  Will be no more frequent than the weekly release of baseline candidates from the integration team Deploy integrated weekly baseline candidate on distributed testbed.  All baseline candidates must pass unit and interface tests integrated into the build system,  All associated system documentation and must be available,  Release notes specifying bugs fixed, modifications to functionality or APIs,  Each site will install tagged baseline locally and independently,  Automatic installation and configuration tool will be used,

JRA1 All hands meeting, June How we plan to work with gLite  Run confirmation tests: system configured correctly, services all started and responding, bug fixes  Expect installation and confirmation testing to take ~ 1 day In parallel, manually install and configure whole system  On independent machines, full clean install and upgrade tests  Test installation guides,  Test whole system installation and configuration and service startup on 1 machine Scheduled automated and manual tests  Run all system tests, Functionality, error recovery, regression, performance, Documentation – manual  Test should be integrated into a testing framework that reports results in a standard defined format.  Run tests on a number of gLite configurations

JRA1 All hands meeting, June How we plan to work with gLite  Test reports produced automatically.  Continuous bug reporting process,  Expect to run for ~2-3 days, maybe longer Exploratory testing  Design and develop new tests through using the system extensively in an uncoordinated manner  Loose cannon style  After all scheduled tests have been run  Expect a few days or until a new baseline candidate is ready for testing

JRA1 All hands meeting, June Testing cycle Implementation activities Baseline Candidate Integration team Confirmation testing Scheduled testing Exploratory testing Legend Baseline One Cycle 1 Baseline Two Cycle 2 Release Candidate to SA1 Cycle N Baseline N Cycle 3 Baseline Three Features Bug fixes Features Bug fixes Features Bug fixes Features Bug fixes Features Bug fixes Features Bug fixes Features Bug fixes

JRA1 All hands meeting, June Other testing Portability  Will have a separate testbed for running the secondary platform  Only basic functionality will be tested to demonstrate portability Stability and resilience  When a release is deemed stable enough and has passed most functionality it will be installed on a separate cluster in the testbed for a longer duration than the release and testing cycle, i.e. 3 months and extensive tests stability, stress testing performed Performance  Simple performance tests on all releases e.g register files  More extensive tests on a stable release later  Collaboration with Dave Colling at Imperial on WMS testing

JRA1 All hands meeting, June Bug tracking and reporting process “egeetest” user created in savannah, All bugs that move into the “Ready for test” stage should be reassigned automatically to egeetest user  No automatic facility Bugs then assigned to an individual tester who will write/execute testcases and assess the bug fix Passed tests will move to the “Ready for review” state, failed tests will be put back into the “Analyzed” state Tests will be added to the automated scheduled testsuites and run during the confirmation testing cycle Refer to the Configuration Management Plan for details of the software defect lifecycle

JRA1 All hands meeting, June Testcases and testsuites Test one aspect of functionality  Based on application use cases, e.g. upload file, grid login, Many testcases will be reused in many testsuites  Many to many relationship Testsuites will be built from the testcase library Granularity of testcases/testsuites to be decided based on applications requirements and use cases Now that we have experience with gLite we will start to design testcases based on application usecases and requirements – Priority for July/August Need input from PTF on priorities Will be packaged in standard way – see Robert’s talk

JRA1 All hands meeting, June Issues Deployment of the prototype on our testbed is not progressing due to lack of coherent documentation and rpms synchronized with prototype testbed updates, It is hard to take the evaluation of frameworks any further without a working system to evaluate against Interface testing cannot progress due to lack of documentation on APIs What are the APIs that we will write tests against ? Will there be any CLIs ? Testing the system functionality via the gLite shell is restrictive  Will not be able to reuse LCG and EDG testsuites  Complete rewrite of all existing tests will be lengthy

JRA1 All hands meeting, June Testplan document Testplan document is being written Will contain information presented in this talk and more First version available in a few days