1. EGEE is a project funded by the European Union under contract IST-2003-508833 JRA1 Testing Activity: Status and Plans Leanne Guy EGEE Middleware Testing Manager EGEE First Conference, Cork, April 19, 2004 www.eu-egee.org

2. EGEE First Conference, Cork, April 19, 2004 - 2 Contents Middleware testing activity  Mandate, milestones and deliverables Test team structure Scope of the middleware testing activity Current tasks, activities and progress to date  Testbed setup, Tools assessment, Test plan preparation Relationship with other activities  NA4, SA1, JRA3, ARDA, Globus

3. EGEE First Conference, Cork, April 19, 2004 - 3 Testing activity mandate Test all middleware components that form part of the integrated middleware system to ensure a production quality release that fulfills the requirements of the applications. Assess that all software requirements have been correctly and completely implemented and are traceable to system requirements, Test the integrated software for scalability, platform independence and stress resilience

4. EGEE First Conference, Cork, April 19, 2004 - 4 Milestones and deliverables – Year 1 Month Deliverables & Milestones Item M03June’04MJRA1.1Testing tools are available to testing, integration and implementation teams. M03June’04MJRA1.2Distributed testing infrastructure available – hardware allocated and software being installed M05August’04MJRA1.3Integration and testing infrastructure in place and operational. Continuous integration and testing begins. Test plan for release candidate 1 core middleware components available. M09December’04MJRA1.4First integrated release candidate 1 available. Integration test suite must be available in time for the delivery of middleware components. M10January’05MJRA1.5Integrated release 1 candidate enters final testing and validation period. M12March’05DJRA1.3First tested major release of EGEE middleware delivered to SA1 including test suites and reports

5. EGEE First Conference, Cork, April 19, 2004 - 5 Milestones and deliverables – year II Month Deliverables & Milestones Item M18September’05MJRA1.6Test plan for core grid components and overall integration for release 2 candidate available M20November’05MJRA1.8Release candidate 2 enters testing and validation phase M21December’05DJRA1.6Release 2 delivered to SA1 including test suites and test reports.

6. EGEE First Conference, Cork, April 19, 2004 - 6 Test team structure (I) Testing activity must be distributed across at least 3 sites  Minimum to test all basic grid functionality  Ensure that assumptions valid at only one site have not been made Requirements on sites  1 sysadmin dedicated at least 50% to the activity  Deploy candidate releases (~weekly)  Diagnose and report installation problems – provide fixes  Sysadmin task at CERN shared with Integration team  At least 15 machines dedicated to the testing and validation activity  Support for the primary platform – RHES 3.0  If support for secondary platform not possible we will need to look to other sites to participate

7. EGEE First Conference, Cork, April 19, 2004 - 7 Test team structure (II) The 3 initial testing sites are CERN, NIKHEF and RAL  More sites can join the testing activity at a later stage !  Must fulfil site requirements CERN:  Main testing site, responsible for coordination and management  Maite Barroso Lopez: Integration and testing manager  Leanne Guy: Test team manager  Diana Bosio, David Collados, Mario Reale : Testers NIKHEF  Kors Bos, David Groep, Davide Salomi : sys admin RAL  Steve Traylen: sys admin

8. EGEE First Conference, Cork, April 19, 2004 - 8 Scope of the testing activity (I) Unit testing  Implementation clusters responsible for providing unit tests  Code coverage analysis tool to assess extent of unit testing  Integrate into the build system  Components failing unit tests or not sufficiently tested will not be integrated  Metrics to be defined  Unit tests will be validated on testing testbed  Include human code reviews API testing  Ensure that all APIs conform to the service specifications  Defined by whatever the architecture team (or equivalent) is !  Automate and integrate into the build system  Reject components that deviate from the reference definition

9. EGEE First Conference, Cork, April 19, 2004 - 9 Scope of the testing activity (II) Functional testing  Fully integrated system validated against the functional requirements as specified in user requirements documents.  Verify that all the required functionality has been correctly and completely implemented  Data management, Job submission, Information publication, etc  Error recovery and reporting  Driven by application uses cases  Each application use case will be directly associated with a testsuite  Overlap with NA4 application testing team  Core of the test team’s work !

10. EGEE First Conference, Cork, April 19, 2004 - 10 Scope of the testing activity (III) Regression testing  Essential to ensure that bugs fixed in one release do not reappear in a later release  Automate as much as is possible and include in the build Security testing  Secure access to data especially biomedical requirements!  Will need help from JRA3 to design security tests ! Platform Independence testing  Need to demonstrate the portability of the middleware  Fully test and validate middleware on the main platform  Demonstrate portability only on the secondary platform.

11. EGEE First Conference, Cork, April 19, 2004 - 11 Scope of the testing activity (IV) Documentation testing  Initial input provided to integration team by implementation clusters  Integration team will consolidate user documentation from developers  Release and installation documentation will be produced by integration team  Test team will verify all official documentation Installation and configuration testing  Based on the installation guides provided by integration team  Tool independent installation and configuration of the full system will be tested by the test team.

12. EGEE First Conference, Cork, April 19, 2004 - 12 Scope of the testing activity (V) Scalability, Stress resilience, Performance testing  Will begin later in the year on a candidate release that is deemed to be sufficiently stable,  Testing on a given release in parallel with the continuous integration and testing process  Overlap with NA4 testing ? External software testing ?  When we find bugs do we simply report them to the vendor or do more tests  We do not have the resources to extensively test external components We will make use of existing testsuites from EDG and LCG

13. EGEE First Conference, Cork, April 19, 2004 - 13 Current status of the testbed 5 Machines at CERN, 2 at NIKHEF, 2 at RAL available Core middleware components of the first middleware prototype for ARDA being installed on the testbed at the moment:  Castor SRM, GFAL, Alien, R-GMA, LRC, VOMS, MyProxy, LCMAPS/LCAS Quattor will be used by all sites for testbed management  SA1 certification and validation team will also use Quattor  NIKHEF and RAL also have Quattor installations already  Quattor installation at CERN for JRA1 underway  Quattor objects for middleware prototype components to be written MJRA1.2: Distributed testing infrastructure available – hardware allocated and software being installed

14. EGEE First Conference, Cork, April 19, 2004 - 14 Middleware test plan Testing activities will be driven by the testplan document Test plan being developed based on user requirements documents:  Application requirements from NA4: HEPCAL I&II, AWG documents, Bio-informatics requirements documents from EDG  Deployment requirements being discussed with SA1  ARDA working document for core Grid services  Security: work with JRA3 to design and plan security testing The test plan is a living document: it will evolve to remain consistent with the evolution of the software MJRA1.3: Test plan for release 1 core components made publicly available beginning September 2004.

15. EGEE First Conference, Cork, April 19, 2004 - 15 Other current activities Assessment of tools to be used for testing middleware components (Diana Bosio)  Intend to leverage the experience of LCG and EDG,  Support for tools from SPI  Web page of the evaluation at http://cern.ch/egee-jra1- testing/tools.html Preparation of automated test procedures integrated with the build system (not started)  Automate as much as is possible all middleware testing  Establish a continuous integration and testing process Preparation of test data (not started) Plan for hardware resources needed Set up CVS repository and savannah for task and bug tracking

16. EGEE First Conference, Cork, April 19, 2004 - 16 Relationship with other activities ARDA  First prototype instance (end April’04) will be used to set up the testing infrastructure only, it will not be tested prior to release  Timelines too tight  Delays in the starting date of ½ the test team to 1 May  Implementation and testing activities need to converge soon!  HEP specific testing activities need to be coordinated with ARDA  Currently the first prototype is being installed on the testbed  Test and evaluate tools and gain experience  Develop Quattor objects  Maybe provide feedback but no commitment initially

17. EGEE First Conference, Cork, April 19, 2004 - 17 Relationship with other activities SA1 Certification and Testing  Overlap in deployment and certification testing  Will share testsuites for validation of basic functionality  Collaborate on Quattor tasks NA4 Test Group  Common goals around functional, stress, performance testing.  Meeting arranged at Cork to discuss plans to collaborate Globus  Meeting with Lisa Childers - common desire to share ideas on testing grid services  Work on a common definition of terminology for testing grid services