1. EMI INFSO-RI-261611 Testbed for project continuous Integration Danilo Dongiovanni (INFN-CNAF) -SA2.6 Task Leader Jozef Cernak(UPJŠ, Kosice, Slovakia)

2. EMI INFSO-RI-261611 Testbed Overview Open Issues Conclusions AHM EMI, Prague 11/2010 EMI Integration Testbeds 2 Outline

3. EMI INFSO-RI-261611 Task partners: INFN, CERN, CESNET, JUELICH, UPJS https://twiki.cern.ch/twiki/bin/view/EMI/TSA26 TASK GOALS: – Provide testbed for integration tests, i.e. functionality tests involving the interaction among different components, generally developed by different Product Teams, deployed on one or more hardware instances. – Large scale acceptance testbed, involving resources and effort from outer world (NGIs..., who else?) 3 Testbed overview AHM EMI, Prague 11/2010 EMI Integration Testbeds

4. EMI INFSO-RI-261611 We identified typical usage scenarios to cover expected needs of PTs. TESTING SCENARIOS: https://twiki.cern.ch/twiki/bin/view/EMI/TestBed#Testbed_overview – EMI Integration testing for a minor/major release imply: (Release Candidate (RC) service X ) tested VS (Production version of other services Y,Z,... ) (Release Candidate (RC) service X ) tested VS (RC version of other services Y,Z,... ) IMPLEMENTATION (ALREADY IN PLACE): https://twiki.cern.ch/twiki/bin/view/EMI/TestBed#Testbed_Resources_Description – A set of instances deploying Production /RC Version of EMI services, published on configurable information systems – Configuration effort to make services talk to each other – Operational facilities (certificates, monitoring, GGUS support tools) 4 Testbed overview: testing scenarios and proposed solutions AHM EMI, Prague 11/2010 EMI Integration Testbeds

5. EMI INFSO-RI-261611 Use Case A: John needs to test his service X (version X1.X2.X3 installed on some instance of his PT) and services Y (Production Version), Z (Release Candidate Version). Solution: – service X is configured to see resources published in the chosen EMI Testbed central information system instance. – John can also setup a new information system merging information from both the mentioned central information system and a local information system publishing some development resources, building a custom testbed view. Use Case B: John needs to test his service X (RC Version) and services Y (Prod Version), Z (RC Version) not currently in the testbed, through a User Interface. Solution: – John requests an account on one of the User Interfaces provided in the testbed, which is configured to see resources published in the chosen EMI Testbed central information system instance. – Depending on the test performed John may need some configuration effort on services Y or Z, to ensure they can interact with X. – Moreover John needs service Z to be installed in the Testbed. John sends a support request to EMI Testbed group. For support requests : !! open a ticket on GGUS (www.ggus.org)!! describing your problem and asking for the ticket to be assigned to EMI-Testbeds Support Unitwww.ggus.org 5 Testbed overview: typical use cases AHM EMI, Prague 11/2010 EMI Integration Testbeds

6. EMI INFSO-RI-261611 https://twiki.cern.ch/twiki/bin/view/EMI/TestBed#Testbed_HOWTO HW Resources: – ARC: 28 Instances covering production version, and some RC versions – gLite: 37 Instances covering production version, and some RC versions – UNICORE: 5 Instances covering production version, no RC – dCache (DESY volunteers for SA2.6: thanks!) Monitoring: Nagios just HW up and reachable, no service probes Testers VO: testers.eu-emi.eu (Subscribe to it) Support: GGUS, with EMI-Testbed Support Unit (on best effort within 48 hours: SA2.6 just 2-3 FTE across 5 sites). Documentation: https://twiki.cern.ch/twiki/bin/view/EMI/TestBedhttps://twiki.cern.ch/twiki/bin/view/EMI/TestBed – Testbed News/Warning – Testbed Overview – Basic HOWTO – Resources Description: EMI components coverage table, inventory of instances with logbooks 6 Testbed overview: what is in place AHM EMI, Prague 11/2010 EMI Integration Testbeds

7. EMI INFSO-RI-261611 https://twiki.cern.ch/twiki/bin/view/EMI/LogBookARCNagios Service Configuration: Service Version Installed: 3.2.1 Configuration info: Access to the Nagios portal is restricted to a limited number of users. In order to see the services registered please use arcuser for login and passord. Service Updates Logbook 1 (DATE of update and ADMINISTRATOR performing it): Report Here each applied update EMI-AHM Prague 11/2010 SA2.6: EMI Integration Testibeds 7 Logbook -example

8. EMI INFSO-RI-261611 EMI-AHM Prague 11/2010 SA2.6: EMI Integration Testibeds 8 Monitoring-Nagios-server state up/down

9. EMI INFSO-RI-261611 Nagios as tool to monitor ARC provided services: visit http://arc-emi.grid.upjs.sk/ to login use login/password: arcuser/arcuserhttp://arc-emi.grid.upjs.sk/ To obtain more information about job status on ARC servers visit Gridmonitor: http://arc-emi.grid.upjs.sk/gridmonitor.html EMI-AHM Prague 11/2010 SA2.6: EMI Integration Testibeds 9 Additional methods to obtain relevant information

10. EMI INFSO-RI-261611 We expect an initial tuning period: – Sustainability of model: effort for setup and configuration (effort needed from PTs to help in service installation/configuration) – Certification was mostly centralized in partner home projects before EMI: Problems from distributed infrastructure: local access policies differ among participant sites Problems from missing coordination among PTs (service certification depending on each others bugs) => fast varying RC candidates – EMI 1.0, cross middleware integration: what consequences on testbeds? – Logs sharing: developers want to see what happen on testbed servers, but difficult to grant root privileges to ALL developers on ALL sites (local security policies): we’ll see the impact of this problem and study solutions for log sharing (AFS, https publishing?)... 10 Testbed tuning AHM EMI, Prague 11/2010 EMI Integration Testbeds

11. EMI INFSO-RI-261611 To be different from EGI Staged Rollout, we should define specific test cases: – How large is large enough? – How do we want to stress the service? – Which particular combination of services do we need to test? Proposed Implementation (Work In Progress): – specific testbed setup accordingly to a given test definition – involving users strongly motivated to test the product: motivated to the point of providing HW and collaborative effort: how to find them? – Friendly NGI close to PTs? 11 Large Scale Acceptance Testbed AHM EMI, Prague 11/2010 EMI Integration Testbeds

12. EMI INFSO-RI-261611 Open Issues: – Cross Middleware Integration: Common Authentication Framework for all Services Resource discovery/publishing across middlewares – HW Dimensioning proportional to: N° Releases * (RC+PROD)*N° Services *N° Platforms *(Some redundancy Factor) Future Issues: – Automation of testing (tests as a post build step..) => Automation of testbed setup 12 Open Issues AHM EMI, Prague 11/2010 EMI Integration Testbeds

13. EMI INFSO-RI-261611 EMI Integration testbed is in place – Covering prod versions of components, plus some RC (ask for more if you need) – Start using it: tuning period expected: accessibility, configuration, coordination with Release Process and PTs... – Open Issues: c ross middleware integration (info system, authentication) Large scale testbeds: work in progress 13 Conclusions AHM EMI, Prague 11/2010 EMI Integration Testbeds

14. EMI INFSO-RI-261611 Thank you 14 EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 14AHM EMI, Prague 11/2010 EMI Integration Testbeds