1. T HE BE/CO T ESTBED AND ITS USE FOR TIMING AND SOFTWARE VALIDATION 22 June 2010 1 BE-CO-HT Jean-Claude BAU

2. OUTLINE The BE-CO Testbed Purpose Scope Architecture The LHC Central Timing extension Purpose How test are made Failure analysis Conclusions 22 June 2010 2 BE-CO-HT Jean-Claude BAU

3. T HE BE/CO T ESTBED Purpose: Integration testing Test the core components of the control system together Validate new versions of the control system before deployment In Scope: –Core control system components (deployed “everywhere”) –Automatic integration testing –Testing of the general- purpose functionality (“the mechanics”) Out of scope: –Specific equipment, e.g. beam-loss monitors, power converters,… –Interactive testing by developers –Performance testing 22 June 2010 BE-CO-HT Jean-Claude BAU 3

4. C OMPONENTS IN THE T ESTBED 22 June 2010 4 BE-CO-HT Jean-Claude BAU Bamboo test executor CMW Proxy CMW Directory Svc RBAC Authentication Config DB Config DB Java Client API CMW, RBAC FESA, Timing CMW, RBAC FESA, Timing PPC/LynxOS LHC Central Timing Extension CMW, RBAC FESA, Timing CMW, RBAC FESA, Timing i86/Linux

5. I NTEGRATION T ESTS DONE THROUGH C LIENT API S 22 June 2010 BE-CO-HT Jean-Claude BAU Bamboo test executor CMW Proxy CMW Directory Svc RBAC Authentication Config DB Config DB Java Client API Timing Simulator Timing Simulator CMW, RBAC FESA, Timing CMW, RBAC FESA, Timing CMW, RBAC FESA, Timing CMW, RBAC FESA, Timing PPC/LynxOS i86/Linux LHC Central Timing Extension 5

6. LHC C ENTRAL T IMING E XTENSION The testbed was extended to integrate the validation of the LHC Central timing Very critical system interacting with other critical ones Impossible to use the operational system to make tests Very short time slot for upgrading LHC C LHC Central Timing LHC Central Timing SPS PS PSB LEIR LIC Central Timing LIC Central Timing SMPV 22 June 2010 6 BE-CO-HT Jean-Claude BAU

7. P URPOSE OF THE LHC CENTRAL TIMING TESTBED Validation of hardware cards Only required functionalities are tested Validation of cabling Understand how the system behaves in case of a cable cut Validation of timing software Check if the behavior is consistent with the technical specification (functional tests) Understand how it behaves when the system is heavily loaded (stress tests) Check interaction with other CO components Be pro-active Try to discover problems before they could show up in the operational system Load Table Check Table Status Load Table Parameters Check Table Status Trigger Table Execution Check Table Status Unload Table Load Table Check Table Status Load Table Parameters Check Table Status Trigger Table Execution Check Table Status Unload Table FUNCTIONAL TEST For i=1 to n { Load Table Load Table Parameters Trigger Table Execution Unload Table } For i=1 to n { Load Table Load Table Parameters Trigger Table Execution Unload Table } STRESS TEST 22 June 2010 7 BE-CO-HT Jean-Claude BAU

8. T HE T ESTBED ARCHITECTURE (1) Replication (soft & hard) of the operational system (2) By applying predefined data to the inputs, its behavior becomes predictable Bamboo test executor LHC Central Timing (1) GMT Monitoring system GMT Monitoring system SMPV Simulator LHC Injector Chain Central Timing Simulator (2) 22 June 2010 8 BE-CO-HT Jean-Claude BAU Notifications

9. E XECUTION OF THE TESTS All functional and stress tests are launched every 10 minutes with a time duration of about 2 minutes. They run 24 hours, 7 days per week. Some tests are launched in parallel to increase the communication traffic and overload the LHC central timing. No need to follow up the test reports. Bamboo sends an email when a test fails. 22 June 2010 9 BE-CO-HT Jean-Claude BAU

10. F AILURE ANALYSIS (1) Three typical sources of test failures Timing software Integrated standard CO components Entities used to test the system Timing and SMPV simulators GMT monitoring system Bamboo Tests The network 22 June 2010 10 BE-CO-HT Jean-Claude BAU

11. F AILURE ANALYSIS (2) Two kind of errors Errors easily reproducible Generally discovered during functional tests Easy to fix Sporadic errors Difficult to solve Implies to add frequently new tests or adjust existing ones Very difficult to reach 100% of success during a long period of time All test failures must be analyzed to check if they are significant or not. This takes time! 22 June 2010 11 BE-CO-HT Jean-Claude BAU

12. C ONCLUSIONS Set up of a Testbed requires time: Hardware installation Software development (tests, simulators, …) Failure analysis  This time should be well planned in projects A Testbed is essential: For critical systems When the deployment time slot is very short When a deployed system cannot be used for testing It is better to spend time in the Testbed rather than use it to fix bugs when they occur We have a better image towards our customers Developers are more relaxed and confident Software quality has a price 22 June 2010 12 BE-CO-HT Jean-Claude BAU

13. Special thanks to Vito B. and Ignacio V. for their help Thank you for your attention. 22 June 2010 13 BE-CO-HT Jean-Claude BAU