1. ICS Test Environment Alexander Söderqvist www.europeanspallationsource.se Dec 9, 2014

2. Content Introduction Test description Result Tools Conclusions Demonstration 2

3. Introduction Usecases Testing during development Continous integration Regression testing Acceptance testing 3

4. Introduction Goals Implement tests for existing EPICS components. Automate and schedule tests. Coordinate test execution between test stands. Monitor and follow trend of test results. Present test result. 4

5. Introduction Tests Channel access Software tests – Simulated Hardware 5 Hardware-in-the-loop – Real hardware

6. Test description Test stand Test stand – Control box – Host for test items – Includes test equipment Test item – Item under test Test equipment – Required to perform tests Separate IOCs 6

7. Example, Hardware in the loop Data-acquisition card & Function Generator 7

8. Test description Test cases 8 Test case – Executed test entity -> Pass or fail – Input – Expected output – Test procedure Parameterized test case – Data driven – Multiple instances – Generic test procedure

9. Test description Test scenario Test scenario – A set of test cases. – Requires same test equipment. – Each scenario is represented by one script, which executes all test cases. 9

10. Test description SIS8300 Example 10

11. Test description SIS8300 Example contd. 11 Independent – Software acquisition – Number of samples Timing receiver – Backplane triggers Test cases are placed in the scenario with minimum test equipment required.

12. Test description Implementation Test scenario – Create test item & equipment a)Common Setup Verify default settings for test item & equipment b)Execute test case c)Common Teardown Set default setting for test item & equipment 12 Test item/equipment – Create Start IOC Connect to PVs Set default values – Set default settings – Verify default settings

13. Result µTCA test stand SIS8300 – Independent – Analog outputs – Function generator – Timing Receiver – Function generator & Timing receiver Configuration – Input coupling (AC/DC) – Memory size – Connections to analog inputs 13 SIS8300BCM – Independent – Function generator – Timing Receiver – Function Generator & Timing receiver

14. Result µTCA discovered issues SIS8300 Attenuation  sampling frequency Software bugs: – Max samples – Analog outputs – Irreversible error state – Decimation offset – Broken read-back values 14 SIS8300BCM FPGA firmware bug affecting: – Interlock – Average window filter – Droop compensation

15. Beam current monitor Test Procedure: Average window filter Method 1, Standard deviation: 1.Sample noise with all window widths. 2.Verify that standard deviation of the acquired waveform is strictly decreasing with window width. Method 2, Input stimuli: 1.Sample DC signal with 15 ns voltage drop for window widths =[2,4,8,16,32]. 2.Verify that minimum value in waveform strictly increases with window width. 15

16. Beam current monitor Test procedure: Channel pair interlock Threshold: 1.Set threshold [0 V, 0.1 V] 2.Generate DC signal on both channels. 3.Verify interlock status. Threshold [0 V, 0.1 V] -> Status [Set, Clear] Offset: 1.Set offset [0, 1000]. 2.Generate DC signal on first channel and delayed pulse on second. 3.Verify interlock status. Offset [0, 1000] -> Status [Set, Clear] 16

17. Beam current monitor Test procedure: Droop Compensation Method 1, Absolute: 1.Set droop compensation to 24%. 2.Sample DC signal, 0.5 V. 3.Verify that last sample is 0.5 V / (1-0.24)  0.658 V Method 2, Relative 1.Set droop compensation to 24%. 2.Sample DC signal. 3.Verify that V_end == V_start/ (1-0.24) 17

18. Test methodology Comparison Absolute Pro – Realistic -> evaluates the complete system. Con – Attenuation causes specific test cases to fail. (Even though the tested feature itself works) Relative Pro – Disregards attenuation -> Verify actual feature. Con – Excludes important parts of the system. 18 Conclusion: Do both absolute and relative tests.

19. Result – Test stands Serial peripheral test stand Prosilica Camera, used with FSIA. – Independent – Timing receiver Position capture, GeoBrick motion controller. – One axis – timing receiver 19

20. Result Soft test stand Fluorescent Screen Image Acquisition Data-on-demand 20

21. Tools Python Pyepics – Channel access interface – PV abstraction – Device abstraction Nose – Extends python Unittest – Support test generators – Junit xml test report 21

22. Tools Jenkins Continuous integration system Building and testing software continuously Tasks are specified in jobs. Configurable execution of jobs Monitoring of jobs 22

23. Tools Jenkins setup 23 Central jenkins Remote access to teststands. Each test item has a job. Each test scenario has a job. Jobs are specified in code.

24. Tools Jenkins – Generator job 24 Configured manually. Runs automatically on changes in the test environment project. 1.Fetch test environment. 2.Installs test item and scenario jobs.

25. Tools Jenkins – Trigger and scenario jobs 25 Trigger job – Test item 1.Install latest version of the test environment. 2.Install latest nightly of the test item software. 3.Trigger all scenario jobs related to test item. Scenario job 1.Executes test scenario. 2.Collects test result and logs.

26. Conclusions Implemented tests can be used as regression tests. Hardware in loop tests uncover fundamental characteristics and limitations. Several bugs discovered. Improved documenation. Sucessfully implemented using Python and Jenkins. 26

27. Future Long term stability testing. – Identify typical usecases. – Run for long period of time. Production controlbox testing – Identify typical production hardware and software setup. – Test compability and higher level functionality. 27

28. Demonstration Jenkins Wiki 28