Presentation is loading. Please wait.

Presentation is loading. Please wait.

EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond.

Similar presentations


Presentation on theme: "EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond."— Presentation transcript:

1 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

2 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Contents 1.Motivation 2.Update on the Testing Framework at Diamond 3.Hardware Regression Test 4.EPICS Base Regression Test

3 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Introduction Diamond is now operating for ~4500 hours per year which will increase to ~5500 hours per year over the next two years Shutdown time is being reduced, and so are becoming increasingly busy –2010 and 2011 see installation Mini Beta Optics in two long straight sections necessitate replacing pairs of SR girders All this results in less time to test new software releases Drive to improve operational reliability –The Control System has not had significant number of faults –Dont want to change this

4 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond The Grand Objective Test all 200 support modules –Test modules and EPICS DB templates Test device/drivers that interface to hardware –Hardware test environment Test the versions of EPICS base we are running –Issues of changing EPICS base and /Tool chain/OS/ version Test software against a tool chain, build and run environment –We want to ensure that operational system use the same environment

5 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Good Software Engineering Practice Four major parts to software engineering –Design: Should come first! –Code: We are good at this. –Documentation: Doxygen etc. –Testing: Often manual and hard to repeat. Testing is not an optional extra –Can be carried out by the Software Developer –Or can be carried out by the Software User –Or can be automated Tests must be properly repeated –At least before every release and preferably more often Tests are easier to repeat if they are completely automatic

6 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Test Framework A software framework that aids the creation of automatic test suites for modules. Based on PyUnit, the Python unit testing library. Test reports conform to the TAP protocol and/or JUnit XML format. Monitoring of EPICS database records for coverage reporting. Hardware and simulation targets can use same test suite.

7 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond The Run Script Searches a directory tree for modules with automatic test support. Runs tests suites it finds, logging results and output. Can run multiple tests in parallel subject to resource constraints. Level of diagnostic output can be specified. Tests run can be restricted to single module, single target, single case. dls-run-tests -h for command help. The run script is designed to operate from the support directory of the work tree.

8 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Directory Structure Changes etc tests.py test support makeDocumentation makeIocs iocs new directories

9 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Device Simulations Based on existing DLS practice. Written in Python. Currently capable of simulating any device that communicates through serial or IP connection. Support for instrumentation to allow protocol coverage reporting.

10 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Framework Class Diagram

11 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Example – The Test Suite from pkg_resources import require require('dls.autotestframework') from dls.autotestframework import * class Fw102TestSuite(TestSuite): def createTests(self): # Define the targets for this test suite Target("simulation", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot", epicsDbFiles="db/fw102Ex.db", simDevices=[SimDevice("controller1", 9016)], guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) Target("hardware", self, iocDirectory="example", iocBootCmd="bin/linux-x86/stfw102Ex.boot", epicsDbFiles="db/fw102Ex.db", guiCmds=['edm -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl']) # The tests CaseLocalIncrementSwitch(self) CaseLocalDecrementSwitch(self) CasePowerOffOn(self) CasePvIncrement(self) CasePvDecrement(self) CasePvMultipleIncrement(self) CasePvMultipleDecrement(self) CasePvSetPosition(self) CasePvTriggerMode(self) CasePvPolling(self)

12 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Example – Intermediate Class class Fw102Case(TestCase): def curDevicePos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self.command("controller1", "getpos") args = self.recvResponse("controller1", "pos", 1) if args is not None: result = int(args[0]) return result def verifyPosition(self, intended): ''' Verify that the wheel is in the intended position ''' if self.simulationDevicePresent("controller1"): self.verify(self.curDevicePos(), intended) self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended) self.verifyPv("FGZ73762:WHEEL1:POSITION", intended) self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)

13 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Example – A Test Case class CaseLocalIncrementSwitch(Fw102Case): def runTest(self): '''The local increment switch.''' if self.simulationDevicePresent("controller1"): # Check the current position of the wheel before = self.initialPosition() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self.command("controller1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self.sleep(2) self.diagnostic("Before=%d, after=%d" % (before, after), 1) self.verifyPosition(after) before = after

14 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Example - Test Report [1] diamondtest]$./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102 [1] ok 1 - The local increment switch. [1] ok 2 - The local decrement switch. [1] ok 3 - Power off and on. [1] ok 4 - The PV increment command. [1] ok 5 - The PV decrement command. [1] ok 6 - Fast increment sync correction. [1] ok 7 - Fast decrement sync correction. [1] ok 8 - Movement directly to a position. [1] ok 9 - Control of the trigger mode. [1] ok 10 - Control of the polling mechanism. [1] # ============================== [1] # Passed 10/10 tests, % okay, in s [1] # [1] # ============================== [1] # Sim device controller1 coverage report: [1] # setpos: ok [1] # settrigger: ok [1] # getpos: ok [1] # gettrigger: ok [1] # [1] # ============================== [1] # EPICS database coverage report: [1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok [1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok [1] # FGZ73762:WHEEL1:RESTART(ai): not touched [1] # FGZ73762:WHEEL1:INPOS(calc): ok [1] # FGZ73762:WHEEL1:REINIT1(fanout): ok [1] # FGZ73762:WHEEL1:POSITION(mbbo): ok [1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok [1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok [1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok [1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok [1] # FGZ73762:WHEEL1:STEPBK(calcout): ok

15 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Hudson Use Hudson framework –For scheduling and reporting of test A Continuous Integration Engine –Checkout from repository –Does some actions Build, Post-build actions, (execute test script) –Determine Status Invoked by period checks on SVN repository Status by web and s to developers Success (all actions successful) Unstable (some tests failed) Failed (compile error)

16 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

17 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Hardware Regression Test Test IOC consists of: –Prodex OMS-58 8 axis servo control card A voltage to quadrature encoder conversion card –Hytec 8001 DIO and 8515 Octal Serial –EPICS IOC test specific db Stimulation IOC: –8005 – digital I/0 for limit switches, home switches –8401 – ADC input card to monitor OMS-58 servo demand output –8402 – DAC to drive the encoder simulation –VxWorks application implemented by cut down drivers for above cards and a simulation of some realish hardware to control Test harness: –Wiring to join Test IOC to Stimulation IOC.

18 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Hardware Test Framework Test Unit (EPICS IOC) Hardware under test Switch Terminal Server Stimulation Unit (vxWorks stimulations) Linux control PC Test suite control, redirection, etc Power strip (IP) Hudson Processor Serial Digital & Analog I/O Test harness Network

19 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Regression Test Suite (1) Came out of the Codeathon at Diamond –Nick Rees, Jon Thompson and Matt Pearson Objective –To generate automated testing of the EPICS Base project code on a diverse range of architectures, both workstation and real-time OSs. –These will be run nightly at Diamond on head of trunk and results published on web site. –Provide as service to the collaboration This is work in progress. Only started it last week.

20 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Regression Test Suite (2) There are established EPICS tests, but these are run manually as part of releases Wrap the established EPICS tests in Python to make callable from test framework. Run test on –EPICS Base, C Unit tests –Report on test coverage for each EPICS Base test –IOCs based tests

21 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Types of tests (1) C unit tests based on epicUnitTest.h functions. –These are mostly used to test libCom functions at this point –Generate TAP output that is interpreted by the test harness –Propose to add new make targets to describe how the tests are run on embedded architectures (give a download binary and a command line function to run) –Make generates a test specification that is used by the test framework to run the tests

22 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Types of tests (2) IOC based system tests based on Diamond module test framework –Python framework that loads IOC under test and a stimulation system (if required) –Can get, set and monitor PVs, and/or control or monitor the IOC command line –Tests reside in a new etc/test directory in the application tree –Have a wide range of verification commands. –Many of the EPICS soft-test suite tests were converted to this framework during the Codeathon

23 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Platforms Plan is to run this for popular development platforms and target architectures. What about historic development platforms, Solaris, and latest platforms say Fedora XX or Ubuntu XX.YY and other target architectures ?

24 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Results will be published as a web page containing: –Results of make runtests on the target architectures –Results of running the soft-tests on the target architectures –Code coverage report for the Linux host. URL to be announced

25 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

26 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

27 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS Base Test Results Web Interface

28 EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond Acknowledgement This is being lead by Jon Thomson at Diamond Its the work of many of the Controls Group at Diamond


Download ppt "EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond."

Similar presentations


Ads by Google