1. Tool for Automated Testing P. Sivera European Southern Observatory Garching – January 2007ESO

2. ALMA Project Garching, January 2007Tool for Automated Testing2 tat tat - Tool for Automated Testing –run multiple tests with one command –produce reproducible output –automatic setup –compare output against reference file(s)

3. ALMA Project Garching, January 2007Tool for Automated Testing3 How tat works Looks for /test/TestList Redefine LD_LIBRARY_PATH, PYTHONPATH, PATH Executes “make clean all” Source a tcl script for additional env vars Executes a prologue script Executes the tests suite and compare output with the reference Executes an epilogue script Executes “make clean”

4. ALMA Project Garching, January 2007Tool for Automated Testing4 TestList directives 1.SOURCE 2.PROLOGUE [ ] 3. (1 or more) 4.EPILOGUE [ ] The 3rd directive is the only mandatory one

5. ALMA Project Garching, January 2007Tool for Automated Testing5 TestList syntax Blank lines are allowed Comment lines start with “#” A test directive can be split on multiple lines => continuation character is “\”

6. ALMA Project Garching, January 2007Tool for Automated Testing6 Test case syntax [.. ] Ex: 1 mySimpleTest checkCompStatus 2 myComplexTest sendMsg checkCompStatus setParam makeCrash 3 myComplexTest sendMsg “@SLEEP 10” checkCompStatus “@SLEEP 10” “setParam 3” “@SLEEP 10” makeCrash

7. ALMA Project Garching, January 2007Tool for Automated Testing7 Cookbook - 1 getTemplate for MODROOT getTemplate for Makefile under test getTemplate for script/program update Makefile write/try the script/program prepare TestList

8. ALMA Project Garching, January 2007Tool for Automated Testing8 Cookbook – 2 create reference file: $ tat –g testscript Reference file acstHello.sh.ref generated Reference file generated. run the test: $ tat testscript TEST acstHello.sh PASSED. PASSED.

9. ALMA Project Garching, January 2007Tool for Automated Testing9 How to deal with the CDB Create directory: test/CDB Put your CDB files Redefine environment variable: ACS_CDB=`pwd` (in the SOURCE script)

10. ALMA Project Garching, January 2007Tool for Automated Testing10 How to run tat Run as –tat (make all, source, prologue, test suite, compare with ref, epilogue, make clean) or –tat makeEnv (make all, source, prologue) –tat or tat –tat cleanEnv (epilogue, make clean)

11. ALMA Project Garching, January 2007Tool for Automated Testing11 Some options -v to run tat in verbose mode -nc (not clean) to run “make all” and not “make clean all” -f to use another different testlist besides the TestList

12. ALMA Project Garching, January 2007Tool for Automated Testing12 Files and log files TestList.grep → grep –v –f TestList.grep TestList.sed → sed –f TestList.sed test/tatlogs/run$$ →.out, out.orig and.diff (.diff is the unix diff between.ref and.out)

13. ALMA Project Garching, January 2007Tool for Automated Testing13 Example of SOURCE SOURCE needs a tcl script under test dir Add a line in TestList.lite: SOURCE.tcl Write the file: set env(VARIABLE) WHATEVER

14. ALMA Project Garching, January 2007Tool for Automated Testing14 Example of prologue #! /bin/bash acsStart > $ACS_TMP/acsStart.log 2>&1 & sleep 3 acsStartContainer bilboContainer >$ACS_TMP2/bilbo.log 2>&1 & sleep 3 See standard prologue: $ACSROOT/bin/acsutilTATPrologue

15. ALMA Project Garching, January 2007Tool for Automated Testing15 Example of epilogue #! /bin/bash acsStopContainer bilboContainer >bilboStop.log 2>&1 sleep 3 acsStop >acsStop.log 2>&1 sleep 3 See standard epilogue: $ACSROOT/bin/acsutilTATEpilogue