1. Regression testing
Tor Stållhane

2. Regression testing During development During maintenance
System integration
System testing and FAT
During maintenance
When building new releases of a software product

3. Challenges in regression testing
Expensive and time consuming
Select modules based on dependencies
Prioritize test scenarios based on
business value

4. What is regression testing – 1
Regression testing is testing done to check that a system update does not re-introduce errors that have been corrected earlier.
All – or almost all – regression tests aim at checking
Functionality – black box tests.
Architecture – grey box tests

5. What is regression testing – 2
Since they are supposed to test all functionality and all previously done changes, regression tests are usually large.
Thus, regression testing needs automatic
Execution – no human intervention
Checking. Leaving the checking to developers will not work.

6. Automating regression tests – 1
We face the same challenge when doing automating regression test as we face when doing automatic test checking in general:
Which parts of each output should be checked against the oracle?
This question gets more important as we need to have more version of the same test due to system variability.

7. Automating regression tests – 2
Some alternatives: check
Only result-part, e.g. numbers and generated text
Result part plus text used for explanation or as lead texts
Any of the above plus its position on the print-out or screen

8. Automating regression tests – 3
Simple but annoying – and some times expensive – problems are e.g.
Use of date in the test output
Changes in number of blanks or line shifts
Other format changes
Changes in lead texts

9. Automating regression tests – 4
A simple solution could be to use assertions. This is, however, not a good alternative since the assertion will have to be
Inside the system all the time – extra space and execution time
Only in the test version – one extra version per variation to maintain.

10. Automating regression tests – 5
The answer to the question on the previous slide is to parameterize the test results.
Instead of using a complete output file as the oracle we use:
A tool to extract the relevant info from the output file
Compare the extracted info with the info stored in the oracle

11. Automating regression tests – 6
Build tests for version x
Test data
Run tests for version x
Build results for version x
Compare
Verdict

12. Run all regression tests
Since a regression test is large, it is always a need to identify which parts of the regression tests need to be run after a change – e.g. an error correction.
Thus, traceability – which components are used to realize which functionality or requirement – is important info for two reasons:
Save execution time. Fast and cheap hardware is, however, steadily reducing this need.
Know which tests to change when we change functionality.

13. Improving the regression test
Once we have a regression test, it is important to update it each time we
Fix a bug
Add, change or remove functionality
Change platform
If we create variants of the system, we also need to create parallel variants of the regression test suite.

14. Bug fixing – 1
When somebody reports a bug, the first activity is to reproduce it – preferably with the simplest input sequence possible.
When the bug is reproduced, we need to make a new entry in the regression test. This entry will be one of the following
The original reported input sequence
The simplest possible input sequence

15. Bug fixing – 2 The original reported input sequence
Pro: is the real thing and will tell us something about how the system is used in real life.
Con: can be large, complex and difficult to understand
The simplest possible input sequence
Pro: is (usually) simple to understand
Con: is an artificial input sequence and has no info beside being able to reproduce a reported bug