1. Regression Testing

2. Evolving Software Palette CAD 2.0 Palette CAD 3.0 2.0.1 Patches
2.1 Features
3.2 Features
3.0.2 Patches
3.0.1 Patches

3. Regression Testing
Over 70% of software development costs are associated with software maintenance.
A large percentage of those costs are associated with testing (in this context, regression testing)
As systems grow, the number of tests we have for those systems grows, and the usefulness of those tests (generally) declines

4. Regression Testing P T P’ ?

5. Retest-all P T P’ T

6. Regression Test Selection
T-T’ T’ P T P’ T’

7. Test Case Prioritization
t1, t4, t3, t5, t2

8. Test Suite Augmentation
T-T’ T’ P
T’’ P’ T’

9. Test Suite Management
T-T’
T-T’
T-T’ P T’
T-T’
T’’ P’ T’

10. Test Suite Reduction T T P P’

11. Regression Test Selection
Objective:
Reduce the resources required to regression test
Needed because
Regression test suites grow with each version
Code changes result in broken, obsolete, uncontrollable, and redundant test cases

12. Inclusiveness, Precision and Safety
Inclusiveness: pct. of tests that may show a fault and were selected
Precision: pct. of tests that cannot reveal a fault and were not selected
Safety: 100% inclusiveness
All Tests
Fault-Revealing Tests

13. Regression Test Selection Techniques
Safe:
Retest All is safe (under certain conditions)
Retest Some can be safe and more efficient
Retest the wrong ones and the selection is not safe anymore!
Not safe:
Sampling based
Profile based
Requirements based

14. Program and Modified Version
Procedure Avg
S1 count = 0
S2 fread(fptr,n)
S3 while (not EOF) do
S if (n<0)
S return(error)
else
S nums[count] = n
S count++
endif
S fread(fptr,n)
endwhile
S9 avg = mean(nums,count)
S10 return(avg)
Procedure Avg’
S1’ count = 0
S2’ fread(fptr,n)
S3’ while (not EOF) do
S4’ if (n<=0)
S5a print(“input error”)
S5’ return(error)
else
S6’ nums[count] = n
endif
S8’ fread(fptr,n)
endwhile
S9’ avg = mean(nums,count)
S10’ return(avg)

15. Execution Traces Procedure Avg S1 count = 0 S2 fread(fptr,n)
S3 while (not EOF) do
S if (n<0)
S return(error)
else
S nums[count] = n
S count++
endif
S fread(fptr,n)
endwhile
S9 avg = mean(nums,count)
S10 return(avg)

16. Execution Traces enter S1 t1 S2 S3 S4 S5 S6 S7 t1 S8 S9 t1 t1 S10 exit
Procedure Avg
S1 count = 0
S2 fread(fptr,n)
S3 while (not EOF) do
S if (n<0)
S return(error)
else
S nums[count] = n
S count++
endif
S fread(fptr,n)
endwhile
S9 avg = mean(nums,count)
S10 return(avg)
enter S1 t1 S2 S3 F T S4 T F S5 S6 S7 t1 S8 S9 t1 t1
S10
exit

17. Test History Information
enter S1
t1,t2,t3 S2 S3
t2,t3 t3 t2 S4 t3 S5 S6 t3 t2 t3 S7 S8
t1,t3 S9
t1,t3
t1,t3
S10
exit

18. CFG and Modified CFG enter S1 enter’ S1’ S2 S2’ t1,t2,t3 S3 S3’ t2,t3
S5a
S6’ t3 S7
S5’ S8
S8’ S9
S9’
S10
exit
S10’
exit’

19. Example 1 enter enter’ S1 S1’ enter’ S1’ enter S1 Dangerous Edge S2
t1,t2,t3 S3
S3’ S3
S3’ F T
t2,t3
t2,t3 F T S4
S4’ T F T F t2 S5 S6
S5a
S6’ t3 S7
S5’ S8
S8’ S9 S9
S9’
S9’
S10
S10
S10’
exit
exit’
exit
S10’
exit’

20. Example 2 enter enter’ S1 S1’ enter’ S1’ enter S1 Dangerous Edge S2 S2
t1,t2,t3 S3
S3’ S3
S3’ F T
t2,t3 F T S4
S4’ S4
S4’ T F T F t2 t2 S5 S6
S6’ S6
S5a
S6’ t3 t3
Dangerous
Edge S7
S5’ S8
S8’ S9 S9
S9’
S9’
S10
S10
S10’
exit
exit’
exit
S10’
exit’

21. Algorithm Dejavu Input: P, P’, T Output: T’
1. Build CFGs G and G’ for P and P’
2. Compare(G.EntryNode,G’.EntryNode)
3. Compare(N,N’)
mark N “N’-visited”
for each pair of successors C and C’ of N and N’
on equivalently labeled edges do
if C is not marked “C’-visited”
if C and C’ are not lexically identical
T’ = T’  TestsOnEdge(N,C,T)
else
Compare(C,C’)

22. Interprocedural Methodologies
1. Compare all pairs of procedures
2. Create & walk interprocedural representation
3. Compare all pairs of procedures identified by configuration management system

23. Empirical Results

24. Test Case Prioritization
Objective
Put your best tests forward
Needed because
Testing may get cut short
Developers can use faster feedback

25. Defining Prioritization
Test scheduling
During regression testing stage
Goal: maximize a criterion/criteria
Increase rate of fault detection
Increase rate of coverage
Increase rate of fault likelihood exposure

26. Prioritization Requirements
Definition of goal (objective function)
Increase rate of fault detection
Measurement criterion
% Of faults detected over life of test suite
Prioritization technique
Randomly
Total statements coverage
Probability of exposing faults

27. Measuring Rate of Fault DetectionX
Faults B A D C E
TESTS 1 2 3 4 5 6 7 8 9 10
A-B-C-D-E
E-D-C-B-A
C-E-B-A-D

28. A Simple Technique: Total Statement Coverage Prioritization
st-total: t3, t1, t2

29. Using Feedback: Additional Statement Coverage Prioritization
st-total: t3, t1, t2
st-addtl: t3, t2, t1

30. Coarser Granularity: Total/Addt’l Function Coverage Prioritization
func-total: t3, t1, t2
func-addtl: t3, t2, t1

31. Incorporating Modification Info: Total/Addt’l Modified Function Coverage Prio.
fn-mod-total: t2, t1, t3 f9
fn-mod-addtl: t2, t1, t3
f10
f11

32. Sources of Prioritization Data
Code Coverage Data
Modification information
Test cost data
Test criticality estimates
History information

33. Case Study of Prioritization

34. Practical Implications
APFD:
Optimal = 99%
Fn-fi-fep-addtl = 98%
Fn-total = 93%
Random = 84%
Time:
Optimal = 1.3
Fn-fi-fep-addtl = 2.0 (+.7)
Fn-total = 11.9 (+10.6)
Random = 16.5 (+15.2)

35. Test Suite Reduction Objective Needed because
Reduce regression testing costs
Needed because
Test suites grow with time, becoming more expensive to execute and maintain
Adding tests creates redundancies

36. Test Suite Reduction Problem

37. Test Suite Reduction Problem

38. Results of Two Studies
WHLM: “A test set which is minimized to preserve its coverage is likely to be as effective for detecting faults at a lower execution cost.”
RHOH: “The fault-detection capabilities of test suites can be severely compromised by reduction.”

39. Issues for Regression Test Automation
Repeatability
Controlled environment
Test oracles
Test suite management
Test resources