1. Who Guards the Guards? On the Validation of Test Case Migration
Presented by Ivan Jovanovikj
© All rights reserved

2. Reuse of Test Cases in Migration Projects
Model-Driven Software Migration OMG ADM
Old
Model
Restructuring
Migrated Model
Reverse
Engineering
Forward
Engineering
Software Migration
Old
Test Cases
Migrated
Test Cases
test
Old
System
Migrated
System
test
Test the desired functionality of the source system
Test Case Design is costly and time consuming
© All rights reserved

3. Reuse of Test Cases in Migration Projects
Restructuring
Old Test Model
Migrated Test Model
Old
Model
Restructuring
Migrated Model
Reverse
Engineering
In this presentation:
How to validate the
migrated test cases in
migration scenario?
Forward
Engineering
Reverse
Engineering
Forward
Engineering
Software Migration
Old
Test Cases
Migrated
Test Cases
test
Old
System
Migrated
System
test
© All rights reserved

4. Validation of Test Case Migration
How to validate a test case migration?
What is a valid test case migration?
Test case migration is a process of transferring test cases into new environments without changing their functionality, i.e., without changing what they test.
Behavioral Equivalence
How to ensure behavioral equivalence in test case migration ?
© All rights reserved

5. Validation of Test Case Migration
True negatives
Passed
Test Cases (negatives)
Problem: Hides potential errors Consequence: System malfunctioning/crashes
False negatives
Test Cases
Migrated
Test Cases
Test Report
Migration
Execution
True positives
Failed
Test Cases (positives)
Problem: Detects non-existing errors
Consequence: Waste of time by looking for non-existing error
False positives
Migrated system works as expected
Errors in the migrated system
© All rights reserved

6. Validation of Test Case Migration
How to ensure behavioral equivalence in test case migration?
How to avoid/detect false positives and false negatives?
Constructive approaches
Analytical Approaches
Mutation Testing
Bottom Line
<=
Cost of Test Case Migration
Cost of Validation of Test Case Migration
Cost of Developing New Test Cases +
© All rights reserved

7. Validation of Test Case Migration
System
Refactored
Refactoring
Test Cases
All Test Cases pre- and post-program refactoring should pass
Code Refactoring
Test Cases
Refactored
Program
[Mutation Coverage]
[Mutation Coverage’] =
Test Case Refactoring
Refactoring
Test Cases
System
Migrated Test Cases
Migration
Migrated System
System and Test Case Migration
Executed against
Test Cases
System
Mutation Coverage
Killed Mutants
All Mutants = -
Equivalent Mutants
mutant killed
Mutation
mutant survived
System Mutant 1
System Mutant 2
System Mutant n …
© All rights reserved

8. Mutation Analysis Scenarios
Test Cases
System
Migrated Test Cases
Migration
Migrated System
System and Test Case Migration
Mutation of Old System 1 6
Mutation of New System 2 4 5
Mutation of System Migration 3 3
Mutation of New Test Cases 4
Mutation of Old Test Cases 5 1 2
Mutation of Test Case Migration 6
Assumptions
Indications
© All rights reserved

9. Mutation Analysis Scenarios
System Mutation
Old
Test Cases
Old System
New Test Cases
New System 1
Mutation of Old System
Old System Mutant
New System Mutant
Test Case Migration
System Migration
(1.a)
(1.b)
Old
Test Cases
Old System
New Test Cases
New System 2
Mutation of New System
Old System Mutant
New System Mutant
Test Case Migration
System Migration i
Old
Test Cases
Old System
New Test Cases
New System 3
Mutation of System Migration
Old System Mutant
New System Mutant
Test Case Migration
System Migration
Test Case Mutation
Old
Test Cases
Old System
New Test Cases
New System 4
Mutation of Old Test Cases
Old Test Case Mutant
New Test Case Mutant
Test Case Migration
System Migration
Old
Test Cases
Old System
New Test Cases
New System 5
Mutation of New Test Cases
Old Test Case Mutant
New Test Case Mutant
Test Case Migration
System Migration i
Old
Test Cases
Old System
New Test Cases
New System 6
Mutation of Test Case Migration
New Test Case Mutant
Test Case Migration
System Migration
Old Test Case Mutant
© All rights reserved

10. Mutation Analysis Scenarios
Old
Test Cases
Old System
New Test Cases
New System 1
Mutation of Old System
Old System Mutant
New System Mutant
Test Case Migration
System Migration
(1.a)
(1.b)
Old
Test Cases
Old System
New Test Cases
New System 2
Mutation of New System
Old System Mutant
New System Mutant
Test Case Migration
System Migration i
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
© All rights reserved

11. Mutation Analysis Scenarios
Assumptions
Old
Test Cases
Old System
New Test Cases
New System 2
Mutation of New System
Old System Mutant
New System Mutant
Test Case Migration
System Migration i
Suitable mutation framework exists
Mutant reverse engineering is possible
Indications
if migrated system mutant is killed then
if old system mutant is killed then
Expected case
else
if old system mutant is equivalent then
No indication
else
Scenario 1a should be revisited
else //migrated system mutant NOT killed
if old system mutant is killed then
At least one migrated test case
is a false negative
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
else
if old system mutant is equivalent then
No indication
else
Scenario 1a should be revisited
© All rights reserved

12. Mutation Analysis Scenarios
Mutation of System Migration 3
Test Case Migration
Old
Test Cases
New Test Cases 3
System Migration
Old System
New System
Old System Mutant
New System Mutant
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
© All rights reserved

13. Mutation Analysis Scenarios
Old
Test Cases
Old System
New Test Cases
New System 4
Mutation of Old Test Cases
Old Test Case Mutant
New Test Case Mutant
Test Case Migration
System Migration
Old
Test Cases
Old System
New Test Cases
New System 5
Mutation of New Test Cases
Old Test Case Mutant
New Test Case Mutant
Test Case Migration
System Migration i
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
© All rights reserved

14. Mutation Analysis Scenarios
Old
Test Cases
Old System
New Test Cases
New System 5
Mutation of New Test Cases
Old Test Case Mutant
New Test Case Mutant
Test Case Migration
System Migration i
Assumptions
Suitable mutation framework exists
Mutant reverse engineering is possible
Indications
if old test case mutant fails then
Expected case
else
if old test case mutant is equivalent then
No indication
else
Bad smell for test case migration
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
© All rights reserved

15. Mutation Analysis Scenarios
Mutation of Test Case Migration 6
Old Test Case Mutant
New Test Case Mutant 6
Test Case Migration
Old
Test Cases
New Test Cases
System Migration
Old System
New System
Mutant (System or Test Case)
Reverse engineered
Mutant
Migrated Mutant
Artefacts
Test execution against system mutant
Test execution against system
Mutant migration
Reverse engineering of Mutant
Mutant generation
Activities
© All rights reserved

16. Comprehensive discussion on the application
Problem Domain
Solution Idea
Comprehensive discussion on the application
© All rights reserved

17. Thank you for your attention Software Innovation Campus
Paderborn University
Fürstenalle 11
33102 Paderborn
Ivan Jovanovikj
Tel.: (05251)
© All rights reserved