1. Specification-Based Unit Test Data Selection: Integrating Daikon and Jtest Tao Xie and David Notkin Computer Science & Engineering, University of Washington
Problems
Which test inputs are more “valuable” ?
Developers manually write unit tests, which are usually insufficient to achieve satisfactory quality assurance of the unit. (Test insufficiency problem)
Unit test generation tools automatically produce a large number of test inputs to exercise the unit. But how to know these automatically generated tests produce correct results? (Test oracle problem)
How do we select “valuable” test inputs from automatically generated ones, equip them with oracles, and augment manually created unit tests? (Test selection problem)
Those new test inputs that exercise at least one new feature not being exercised by any test case in the existing test suite.
Black-box features: functionalities, predicates/states in specifications, etc.
White-box features: methods, statements, branches, def-use pairs, etc.
In traditional black box testing, , either formal or informal specifications are usually needed to help identify features, but in practice programs lack of specifications most of the time.
A specification-based approach without a priori specification
Infer likely specifications from existing test executions
They reflect both the program behavior and the test history
Any new tests that violate the inferred specifications are used as the candidates for test data selections – These tests possibly exercise new program behavior
Jtest (
Daikon (
A popular commercial unit test tool that can produce a large number of test inputs for a Java class
An invariant detection tool that can infer likely specifications from program executions
Bounded Stack Example:
Manual basic test #
Manual JAX test #
8 (67%)
16 (97%)
Auto generated test # (length 1)
Auto generated test # (length 2)
Auto generated test # (length 3)
14 (63%)
96 (86%)
1745 (86%)
The number of selected tests, violating tests, and violated specifications
Precondition Guard Removal
Iteration. Type
Test call length 1
Test call length 2
Test call length 3
1. Basic Full
0/1: 2
0/3: 3
0/0: 0
1. Basic No Pre
5/7: 15
8/13: 15
1/2: 2
1. JAX Full
1/3: 3
1/1: 1
1. JAX No Pre
1/3: 6
10/24: 41
2. Basic Full
2. Basic No Pre
0/1: 1
2. JAX Full
2. JAX No Pre
Iterations reaching fixed point
This work was supported in part by the National Science Foundation under grant ITR