1. Symbolic Execution and Test-input Generation
Willem Visser
&
Corina Pasareanu and Peter Mehlitz
RIACS/Kestrel Technology/CSC
NASA Ames Research Center

2. Overview Motivation for Model Checking Programs
Introduction to Java PathFinder
Symbolic Execution
Coverage based Test-input generation
Conclusions

3. Motivation Mars Polar Lander Ariane 501
Software Errors can be very costly

4. More Recently
Software problem with Spirit

5. Model Checking OK Finite-state model or Error trace Model Checker
(F W)
Line 5: …
Line 12: …
Line 15:…
Line 21:…
Line 25:…
Line 27:… …
Line 41:…
Line 47:…
Temporal logic formula

6. The Dream OK Program or Error trace Checker Requirement
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail]; OK
Program or
Error trace
Checker
Property 1: …
Property 2: … …
Requirement

7. Overview Motivation for Model Checking Programs
Introduction to Java PathFinder
Symbolic Execution
Coverage based Test-input generation
Conclusions

8. Java PathFinder (JPF) Java Code Bytecode JAVAC JVM Model Checker
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Java Code
0: iconst_0
1: istore_2
2: goto #39
5: getstatic
8: aload_0
9: iload_2
10: aaload
Bytecode
JAVAC
JVM
Special
JVM
Model
Checker

9. Demo “oldclassic.java” Fixing oldclassic!
Simplified version of the deadlock encountered in the Remote Agent
Fixing oldclassic!
Or rather trying to fix… T1 T2
signal
notify
if (no_action)
wait();
signal();

10. Key Points Models can be infinite state Handle full Java language
Unbounded objects, threads,…
Depth-first state generation (explicit-state)
Verification requires abstraction
Handle full Java language
mostly only for closed systems
Cannot handle native code
no input/output through GUIs, files, Networks, …
must be modeled by java code instead
Allows Nondeterministic Environments
JPF traps special nondeterministic methods
Checks for user-defined assertions, deadlock and LTL properties
Incorporates a number of search strategies
DFS, BFS, A*, Best-first, etc.

11. Verify.ignoreIf(N > 10);
Example Annotations
public static void m(int N) {
int x = 0; int y = 0;
while (x < N) {
x++; y++; }
while (x != 0) {
x--; y--;
assert (y == 0);
Thread making OS calls
If (Verify.randomBool())
Kernel.yieldCPU();
else
Kernel.deleteThread();
Method returning 5 values
return Verify.random(4);
Verify.ignoreIf(N > 10);

12. Overview Motivation for Model Checking Programs
Introduction to Java PathFinder
Symbolic Execution
Coverage based Test-input generation
Conclusions

13. Concrete Execution Path (example)
int x, y;
if (x > y) {
x = x + y;
y = x – y;
x = x – y;
if (x – y > 0)
assert(false); }
x = 1, y = 0
1 >? 0
x = = 1
y = 1 – 0 = 1
x = 1 – 1 = 0
0 – 1 >? 0

14. Symbolic Execution Tree (example)
int x, y;
if (x > y) {
x = x + y;
y = x – y;
x = x – y;
if (x – y > 0)
assert(false); }
x = X, y = Y
X >? Y
[ X <= Y ] END
[ X > Y ] x = X + Y
[ X > Y ] y = X + Y – Y = X
[ X > Y ] x = X + Y – X = Y
[ X > Y ] Y - X >? 0
[ X > Y, Y – X <= 0 ] END
[ X > Y, Y – X > 0 ] END

15. Forward Symbolic Execution
technique for executing a program on symbolic input values
explore program paths
for each path, build a path condition
check satisfiability of path condition
state
symbolic values of variables, path condition, and counter
various applications
test generation
program verification
traditional use
programs with fixed number of int variables

16. Challenges in Generalizing Symbolic Execution
how to handle fields in dynamic structures?
how to handle aliasing?
how to generate tests?
satisfy criteria
satisfy precondition
are in-equivalent

17. NullPointerException
Example
NullPointerException
class Node { int elem; Node next; Node swapNode() { if (next != null) if (elem > next.elem) { Node t = next; next = t.next; t.next = this; return t; } return this; } } ?
null E0 E1
Input list
Constraint
Output list
E0 > E1
none
E0 <= E1

18. Generalized Symbolic Execution
model checker generates and explores “symbolic” execution tree
path conditions are added at each branch point
off-the-shelf decision procedures check path conditions
model checker backtracks if not satisfiable
non-determinism handles aliasing
explore different heap configurations explicitly
concurrency
lazy initialization
initializes program’s inputs on an “as-needed” basis
no a priori bound on input sizes
preconditions to initialize inputs only with valid values

19. Algorithm (aliasing)
when method execution accesses field f if (f is uninitialized) { if (f is reference field of type T) { non-deterministically initialize f to  null  a new object of class T (with uninitialized fields)  an object created during prior field initialization (alias) } if (f is numeric/string field) initialize f to a new symbolic value }

20. Algorithm (illustration)
consider executing next = t.next; E0
next E1 t
Precondition: acyclic list E0
next E1 t
null t E0
next E1 ? E0
next E1 t E0
next E1 t E0 E1
next t
null ?

21. Implementation via Instrumentation
decision
procedure
continue/
backtrack
state:
path condition (data)
heap configuration
thread scheduling
program
instrumentation
original program
instrumented program
model
checking
correctness specification
counterexample(s)/test suite
[heap+constraint+thread scheduling]

22. Overview Motivation for Model Checking Programs
Introduction to Java PathFinder
Symbolic Execution
Coverage based Test-input generation
Conclusions

23. White- & Black-Box Testing
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Specification Coverage
Oracle
Input Generator
Requirements
Specification
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Coverage of
Specification & Code
Input Generator
Oracle

24. White- & Black-Box Testing
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Specification Coverage
Oracle
Input Generator
Acyclic Linked List
Input Spec
Functional Spec
After removing the last element the list is empty
Adding to a full list is not allowed
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Coverage of
Specification & Code
Input Generator
Oracle

25. Model Checking & TIG OK or Error trace Model Checker No test-input
can achieve
desired coverage OK
Executable Specification or
Error trace
Model Checker
(F W)
Line 5: …
Line 12: …
Line 15:…
Line 21:…
Line 25:…
Line 27:… …
Line 41:…
Line 47:…
Property specifying
coverage cannot be achieved
Test-input to achieve coverage

26. Test-Input Generation (TIG) with Symbolic Execution
“… symbolic execution for testing programs is a more exploitable technique in the short term than the more general one of program verification”
James King CACM 19:7, 1976
… is it still true?
White-box versus black-box
Symbolic execution most often white-box
Simple data is straightforward
Complex data
Black-box is (reasonably) straightforward – Korat (ISSTA’02)
White-box?

27. White- & Black-Box Testing for Complex Data
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Input Specification Coverage
Oracle
Input Generator
Class Invariant
Pre-condition to every method
boolean repOk()
Input Spec
Functional Spec
no runtime errors exist
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Testing Criteria
Coverage of
Input Specification & Code
Input Generator
Oracle

28. Self-balancing Binary Search Trees
Red-Black Trees
Self-balancing Binary Search Trees
Java SortedMap Implementation
repOk(): conditions (1)-(5)
(1) The root is BLACK
(3) All paths from a node to its leaves contain the same
number of black nodes.
(2) Red nodes can only have black children
(4) Acyclic
(5) Consistent Parents

29. repOk() Fragment boolean repOk(Entry e) {
// root has no parent, root is black,…
// RedHasOnlyBlackChildren
workList = new LinkedList();
workList.add(e);
while (!workList.isEmpty()) {
Entry current=(Entry)workList.removeFirst();
Entry cl = current.left;
Entry cr = current.right;
if (current.color == RED) {
if(cl != null && cl.color == RED) return false;
if(cr != null && cr.color == RED) return false; }
if (cl != null) workList.add(cl);
if (cr != null) workList.add(cr);
// equal number of black nodes on left and right sub-tree…
return true;

30. Black-box TIG Generate inputs based on analysis of input structure
e.g. Rover plan generation, Korat
100% “coverage” of input structures up to a predetermined upper-bound
e.g. all red-black trees with 6 or less nodes
Complex data requires that only valid structures be considered
Use class invariant to reduce number of input structures to consider
a predicate characterizing all the instances of a class
boolean repOk()
Check code coverage using generated structures as input
Advantage – test code for which no source is available

31. Symbolic execution for black-box TIG
Symbolic execution of repOk()
Generate new structures only when repOk() returns true
Limit the size of the structures generated
Only correct structures will be generated
repOk() returns true after all nodes in the tree have been visited, hence they must all be concrete
symbolic (partial) structures can fail repOk()
Similar to Korat
Except we can also deal with data constraints

32. Symbolic Execution of repOk() Example
public static boolean repOk(Entry e) {
if (e == null)
return true;
if (e.color == RED)
return false; …

33. White-box TIG
Consider code coverage criterion when generating test inputs
Challenge
Treating complex data with symbolic execution
Use repOk() as a method precondition during symbolic execution of source code:
use repOk() to convert “symbolic” input structures into concrete structures that cover the code and pass repOk()
use repOk() also to eliminate “symbolic” structures during lazy initialization, thus reducing the search space

34. repOk() x 2 abstract and concrete
During Lazy Initialization
check Abstract repOK()
Symbolic Execution of Code
To concretize inputs by symbolic execution of Concrete repOk()
over symbolic structures
- as with Black-box TIG -
When coverage is achieved,
solve the symbolic constraints to create concrete inputs

35. White-box TIG: cover branches in deleteEntry(Entry p)
/* precondition: p.repOk() */
private void deleteEntry(Entry p) {
if (p.left != null && p.right != null) {
Entry s = successor(p);
swapPosition(s, p); }
Entry replacement = (p.left != null ? p.left : p.right);
if (replacement != null) {
replacement.parent = p.parent;
if (p.parent == null)
root = replacement;
else if (p == p.parent.left) {
p.parent.left = replacement;
else
p.parent.right = replacement;
p.left = p.right = p.parent = null;
if (p.color == BLACK)
fixAfterDeletion(replacement); ...

36. Symbolic Execution for white-box TIG
if (p.left != null && p.right != null) { ...
Symbolic structure
before executing
branch
Symbolic structure(s)
that cover
the branch
This structure “passes”
the abstract repOk()
Concretize
Concrete structure
that will cover the
code
The symbolic structure
is used as input to
repOk() and lazily
executed to obtain the
concrete structure

37. Conservative repOk()
Used to eliminate symbolic structures that cannot be converted to a concrete structure that satisfy repOk() and therefore do not describe valid inputs
Because of symbolic structures we use abstraction
conservative_RepOk() can return TRUE, FALSE or Don’t Know
if FALSE, ignore that structure by backtracking
if TRUE or Don’t Know, continue ...
Example: (2) Red nodes have only black children.
FALSE
TRUE
Don’t Know

38. Conservative repOk() // root has no parent, root is black,…
// RedHasOnlyBlackChildren
workList = new LinkedList();
workList.add(e);
while (!workList.isEmpty()) {
Entry current=(Entry)workList.removeFirst();
Entry cl = current.left;
Entry cr = current.right;
if (current.color == RED) {
if(current._left_is_initialized && cl != null && cl.color == RED) return false;
if(current._right_is_initialized && cr != null && cr.color == RED) return false; }
if (current._left_is_initialized && cl != null) workList.add(cl);
if (current._right_is_initialized && cr != null) workList.add(cr);
// equal number of black nodes on left and right sub-tree…
return true;

39. Cover branches in deleteEntry(Entry p)
/* precondition: p.repOk() */
private void deleteEntry(Entry p) {
if (p.left != null && p.right != null) {
Entry s = successor(p);
swapPosition(s, p); }
Entry replacement = (p.left != null ? p.left : p.right);
if (replacement != null) {
replacement.parent = p.parent;
if (p.parent == null)
root = replacement;
else if (p == p.parent.left) {
p.parent.left = replacement;
else
p.parent.right = replacement;
p.left = p.right = p.parent = null;
if (p.color == BLACK)
fixAfterDeletion(replacement);...

40. Lazy Initialization from Partial Structures
Partial structure satisfying
conservative_RepOk()
Concretization
By lazy initialization of
repOK()
Solution that satisfies repOk()
Not a solution!

41. Black Box Results N Time Structs Tests delEnt%BC fixD %BC fixIns %BC 1
Candidate Structures
Tests
delEnt%BC
fixD %BC
fixIns %BC 1 3
1(1) 5 2 18 6
3.2
3(2) 24 8 68
5.5
5(2)
103 16 72 50 88 4 15
9(4)
432 36 86 90 60
17(6)
1830 84
100
292
33(16)
7942
196
Size 7: Korat candidates vs 35804

42. White-box Time Mem Tests delEnt %BC fixD %BC fixIns %BC 92 7.3 11062
Candidate Structures
Tests
delEnt %BC
fixD %BC
fixIns %BC 92
7.3
11062
11(53) 86
100 88

43. Conclusions Other JPF features Test-input Generation
Partial-order reductions
Observations
Test-input Generation
Examples with primitive data as well as complex data
Make link with Shape Analysis
Derive conservative repOk() from concrete repOk() automatically
Symbolic Execution
Invariant generation
Combining Test-input generation and runtime monitoring
X9 testing framework for a next generation Mars Rover