1. Automatic Test Generation SymCrete
Willem Visser Stellenbosch University
Joint work with Corina Pasareanu and Neha Rungta from NASA Ames Research Center

2. How do we get there?
How did we get here?

3. How do we obtain Statement Coverage?
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
How do we obtain
Statement Coverage?

4. How do we obtain Statement Coverage?
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
How do we obtain
Statement Coverage?

5. might work if you are moderately lucky
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
Random Inputs
might work if you are moderately lucky
But there is a better way!
Where you don’t need to win the Lottery

6. Symbolic Execution void test(int x, int y) { if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
test(X,Y)
[ X > 0 ]
[ X > 0 ] hash (X)
[ 0<X<=10 & Y=X*10 ] S0
[ X>10 & … ] …
[ 0<X<=10 & Y!=X*10 ] S1
[ 3<X<=10 & 10<Y=X*10] S3
[ 0<X<=10 & Y=X*10 & ! (X>3 & Y>10) ] S4
[ 0<X<=10 & Y!=X*10 & ! (X>3 & Y>10) ] S4
[ 3<X<=10 & 10<Y!=X*10] S3
[ X > 0 ] hash (X)
[ 0<X<=10 ] ret X*10
[ X>10] ret 0

7. Symbolic Execution void test(int x, int y) { if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
test(X,Y)
[ X > 0 ]
[ X > 0 ] hash (X)
Solve
[ X>10 & … ] …
X=1,Y=10
[ 0<X<=10 & Y=X*10 ] S0
Solve
[ 0<X<=10 & Y!=X*10 ] S1
X=1,Y=0
Solve
X=4,Y=11
[ 3<X<=10 & 10<Y=X*10] S3
[ 3<X<=10 & 10<Y!=X*10] S3
Solve
X=1,Y=10
[ 0<X<=10 & Y=X*10 & ! (X>3 & Y>10) ] S4
[ 0<X<=10 & Y!=X*10 & ! (X>3 & Y>10) ] S4
[ X > 0 ] hash (X)
[ 0<X<=10 ] ret X*10
[ X>10] ret 0

8. Symbolic Execution is not
the best thing since
it has a
few serious
namely :
It is inherently white-box
Only as good as the decision procedures

9. Code is not available so no SE is possible
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x); OR
int hash(x) {
return x*x % 1023
Code is not available so no SE is possible
Assuming we only have a linear integer arithmetic DP we cannot handle the non-linearity here

10. Concolic Execution or
Directed Automated Random Testing (DART) Godefroid, Klarlund and Sen 2005
Novel combination of concrete and symbolic execution to overcome the two weaknesses of classic symbolic execution
Executes program concretely, but collects the path condition, negates constraints on the PC after a run and executes again with the newly found solutions.

11. [ X>0 & Y!=40 & X>3 & Y<= 10]
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X>0 & Y!=10 & X>3]
test(1,0)
test(4,0)
[ X > 0 ]
[ X > 0 ]
[ X > 0 & Y != 40 ]
[ X>0 & Y!=40 & X>3 & Y<= 10]
[ X > 0 & Y != 10 ]
[ X>0 & Y!=10 & X<=3]
Concolic Execution

12. Concolic Execution void test(int x, int y) { if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X>0 & Y!=40 & X>3 & Y>10]
[ X>0 & Y=40 & X>3 & Y>10]
test(4,11)
[ X > 0 ]
[ X > 0 & Y != 40 ]
[ X>0 & Y!=40 & X>3 & Y>10]
test(4,40)
[ X > 0 ]
[ X > 0 & Y = 40 ]
[ X>0 & Y=40 & X>3 & Y>10]
Concolic Execution

13. Concolic Execution void test(int x, int y) { if (x > 0) {
if (y == 40)
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X>0 & Y=40 & X>3 & Y>10]
test(4,40)
[ X > 0 ]
[ X > 0 & Y = 40 ]
[ X>0 & Y=40 & X>3 & Y>10]
Concolic Execution 13

14. Divergence! Concolic Execution
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X>0 & Y=40 & X<=3 & Y>10]
test(1,40)
[ X > 0 ]
Divergence!
Aimed to get S0;S4
But reached S1;S4
[ X > 0 & Y != 10 ]
[ X>0 & Y!=10 & X<=3 & Y>10]
ASSERT not via S0
Concolic Execution

15. Symbolic Execution with Mixed Concrete-Symbolic Solving Pasareanu, Rungta, Visser 2011
Symbolic Execution that falls back onto concrete values when it doesn’t have access to the code or the decision procedures don’t work.
SymCrete = Symbolic + Concrete vs
Concolic = Concrete + Symbolic

16. Symbolic Execution void test(int x, int y) { if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
test(X,Y)
[ X > 0 ]
[ X > 0 ] hash (X)
[ X>10 & … ] …
[ 0<X<=10 & Y=X*10 ] S0
[ 0<X<=10 & Y!=X*10 ] S1
[ 3<X<=10 & 10<Y=X*10] S3
[ 3<X<=10 & 10<Y!=X*10] S3
[ 0<X<=10 & Y=X*10 & ! (X>3 & Y>10) ] S4
[ 0<X<=10 & Y!=X*10 & ! (X>3 & Y>10) ] S4
[ X > 0 ] hash (X)
[ 0<X<=10 ] ret X*10
[ X>10] ret 0

17. Symbolic Execution void test(int x, int y) { if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
test(X,Y)
[ X > 0 ]
[ X > 0 ] hash (X)
SymCrete 3 Steps
Split PC into two parts:
Part you can solve
Part you cannot solve
Solve the easy part and evaluate the hard part with the solutions
Replace the hard part with the evaluated results and check SAT

18. SymCrete Execution void test(int x, int y) { if (x > 0) {
test(X,Y)
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X > 0 ]
[ X > 0 ] hash (X)
[ X>0 & Y=hash(X) ] S0
[ X>0 & Y!=hash(X) ] S1
easy
hard
X>0 & Y!=10 is SAT 1
X>0
Y=hash(X) 2
X=1
Y=hash(1)=10 3
X>0 & Y=10 is SAT

19. SymCrete Execution void test(int x, int y) { if (x > 0) {
test(X,Y)
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X > 0 ]
[ X > 0 ] hash (X)
[ X>0 & Y=hash(X) ] S0
[ X>3 & Y=hash(X) & Y>10 ] S3
[ 3>=X>0 & Y=hash(X)] S4
3>=X>0
Y=hash(X)
X=1
Y=hash(1)
[3>=X>0 & Y=10 is SAT
X>3 & Y>10
Y=hash(X)
X=4 & Y=11
Y=hash(4)
[X>3 & Y=40 & Y>10 is SAT

20. SymCrete Execution void test(int x, int y) { if (x > 0) {
test(X,Y)
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (x > 3 && y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X > 0 ]
[ X > 0 ] hash (X)
x=1,y=10
[ X>0 & Y=hash(X) ] S0
x=1,y=0
[ X>0 & Y!=hash(X) ] S1
[ X>3 & Y=hash(X) & Y>10 ] S3
[ X>3 & Y!=hash(X) & Y>10 ] S3
x=4,y=40
x=4,y=11
[ 3>=X>0 & Y=hash(X)] S4
[ 3>=X>0 & Y!=hash(X)] S4
x=1,y=10
x=1,y=0

21. The Risk of Unsoundness
test (int x, int y) {
if (x>=0 && x>y && y == x*x)
S0;
else
S1; }
Not Reachable
[ X>=0 & X > Y & Y = X*X ] S0
Must add constraints on the solutions from Step 2 in Step 3
X>=0 & X>Y
Y = X*X
X=0, Y=-1
Y=0*0=0
X>=0 & X>Y & Y=0 & X=0
NOT SAT
X>=0 & X>Y & Y=0
Is SAT which implies S0 is Reachable
Concolic will diverge instead

22. 3 More Enhancements Incremental Solving User Annotations
Random Solving

23. After Negation Concolic is Stuck
Problem for Concolic
[ X>0 & Y!=10 & Y>10]
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
test(1,11)
test(1,0)
[ X > 0 ]
[ X > 0 ]
[ X > 0 & Y != 10 ]
[ X > 0 & Y != 10 ]
[ X>0 & Y!=10 & Y>10]
[ X>0 & Y!=10 & Y<=10]
After Negation Concolic is Stuck
[ X>0 & Y=10 & Y>10]

24. SymCrete Execution void test(int x, int y) { if (x > 0) {
test(X,Y)
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X > 0 ]
[ X > 0 ] hash (X)
[ X>0 & Y=hash(X) ] S0
[ X>0 & Y=hash(X) & Y>10 ] S3
X>0 & Y>10
Y=hash(X)
X=1
Y=hash(1) =10
X>0 & Y>10 & Y=10 & X=1 UNSAT
Get another solution!
X=2
Y=hash(2) =20
X>0 & Y>10 & Y=20 & X=2 is SAT

25. SymCrete Execution @Partition({“x>3”,”x<=3”})
test(X,Y)
@Partition({“x>3”,”x<=3”})
void test(int x, int y) {
if (x > 0) {
if (y == hash(x))
S0;
else
S1;
if (y > 10)
S3;
S4; }
native int hash(x) {
if (0<=x<=10) return x*10;
else return 0;
[ X > 0 ]
[ X > 0 ] hash (X)
[ X>0 & Y=hash(X) ] S0
[ X>0 & Y=hash(X) & Y>10 ] S3
X>0 & Y>10
Y=hash(X)
X=1
Y=hash(1) =10
X>0 & Y>10 & Y=10 & X=1 UNSAT
Add user partitions one at a time
X>0 & Y>10 & X > 3
Y=hash(X)
X=4
Y=hash(4) =40
X>3 & Y>10 & Y=40 & X=4 is SAT

26. - Not all solvers support the general feature -
Random Solving
Pick solutions randomly from the solution space
Current implementation only picks randomly if the solution space is completely unconstrained
- Not all solvers support the general feature -

27. Symbolic PathFinder SPF
Implementation
Symcrete
Custom Listeners on SPF
Symbolic PathFinder SPF
Symbolic Execution extension for JPF called jpf-symbc
Model Checker for Java Open Source
JavaPathFinder

28. Conclusions
Symbolic driven use of concrete values to address problems with classic symbolic execution
In the process addresses issues with DART/Concolic analysis
But is still incomparable in strength (see next slide)
Open source implementation for Java

29. DART/Concolic gets “Lucky”
public void test (boolean b, int x, int y) {
if (b)
if(y <= 0)
{ ... }
else
else if(x <= 0 && identity(y) == 1)
{ HERE! } }

30. DART/Concolic gets “Lucky”
public void test (boolean b, int x, int y) {
if (b)
if(y <= 0)
{ ... }
else
else if(x <= 0 && identity(y) == 1)
{ HERE! } }
b=true,x=0,y=0
b,y<=0
Negating last constraint y <= 0
b=true,x=0,y=1
Now b branches done, so negate b
b=false,x=0,y=1