1. Masahiro Fujita Yoshihisa Kojima University of Tokyo May 2, 2008
Intelligent automatic test pattern generation for C-based HW/SW co-design descriptions through combined use of concrete and symbolic simulations
Masahiro Fujita　　　Yoshihisa Kojima
University of Tokyo
May 2, 2008

2. Background
In high-level SoC design, system behavior can be described in C-like programming languages
Target both hardware and software
Tool support is not sufficient
Difficulties compared with RTL or lower design descriptions
Many wide-bit word-level signals (large exploration space)
Complicated control flow (many paths)
Difficulty in modeling various descriptions
SW: pointers, pointer-arithmetic, casting, dynamic allocation, recursive calls…
HW: concurrency, synchronization, throughput, latency…
Our goal is to assist test case generation for system-level descriptions in C-like languages
Automatic input pattern generation
Assertion-based verification to find bugs
For higher code coverage that results in higher confidence

3. Most important issues in debugging
Generally speaking, counter examples generated by simulation/emulation are very “long”
Could be billions of cycles
Not east at all to understand why error occurs
Need much shorter counter examples just to understand why the bug happens
Are those long sequences really necessary ?
Bounded model checking is based on assertions with “constraints”
Bounds cannot be large
Can we drive good constraints from the counter examples found in simulation/emulation ?
There can be more direct path
Initial state
Bug
Bug
Initial state
Loops can be skipped
State space
State space

4. Target language SpecC = ANSI-C + mechanisms for HW
Structural hierarchy
Parallelism
Synchronization
Channel
Languages discussed here
C language
Some additional features
Behavior
Ports
Interfaces
Channel b1 b2 v1 c1 B p1 p2
Variable
(wire)
Child behaviors

5. Outline Background Problem definitions for input pattern generation
Preliminaries
branch / path / coverage definitions
Concrete/symbolic hybrid simulation
Concrete simulation, symbolic simulation
Hybrid simulation
Proposed Method for branch coverage
Implementation
Experimental Results
Conclusion and Future work

6. Requirements for input pattern generation (1)
For assertion failure detection
Given a design description annotated with
Input variable definitions
Assumption for input variables as predicates
Assertion predicates
Possible result
Assertion violation (and input value assignments),
Assertion holds for all possible input values,
Unknown
int func(int x, int y) {
int r = 0;
if (x – y > 0)
r = x - y;
else
r = y – x;
return r; }
Assertion failure
Counter examples exist:
(x = 0, y = 0)
(x = 3, y = 3)
...
int x, y;
FL_INPUT(x);
FL_INPUT(y);
FL_ASSUME(x >= 0);
FL_ASSUME(y >= 0);
FL_ASSERT(func(x, y) > 0);

7. Requirements for input pattern generation (2)
For branch coverage:
Given design description with annotations and target branch coverage
Generate set of test cases (input value assignments) to cover branches
Tell how to activate code fragments as many as possible (over multiple runs)
int x, y;
FL_INPUT(x);
FL_INPUT(y);
if (x > 2) { }
if (y > 2) {
Test cases of
(1) (x = 0, y = 0)
(2) (x = 3, y = 3) will achieve 100% branch coverage

8. Outline Background Problem definitions for input pattern generation
Preliminaries
branch / path / coverage definitions
Concrete/symbolic hybrid simulation
Concrete simulation, symbolic simulation
Hybrid simulation
Proposed Method for branch coverage
Implementation
Experimental Results
Conclusion and Future work

9. Branch / path definitions
A (pair of) conditional branch(es):
Associated with if, do-while, for, switch-case, and while statements
A branch is covered when the associated condition has been evaluated as true (or false) at least once (over multiple runs)
if (cond)
else
then
BC = cond
BC = ! cond

10. Branch / path definitions
A path is a sequence of branches taken
A path condition is defined as the conjunction of all the branch conditions taken
A false (infeasible) path is a path such that there is no value assignment which satisfies the path condition
1: void func(int x, int y) {
2: if (x > 2) {
3: }
4: if (x < 2) {
5: }
6: }
1: void func(int x, int y) {
2: if (x > 2) {
3: } else {
4: }
5: if (y > 2) {
6: } else {
7: }
8: }
There appear to be 4 paths;
But the path condition is
(x > 2) AND (x < 2)
INFEASIBLE!
There are 4 paths;
The path condition is (x > 2) AND NOT(y > 2)

11. Branch / path coverage definitions
Branch coverage
# of branches covered out of # of all branches
Path coverage
# of paths covered out of # of all (or feasible) paths
Difficult to use in practice because:
The number of feasible paths cannot be known so easily
The number of possible paths can be huge
Exponential w.r.t. # of if-statements * loop iterations if
Exercised 2 runs:
branch coverage: 4 / (2 + 2) (100%)
path coverage: 2 / (2 * 2) (50%) if

12. Outline Background Problem definitions for input pattern generation
Preliminaries
branch / path / coverage definitions
Concrete/symbolic hybrid simulation
Concrete simulation, symbolic simulation
Hybrid simulation
Proposed Method for branch coverage
Implementation
Experimental Results
Conclusion and Future work

13. Traditional (concrete) simulation approach
Create test cases (input values) by hand
Not so easy
Or, generate randomly
Automated, but maybe difficult to activate the corner cases
In system level descriptions, the search space can be huge (e.g. 32-bit word level signals)
Run simulation
Very simple, but how long does it take to hit the failure?
Incomplete: cannot prove the assertion ALWAYS holds
unless all possible values have been exercised (not practically possible)
Confidence (quality of tests): given by coverage metrics
E.g. Branch-coverage
Try (x=3, y=100) => r=97 > 0 OK
Try (x=1, y=20) => r=19 > 0 OK
...
Try (x=10, y=10) => r=0 > 0 NG! (may eventually happen, but much rarely)

14. Formal approach Word-level approach: Symbolic simulation
Build the formal expressions and mathematically solve the constraints
Precise & Complete
Computationally expensive
Word-level approach: Symbolic simulation
Evaluates values as symbolic expressions instead of concrete values

15. Symbolic Simulation Needs to enumerate all the paths
Sometimes the path can be infeasible (false-path problem)
path-condition
Path1:
(r_1=0) (x – y > 0)
(r_2=x - y)
(x>=0)
(y>=0)
-> (r_2>0)
Path2:
(r_1=0)
NOT(x – y > 0)
(r_2=y -x)
(x>=0)
(y>=0)
-> (r_2>0)
int func(int x, int y) {
int r = 0;
if (x – y > 0)
r = x - y;
else
r = y – x;
return r; }
Path1
path2
Enumerates possible paths
(including infeasible ones)
INVALID
Counter Example: (y - x=0)
(some of them may be reported)
VALID
for all x,y

16. Symbolic simulation (cont’d)
Employs SMT (satisfiability modulo theory) solver
To solve path conditions
To evaluate assertions
For each path:
One symbolic simulation on a path corresponds to concrete simulations of all possible values on that path
Limitations:
# of paths (including false paths)
Size of symbolic expressions
Solver capability (non-linear algebra)
How to model complicated descriptions
May not be applied straightforwardly to complex / large descriptions

17. Concrete-symbolic hybrid approach
Combines concrete simulation and symbolic simulation (originally proposed by Larson[5])
CUTE[11] is proposed for unit testing
Exhaustive traversal on all paths
Concrete run guides the path for symbolic simulation (initially random simulation)
Symbolic run on that path derives the path-condition
Use concrete values for approximation if the constraints cannot be processed (e.g. non-linear)
Solve the constraints to guide the path to another
Negate some path-condition term to take another branch

18. Concolic Simulation (1st)
initially
random
Concrete States
x=0
y=0
z=0
(0 > 3)?
-> no!
Symbolic States
x=i1
y=i2
z=i3
(i1 > 3)?
Path Condition
(i1 <= 3)
1: void test(int x,
int y,
int z) {
2: if (x > 3) // B1
3: if (y > 11) // B2
4: if (z == y*y) // B3
5: if (x < 5) // B4
6: reach_me();
7: }
Negate this condition
And solve to take THEN branch at B1
Find the inputs to reach reach_me()

19. Concolic Simulation (2nd)
Concrete States
x=10
y=0
z=0
(10 > 3)
(0 > 11)?
-> no!
Symbolic States
x=i1
y=i2
z=i3
(x > 3)
(y <= 11)
Path Condition
(i1 > 3)
(i2 <= 11)
1: void test(int x,
int y,
int z) {
2: if (x > 3) // B1
3: if (y > 11) // B2
4: if (z == y*y) // B3
5: if (x < 5) // B4
6: reach_me();
7: }
Negate this condition
And solve to take THEN branch at B2
Find the inputs to reach reach_me()

20. Concolic Simulation (3rd)
Concrete States
x=10
y=20
z=0
(10 > 3)
(20 > 11)
(0 == 400)?
-> no!
Symbolic States
x=i1
y=i2
z=i3
(x > 3)
(y > 11)
(z == y*y)
Path Condition
(i1 > 3)
(i2 > 11)
(i3 != 400)
1: void test(int x,
int y,
int z) {
2: if (x > 3) // B1
3: if (y > 11) // B2
4: if (z == y*y) // B3
5: if (x < 5) // B4
6: reach_me();
7: }
Non-linear i2*i2 is replaced by 400.
Negate this condition
And solve to take THEN branch at B3
Find the inputs to reach reach_me()

21. Concolic Simulation (4th)
Concrete States
x=10
y=20
z=400
(10 > 3)
(20 > 11)
(400 == 400)
(10 < 5)?
-> no!
Symbolic States
x=i1
y=i2
z=i3
(x > 3)
(y > 11)
(z == 400)
(x >= 5)
Path Condition
(i1 > 3)
(i2 > 11)
(i3 == 400)
(i1 >= 5)
1: void test(int x,
int y,
int z) {
2: if (x > 3) // B1
3: if (y > 11) // B2
4: if (z == y*y) // B3
5: if (x < 5) // B4
6: reach_me();
7: }
Find the inputs to reach reach_me()
Negate this condition
And solve to take THEN branch at B4

22. Concolic Simulation (5th)
Concrete States
x=4
y=20
z=400
(4 > 3)
(20 > 11)
(400 == 400)
(4 < 5)
Symbolic States
x=i1
y=i2
z=i3
(x > 3)
(y > 11)
(z == 400)
(x < 5)
Path Condition
(i1 > 3)
(i2 > 11)
(i3 == 400)
(i1 < 5)
1: void test(int x,
int y,
int z) {
2: if (x > 3) // B1
3: if (y > 11) // B2
4: if (z == y*y) // B3
5: if (x < 5) // B4
6: reach_me();
7: }
Find the inputs to reach reach_me()
Reached successfully!

23. Concolic approach Can be applied to work-around non-linear
Can be used to enumerate the paths
Good for path coverage
Can be used to guide the path
But CUTE does not think about which path should be tried next
As CUTE’s strategy is exhaustive
May not terminate if # of paths is huge

24. Outline Background Problem definitions for input pattern generation
Preliminaries
branch / path / coverage definitions
Concrete/symbolic hybrid simulation
Concrete simulation, symbolic simulation
Hybrid simulation
Proposed Method for branch coverage
Implementation
Experimental Results
Conclusion and Future work

25. Proposed method
Flip a branch condition on a path only when not covered yet
Gives the priority for path enumeration
Skips the uncovered paths that do not contribute to the branch coverage
Terminates when the target coverage is achieved
Tries to avoid enumerating all the paths
Not guaranteed to cover all possible branches
Derived alternative paths may not be feasible
Worst case: all paths need to be enumerated
Also limited by the solver’s capability (i.e. path condition may not be solved)

26. Our implementation Implemented on FLEC (our C-Equivalence Checker)
Used as SpecC[3] frontend
Control/data/communication/… dependencies have been extracted
AST interpreter
Evaluates AST node (expression / statement) one by one
C.f. CUTE: instrument & compile
We can start from any points in the program !
Concrete simulator evaluates with concrete values
Symbolic simulator evaluates with symbolic expressions
Branch/Path coverage profiler
Input pattern generator
For alternative path
For assertion failure
SMT solver: CVC3[12]
To generate input patterns
To evaluate assertions
C.f. CUTE: lpsolve

27. Outline Background Problem definitions for input pattern generation
Preliminaries
branch / path / coverage definitions
Concrete/symbolic hybrid simulation
Concrete simulation, symbolic simulation
Hybrid simulation
Proposed Method for branch coverage
Implementation
Experimental Results
Conclusion and Future work

28. Experimental results (1/3)
Simple example
Achieved 2 / 2 (100%) branch coverage with 2 runs
Detected assertion failure with (x=0, y=0)
1: int func(int x, int y) {
2: int r = 0;
3: if (x – y > 0)
4: r = x – y;
5: else
6: r = y – x;
7: return r;
8: }
9: void main() {
10: int x, y;
11: FL_INPUT(x);
12: FL_INPUT(y);
13: FL_ASSUME(x >= 0);
14: FL_ASSUME(y >= 0);
15: FL_ASSERT(func(x, y) > 0);
16: }

29. Experimental results (2/3)
1: unsigned int fact_rec(unsigned int s) {
2: if ( s <= 1) {
3: return 1;
4: } else {
5: unsigned int t;
6: unsigned int p;
7: t = s * fact_rec(s – 1);
8: return t;
9: }
10: unsigned int fact_for(unsigned int s) {
11: unsigned int i;
12: unsigned int p;
13: p = 1;
14: for (i = 1; i <= s; i++) {
15: p *= I;
16: }
17: return p;
18: }
19: void main() {
20: int i, o1, o2;
21: FL_INPUT(i);
22: FL_ASSUME(i <= 10);
23: o1 = fact_for(i);
24: o2 = fact_rec(i);
25: FL_ASSERT(o1 == o2);
26: }
Calculate factorial with two implementations
With recursive function calls
With for-loop
Validated for one path (i = 8)
Achieved 4/4 (100%) branch coverage with 1 run

30. Experimental results (3/3)
# of branches: 10
# of paths: 4 * 2^100
Achieved 10 / 10 (100%) branch coverage with 5 runs
Detected assertion failure with (x=1, y=2, z=3)
CUTE got stuck due to too many paths
1: int f(int x,int y, int z) {
2: int p;
3: if (x+y+z == 6)
4: if (2*x+7*y+3*z==25)
5: if(-4*x-2*y+2*z==-2)
6: FL_ASSERT(0);
7: for (p = 0; p < 100; p++) {
8: if (p == z) {
9: }
10: }
11: }
12: void main() {
13: int x, y, z;
14: FL_INPUT(x);
15: FL_INPUT(y);
16: FL_INPUT(z);
17: f(x, y, z);
18: }

31. Elevator controller profile
Elevator controller (abstracted model)
Cycle-based behavior
Simple, but designed by real engineer
There is a not-intended bug
Inputs:
3 Floors
Up request buttons on 1F and 2F
Down request buttons on 2F and 3F
1 Cabin
3 buttons for floor stop request
2 buttons for door open / close
Outputs:
Up, Down request status
Floor stop request status
Door open/close
Cabin vertical speed (0: stopped, +1: up, -1: down)
Cabin position (on 1F, b/w 1F and 2F, on 2F, b/w 2F and 3F, on 3F)
Service direction (0: none, +1: up, -1: down) 3F 1F 2F 3F 2F
open
close 1F

32. Elevator controller profile (cont’d)
State variables:
Up/Down request status (2+2)
Floor stop request status (3)
Door status (1)
Cabin position (on 1F, b/w 1F and 2F, on 2F, b/w 2F and 3F, on 3F)
Cabin speed (0: stopped, +1: up, -1: down)
Service direction (0: none, +1: up, -1: down)
2^8 * 5 * 3 * 3 = 11.5k states (including infeasible ones)
Initially stopped on 1F, door closed, no request active
Original code: 396 lines in SpecC
145 million paths (including infeasible)
Replaced if-then-else & switch-case statements with conditional (cond ? True : false) expressions
To handle multiple paths at once
Simple control flow (straight line), but very complex data flow
Reduced to 155 lines

33. Elevator controller profile (cont’d)
Property examples
Elevator must be on or between 1F and 3F
ASSERT((out_position >= 0) && (out_position <= 4));
Door opens only when the elevator is stopped on either of 1F, 2F and 3F
ASSERT
(!out_door ||
( (out_speed == 0) && ( (out_position == 0) || (out_position ==2) || (out_position == 4))))

34. Symbolic simulation result
Symbolic expression explodes in 3-4 cycles of symbolic simulation
With constant propagation/substitution
With simplifications for ITE, AND, OR, and other operators
Without concrete-value substitution (approximation)
Without common sub-expression sharing
# of cycles of symbolic simulation must be highly bounded!
Beginning of
Symbolic simulation
300k nodes
and more!
Reset
sequence

35. User guided simulation
Starts symbolic simulation from the specified state by the user
Explore with respect to the states of user’s interest
Some of the states (proved to be) reachable by concrete (random) simulation
Jump into the states (which may or may not be feasible)
Will need to check its feasibility later
Cycle is bounded
Concrete
simulation
Symbolic
simulation
Symbolic
simulation
State space
Might be infeasible
Initial states
Paths unknown

36. User guided result (1)
Try to generate the input pattern to make a situation where
Located on 2F
Speed = -1 (down)
(not a bug)
I.e. to violate ASSERT (!((out_speed == -1) && (out_position == 2)))
This state is out of bound from the initial state (stopped on 1F)
Need more than 3 cycles for elevator to accept request on 1F, start moving, go up at least to 2F, and go down…

37. User guided result (1) (cont’d)
So let’s jump in to one of the feasible state
state_position = 4, state_door = false, state_speed = 0 …
Known as a reachable state by random simulation a priori
Found one of the input pattern to violate the cycle 5 (3rd cycle of symbolic sim.)
Up request on cycle 1 = true
Up request on cycle 1 = false
Down request on cycle 1 = false
Stop on 1F cycle 1 = false
Stop on 2F cycle 1 = false

38. User guided result (2) Try to violate the assertion
Elevator must be on or between 1F and 3F
ASSERT((out_position >= 0) && (out_position <= 4));
Let’s jump into one of the state
state_position = 4 (on 3F)
state_speed = +1 (up)
next state goes into
out_position = 5 (higher than 3F!)
And violates the assertion!
However, the state (state_position = 4, state_speed = +1) is actually infeasible
Wrong assumption may lead a wrong conclusion
The feasibility of the originating state should be verified in some way

39. Conclusion & Future work
Implemented concrete/symbolic hybrid simulator based on AST interpreter
Proposed a method for input pattern generation for branch coverage
Experimental results demonstrate the input pattern generation
For assertion failure detection
For better branch coverage
Future work
Capability to cover the specified target branch
Handling of concurrent executions
Hybrid simulation heuristic tuning
Efficient management of symbolic expressions

40. References
[3] D. D. Gajski, J. Zhu, R. Domer, A. Gerstlauer, and S. Zhao. SpecC: Specification Language and Methodology. Kluwer Academic Publishers, 2000.
[5] E. Larson and T. Austin. High coverage detection of input-related security facults. In SSYM’03: Proc of 12th conf on USENIX Security Symbosium, 2003.
[11] K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for c. In Proc. Of Esec/SIGSOFT FSE-13, 2005.
[12] A. Stump, C. Barrett, and D. Dill. CVC: a cooperating validity checker. In 14th int’l conf on computer-aided verification, 2002

41. Difficulty compared with RTL or lower
In traditional methodology for RTL or gate-level
Word signals are converted into bit-vector
Then, solved with Boolean algebra
Efficient algorithms available: SAT, BDDs…
In system-level descriptions
Too many word signals, too wide words (32 bit / 64 bit)
Too wide space to explore
Complicated control-flow
Data-flow dynamically changes depending on the path
Control-conditions are complex
Too many paths

42. Difficulty compared with RTL or lower (cont’d)
In system-level descriptions
To model software
Recursive calls, pointers, pointer-arithmetic, type-casting, dynamic-allocations…
To model hardware
Concurrency, synchronization, throughput, latency…
As word-level solvers, SMT solvers can be employed, but with limited capability
Usually up to linear algebra
Need approximation / workaround, otherwise it would not work!