1. Automated Whitebox Fuzz Testing
Patrice Godefroid (Microsoft Research)‏
Michael Y. Levin (Microsoft Center for Software Excellence)‏
David Molnar (UC-Berkeley, visiting MSR)
Presented by Yuyan Bao

2. Acknowledgement This presentation is extended and modified from
The presentation by David Molnar
Automated Whitebox Fuzz Testing
The presentation by Corina Păsăreanu (Kestrel)
Symbolic Execution for Model Checking and Testing

3. Agenda Fuzz Testing Symbolic Execution Dynamic Testing Generation
Generational Search
SAGE Architecture
Conclusion
Contribution
Weakness
Improvement
Traditionally fuzz testing tools

4. Fuzz Testing An effective technique for finding
security vulnerabilities in software
Apply invalid, unexpected, or
random data to the inputs of a program.
If the program fails(crashing, access violation exception), the defects can be noted.
Traditionally fuzz testing tools

5. Whitebox Fuzzing
This paper present an alternative whitebox fuzz testing approach inspired by recent advances in symbolic execution and dynamic test generation
Run the code with some initial well-formed input
Collect constraints on input with symbolic execution
Generate new constraints (by negating constraints one by one)
Solve constraints with constraint solver
Synthesize new inputs
Just to give a brief overview, whitebox fuzzing is divided into several steps.

6. White Box Testing and Symbolic Execution
A method for deriving test cases which satisfy a given path
Performed as a simulation of the computation on the path
Initial path function = Identity function, Initial path condition = true
Each vertex on the path induce a symbolic composition on the path function and a logical constraint on the path condition:
If an assignment was made:
If a conditional decision was made:
path condition path condition branch condition
Output: path function and path condition for the given path
Slide from “White Box Testing and Symbolic Execution” by Michael Beder
White Box Testing and Symbolic Execution

7. Symbolic Execution Symbolic execution tree: Code: Not reachable FALSE!
(PC=“path condition”)
- “Simulate” the code using symbolic values instead of program numeric data
Symbolic execution tree:
x:X,y:Y
PC: true
PC: X>Y
PC: X<=Y
true
false
x:X+Y,y:Y
x:X+Y,y:X
x:Y,y:X
PC: X>Y  Y-X>0
PC:X>Y  Y-X<=0
FALSE!
Not reachable
int x, y;
if (x > y) {
x = x + y;
y = x - y;
x = x - y;
if (x – y > 0)
assert (false); }

8. Dynamic Test Generation
Executing the program starting with some initial inputs
Performing a dynamic symbolic execution to collect constraints on inputs
Use a constraint solver to infer variants of the previous inputs in order to steer the next executions

9. Dynamic Test Generation
void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
input = “good”
Point out this is a dynamic technique. Consider the program shown. This program takes 4 bytes as input and contains an error when the value of the variable cnt is greater than or equal to 3 at the end of the function top.
We start running the function top with the initial 4-letters string “good”.
Running the program with random values for the 4 input bytes is unlikely to discovery the error: a probability of about
Traditional fuzz testing is difficult to generate input values that will drive program through all its possible execution paths. 9

10. Dynamic Test Generation
void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
input = “good”
I0 != ‘b’
I1 != ‘a’
I2 != ‘d’
I3 != ‘!’
Collect constraints from trace
Create new constraints
Solve new constraints  new input. 10

11. Depth-First Search good I0 != ‘b’ I1 != ‘a’ void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
I2 != ‘d’
I0 != ‘b’
I3 != ‘!’
I1 != ‘a’
I2 != ‘d’
We collect constraints from trace path that represents all the input vectors that drive the program through the path.
I3 != ‘!’
good

12. Depth-First Search good goo! void top(char input[4]) { int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
I0 != ‘b’
I1 != ‘a’
I2 != ‘d’
To force the program through a different path constraint, one can negate the last predicate of the current path constraint. And “goo!” is a solution to this path constraint.
I3 == ‘!’
good
goo!

13. Practical limitation good godd
void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
I0 != ‘b’
I1 != ‘a’
I2 == ‘d’
No matter Depth-First Search or Breadth-First Search, every time only one constraint is expanded.
However realistic programs always have large inputs and deep paths. This paper presents a new search algorithm to address path explosion limitation
I3 != ‘!’
good
godd
Every time only one constraint is expanded, low efficiency!

14. Generational search BFS with a heuristic to maximize block coverage
Score returns the number of new blocks covered
Slide from “Symbolic Execution” by Kevin Wallace, CSE

15. Key Idea: One Trace, Many Tests
Generational Search
Key Idea: One Trace, Many Tests
Office 2007 application:
Time to gather constraints: m30s
Tainted branches/trace: ~1000
Time per branch to solve, generate new test, check for crashes: ~1s
Therefore, solve+check all branches for each trace!

16. Key Idea: One Trace, Many Tests
Generational Search
Key Idea: One Trace, Many Tests
bood
void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
gaod
I0 == ‘b’
godd
I1 == ‘a’
I2 == ‘d’
Staring with an initial input and initial path constraint , the new search algorithm will attempt to expand all constraints, instead of just the last one with a depth-first search, or the first one with a breadth-first search.
I3 == ‘!’
good
goo!
“Generation 1” test cases

17. The Generational Search Space
void top(char input[4]) {
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash(); }
Consider again the program with initial input 4 letters string good. From this parent run, a generational search generates four children whose leafs are labeled with 1. Indeed those four paths each correspond to negating one constraint in the original path constraint. Each of those first generation execution paths can be expanded to generate second-generation children. By repeating this process, all feasible execution paths of the function top are eventually generate exactly once.
good 1
goo! 1
godd 2
god! 1
gaod 2
gao! 2
gadd 1
bood 2
boo! 2
bodd 2
baod
gad!
bod!
baod
badd
bad!

18. Task: SymbolicExecutor Constraints (Disolver)
SAGE Architecture (Scalable Automated Guided Execution)
Input0
Trace File
Constraints
Task: Tester
Check for
Crashes
(AppVerifier)‏
Task: Tracer
Trace
Program
(Nirvana)‏
Task:
CoverageCollector
Gather
Constraints
(Truscan)‏
Task: SymbolicExecutor
Solve
Constraints (Disolver)
The generational search algorithm has been implemented in a new tool named SAGE, which stands for…
SAGE can test any file-reading program running on windows by treating bytes read from files as symbolic inputs.
Another key novelty of SAGE is that it performs symbolic execution of program traces at the x86 binary level
Input1
Input2 …
InputN

19. Initial Experiences with SAGE
Since 1st MS internal release in April’07: dozens of new security bugs found (most missed by blackbox fuzzers, static analysis)‏
Apps: image processors, media players, file decoders,…
Many bugs found rated as “security critical, severity 1, priority 1”
Now used by several teams regularly as part of QA process
SAGE now is used by several teams in Microsoft, many high security priority bugs found by SAGE and most were missed by blackbox fuzzing efforts.

20. Experiments ANI Parsing - MS07-017
RIFF...ACONLIST
B...INFOINAM....
3D Blue Alternat
e v1.1..IART....
1996..anih$...$.
..rate
seq ..
..LIST....framic on
RIFF...ACONB
B...INFOINAM....
3D Blue Alternat
e v1.1..IART....
1996..anih$...$.
..rate
seq ..
..anih....framic on
Only 1 in 232 chance at random!
In contrast,SAGE can generate a crash starting from a well-formed specific file, having no knowledge of the file format.
On the left, an ASCII rendering of init well-formed input file. On the right, the SAGE- generated crash for MS SAGE test case changes the LIST to an additional anih record.
It takes more than 7 hours and near 8000 test case are generated.
Initial input generate : 341 branch constraints, more than 1 million total instructions.
Initial input
Crashing test case
Initial input : 341 branch constraints, 1,279,939 total instructions. Crash test cases: 7706 test cases, 7hrs 36 mins

21. Initial Experiments #Instructions and Input size largest seen so far
App Tested
#Tests
Mean Depth
Mean #Instr.
Mean Size
ANI
11468
178
2,066,087
5,400
Media 1
6890 73
3,409,376
65,536
Media 2
1045
1100
271,432,489
27,335
Media 3
2266
608
54,644,652
30,833
Media 4
909
883
133,685,240
22,209
Compression
1527 65
480,435
634
Office 2007
3008
6502
923,731,248
45,064
Clarify depth is in path constraint size. Clarify instructions are post-file-read. Size is variable because of different input-dependent paths. 21

22. Different Crashes From Different Initial Input Files
100
zero
bytes
Bug ID
seed1
seed2
seed3
seed4
seed5
seed6
seed7 X
Stack hash on left. Includes address of faulting instruction. All crashes are unhandled second-chance exceptions
5.9 crashes per machine hour. Does not include bogus-2,3,4,5 – didn’t find crashes
Other bogus files found 0 bugs.
Media 1: 60 machine-hours, 44,598 total tests, 357 crashes, 12 bugs 22

23. Conclusion Blackbox vs. Whitebox Fuzzing
Cost/precision tradeoffs
Blackbox is lightweight, easy and fast, but poor coverage
Whitebox is smarter, but complex and slower
Recent “semi-whitebox” approaches
Less smart but more lightweight: Flayer (taint-flow analysis, may generate false alarms), Bunny-the-fuzzer (taint-flow, source-based, heuristics to fuzz based on input usage), autodafe, etc.
Which is more effective at finding bugs? It depends…
Many apps are so buggy, any form of fuzzing finds bugs!
Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox and/or user-provided guidance (grammars, etc.)‏
Bottom-line: in practice, use both!

24. Contribution
A novel search algorithm with a coverage-maximizing heuristic
It performs symbolic execution of program traces at the x86 binary level
Tests larger applications than previously done in dynamic test generation

25. Weakness Gathering initial inputs
The class of inputs characterized by each symbolic execution is determined by the dependence of the program’s control flow on its inputs
Not a general solution
Only for x86 windows applications
Only focus on file-reading applications
Nirvana simulates a given processor’s architecture

26. Improvement Portability
Translate collected traces into an intermediate language
Reproduce bugs on different platforms without re-simulating the program
Better way to find initial inputs efficiently

27. Thank you!
Questions?