Presentation is loading. Please wait.

Presentation is loading. Please wait.

Automated Whitebox Fuzz Testing Patrice Godefroid (Microsoft Research)‏ Michael Y. Levin (Microsoft Center for Software Excellence)‏ David Molnar (UC-Berkeley,

Similar presentations


Presentation on theme: "Automated Whitebox Fuzz Testing Patrice Godefroid (Microsoft Research)‏ Michael Y. Levin (Microsoft Center for Software Excellence)‏ David Molnar (UC-Berkeley,"— Presentation transcript:

1 Automated Whitebox Fuzz Testing Patrice Godefroid (Microsoft Research)‏ Michael Y. Levin (Microsoft Center for Software Excellence)‏ David Molnar (UC-Berkeley, visiting MSR)

2 Fuzz Testing An effective technique for finding security vulnerabilities in software Apply invalid, unexpected, or random data to the inputs of a program. If the program fails(crashing, access violation exception), the defects can be noted.

3 Vulnerability Finding Today One Security bug can bring $500-$100,000 on the open market Good bug finders make $180-$250/hr consulting Few companies can find good people, many don’t even realize this is possible. Still largely a black art

4 Whitebox Fuzzing This paper present an alternative whitebox fuzz testing approach inspired by recent advances in symbolic execution and dynamic test generation – Run the code with some initial well-formed input – Collect constraints on input with symbolic execution – Generate new constraints (by negating constraints one by one) – Solve constraints with constraint solver – Synthesize new inputs

5 Dynamic Test Generation void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } input = “good” Running the program with random values for the 4 input bytes is unlikely to discovery the error: a probability of about Traditional fuzz testing is difficult to generate input values that will drive program through all its possible execution paths.

6 Dynamic Test Generation void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } input = “good” I 0 != ‘b’ I 1 != ‘a’ I 2 != ‘d’ I 3 != ‘!’ Collect constraints from trace Create new constraints Solve new constraints  new input.

7 Depth-First Search void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I 0 != ‘b’ I 1 != ‘a’ I 2 != ‘d’ I 3 != ‘!’ good I 0 != ‘b’ I 1 != ‘a’ I 2 != ‘d’ I 3 != ‘!’

8 Depth-First Search goo!good void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I 0 != ‘b’ I 1 != ‘a’ I 2 != ‘d’ I 3 == ‘!’

9 Practical limitation godd void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I 0 != ‘b’ I 1 != ‘a’ I 2 == ‘d’ I 3 != ‘!’ good Every time only one constraint is expanded, low efficiency!

10 Generational Search Key Idea: One Trace, Many Tests Office 2007 application: Time to gather constraints: 25m30s Tainted branches/trace: ~1000 Time per branch to solve, generate new test, check for crashes: ~1s Therefore, solve+check all branches for each trace!

11 Generational Search Key Idea: One Trace, Many Tests goo! godd gaod bood “Generation 1” test cases good void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } I 0 == ‘b’ I 1 == ‘a’ I 2 == ‘d’ I 3 == ‘!’

12 void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); } 0 good 1 goo! 1 godd 2 god! 1 gaod 2 gao! 2 gadd gad! 1 bood 2 boo! 2 bodd 2 baod bod! bad! badd baod The Generational Search Space

13 void top(char input[4]) { int cnt = 0; if (input[0] == ‘b’) cnt++; if (input[1] == ‘a’) cnt++; if (input[2] == ‘d’) cnt++; if (input[3] == ‘!’) cnt++; if (cnt >= 3) crash(); }

14 SAGE Architecture (Scalable Automated Guided Execution) Check for Crashes (AppVerifier)‏ Trace Program (Nirvana)‏ Gather Constraints (Truscan)‏ Solve Constraints (Disolver) Input0Trace File Constraints Input1 Input2 … InputN

15 Since 1 st MS internal release in April’07: dozens of new security bugs found (most missed by blackbox fuzzers, static analysis)‏ Apps: image processors, media players, file decoders,… Many bugs found rated as “security critical, severity 1, priority 1” Now used by several teams regularly as part of QA process Initial Experiences with SAGE

16 Experiments ANI Parsing - MS RIFF...ACONLIST B...INFOINAM.... 3D Blue Alternat e v1.1..IART anih$...$ rate seq LIST....framic on RIFF...ACONB B...INFOINAM.... 3D Blue Alternat e v1.1..IART anih$...$ rate seq anih....framic on Initial input Crashing test case

17 ANI Parsing - MS RIFF...ACONLIST B...INFOINAM.... 3D Blue Alternat e v1.1..IART anih$...$ rate seq LIST....framic on RIFF...ACONB B...INFOINAM.... 3D Blue Alternat e v1.1..IART anih$...$ rate seq anih....framic on Only 1 in 2 32 chance at random! Initial input Crashing test case Initial input : 341 branch constraints, 1,279,939 total instructions. Crash test cases: 7706 test cases, 7hrs 36 mins

18 Initial Experiments #Instructions and Input size largest seen so far App Tested#TestsMean DepthMean #Instr.Mean Size ANI ,066,0875,400 Media ,409,37665,536 Media ,432,48927,335 Media ,644,65230,833 Media ,685,24022,209 Compression , Office ,731,24845,064

19 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: ; h: ; h: ; h: ; h: ; h: ; h: ;.... Generation 0 – initial input – 100 bytes of “00”

20 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: ; RIFF h: ; h: ; h: ; h: ; h: ; h: ;.... Generation 1

21 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: ** ** ** ; RIFF....*** h: ; h: ; h: ; h: ; h: ; h: ;.... Generation 2

22 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ; h: ; h: ; h: ;.... Generation 3

23 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh h: ; h: ; h: ;.... Generation 4

24 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: ; h: ; h: ;.... Generation 5

25 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: ;....strf h: ; h: ;.... Generation 6

26 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: ;....strf....( h: ; h: ;.... Generation 7

27 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: ;....strf....( h: C9 9D E4 4E ; ÉäN h: ;.... Generation 8

28 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: ;....strf....( h: ; h: ;.... Generation 9

29 Zero to Crash in 10 Generations Starting with 100 zero bytes … SAGE generates a crashing test: h: D ** ** ** ; RIFF=...*** h: ; h: ; h: ;....strh....vids h: B A ;....strf²uv:( h: ; h: ;.... Generation 10 – bug ID ! Found after only 3 generations starting from “well-formed” seed file

30 Different Crashes From Different Initial Input Files 100 zero bytes X XXXX XXXXX XX XX XX XXX X XXX X X X Media 1: 60 machine-hours, 44,598 total tests, 357 crashes, 12 bugs Bug ID seed1seed2seed3seed4 seed5seed6seed7

31 Most Bugs Found are “Shallow” Generation

32 Conclusion Blackbox vs. Whitebox Fuzzing Cost/precision tradeoffs – Blackbox is lightweight, easy and fast, but poor coverage – Whitebox is smarter, but complex and slower – Recent “semi-whitebox” approaches Less smart but more lightweight: Flayer (taint-flow analysis, may generate false alarms), Bunny-the-fuzzer (taint-flow, source- based, heuristics to fuzz based on input usage), autodafe, etc. Which is more effective at finding bugs? It depends… – Many apps are so buggy, any form of fuzzing finds bugs! – Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox and/or user-provided guidance (grammars, etc.)‏ Bottom-line: in practice, use both!

33 Thank you! Questions?

34 Most bugs are “shallow”

35 Code Coverage and New Bugs

36 Some of the Challenges Path explosion When to stop testing? Gathering initial inputs Processing new test cases – Triage: which are real crashes? – Reporting: prioritize test cases, communicate to developers


Download ppt "Automated Whitebox Fuzz Testing Patrice Godefroid (Microsoft Research)‏ Michael Y. Levin (Microsoft Center for Software Excellence)‏ David Molnar (UC-Berkeley,"

Similar presentations


Ads by Google