Download presentation
Presentation is loading. Please wait.
1
DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel
2
Motivation ► ► Software Testing emerges as one of the most important aspects in Software Engineering ► ► Unit Testing ► ► Limitations in Unit Testing ► ► DART a novel approach to counter limitations in Unit Testing
3
Unit Testing Testing small portions of programs, such as methods, groups of methods, or classes. ► ► a sequence of calls to the units (for e.g. methods) ► ► BEFORE the calls some code to generate parameters for the methods ► ► AFTER the calls some code checking whether the method has performed correctly.
4
Goals of Unit Testing ► ► Detect Errors in the components logic ► ► Check ALL corner cases ► ► Provide 100% code coverage
5
Question What are some limitations of Unit Testing?
6
Limitations of Unit Testing ► Requires test driver and harness code which is representative of ALL external environment ► Expensive and hard to perform manually (rarely done properly)
7
DART To The Rescue Automates unit testing removing need for writing test drivers Automates unit testing removing need for writing test drivers ► Automated extraction of the program interface by static code parsing ► Automatic generation of test driver for this interface that performs random testing ► Dynamic Analysis of the program behavior to generate specific test inputs for the direction of program execution through alternate program paths
8
Execution Model For DART ► Concrete Execution: based on a memory model M that maps addresses to values ► Symbolic Execution: based on symbolic memory S that maps addresses to expressions
9
Directed Testing by Constraint Building ► Progressively constraints are built as branches are encountered ► Linear constraints are solved to explore various execution paths ► In case of getting stuck or non linear constraints symbolic execution falls back to concrete execution
10
Consider An Example …. int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}
11
Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);}
12
Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);} Evident Problem: Probability of Reaching Abort is very low
13
DART Approach main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Full example taken from Presentation by Koushik at http://osl.cs.uiuc.edu/~ksen/slides/dart-fm.ppt
14
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36t1=m
15
DART Approach Concrete Execution concrete state symbolic stateconstraints t1=36, t2=-7t1=m, t2=nmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution
16
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36, t2=-7t1=m, t2=n
17
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7x=m, y=n
18
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m
19
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m 2m != n
20
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m != n x=36, y=-7, z=72x=m, y=n, z=2m
21
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n m=1, n=2 2m != n x=36, y=-7, z=72x=m, y=n, z=2m
22
Question Which portion of code identifies a new constraint ?
23
The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}
24
The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}
25
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1t1=m
26
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n
27
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n
28
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n
29
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n
30
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n
31
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n 2m = n m != n+10
32
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m
33
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m
34
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n and m=n+10 m= -10, n= -20 2m = n x=36, y=-7, z=72x=m, y=n, z=2m m != n+10
35
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10t1=m
36
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n
37
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n
38
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20x=m, y=n
39
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20x=m, y=n, z=2m
40
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m2m = n
41
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10
42
DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution Program Error x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10
43
Properties of DART ► Sound w.r.t errors found (no false positives) ► Complete (in a limited sense) If DART terminates without reporting a bug, no bug exists If DART terminates without reporting a bug, no bug exists
44
Advantages: ► Dynamic Data Analysis ► Sound Errors
45
Dynamic Data Analysis struct foo { int i; char c; } bar (struct foo *a) { if (a->c == 0) { *((char *)a + sizeof(int)) = 1; if (a->c != 0) abort(); }
46
Dynamic Data Analysis ► Static Analyzers would not be able to detect the change in the value of a->c and declare the program to be safe ► DART locates the error by means of satisfying the constraint a->c==0
47
Sound Errors foobar(int x, int y){ if (x*x*x > 0){ if (x>0 && y==10) abort(); } else { if (x>0 && y==20) abort(); }
48
Sound Errors ► Static analysis using predicate abstraction state that both aborts may be reachable ► Test-generation using symbolic execution get stuck at first conditional ► DART detects first abort with high probability
49
DART For C ► Interface Extraction done by light weight static parsing of source code ► Test Driver Generation involves initialization of external variables with random values, stubs for external functions ► Directed Search comprises of implementation of constraint satisfaction, parsing and analysis of C code
50
Discussion
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.