Presentation is loading. Please wait.

Presentation is loading. Please wait.

DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel.

Similar presentations


Presentation on theme: "DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel."— Presentation transcript:

1 DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel

2 Motivation ► ► Software Testing emerges as one of the most important aspects in Software Engineering ► ► Unit Testing ► ► Limitations in Unit Testing ► ► DART a novel approach to counter limitations in Unit Testing

3 Unit Testing Testing small portions of programs, such as methods, groups of methods, or classes. ► ► a sequence of calls to the units (for e.g. methods) ► ► BEFORE the calls some code to generate parameters for the methods ► ► AFTER the calls some code checking whether the method has performed correctly.

4 Goals of Unit Testing ► ► Detect Errors in the components logic ► ► Check ALL corner cases ► ► Provide 100% code coverage

5 Question What are some limitations of Unit Testing?

6 Limitations of Unit Testing ► Requires test driver and harness code which is representative of ALL external environment ► Expensive and hard to perform manually (rarely done properly)

7 DART To The Rescue Automates unit testing removing need for writing test drivers Automates unit testing removing need for writing test drivers ► Automated extraction of the program interface by static code parsing ► Automatic generation of test driver for this interface that performs random testing ► Dynamic Analysis of the program behavior to generate specific test inputs for the direction of program execution through alternate program paths

8 Execution Model For DART ► Concrete Execution: based on a memory model M that maps addresses to values ► Symbolic Execution: based on symbolic memory S that maps addresses to expressions

9 Directed Testing by Constraint Building ► Progressively constraints are built as branches are encountered ► Linear constraints are solved to explore various execution paths ► In case of getting stuck or non linear constraints symbolic execution falls back to concrete execution

10 Consider An Example …. int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

11 Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);}

12 Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10){ printf(“I am fine here”); printf(“I am fine here”); } else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2);} Evident Problem: Probability of Reaching Abort is very low

13 DART Approach main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Full example taken from Presentation by Koushik at http://osl.cs.uiuc.edu/~ksen/slides/dart-fm.ppt

14 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36t1=m

15 DART Approach Concrete Execution concrete state symbolic stateconstraints t1=36, t2=-7t1=m, t2=nmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution

16 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=36, t2=-7t1=m, t2=n

17 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7x=m, y=n

18 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m

19 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=36, y=-7, z=72x=m, y=n, z=2m 2m != n

20 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m != n x=36, y=-7, z=72x=m, y=n, z=2m

21 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n m=1, n=2 2m != n x=36, y=-7, z=72x=m, y=n, z=2m

22 Question Which portion of code identifies a new constraint ?

23 The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

24 The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}}

25 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1t1=m

26 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n

27 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=1, t2=2t1=m, t2=n

28 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

29 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

30 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n

31 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=1, y=2x=m, y=n 2m = n m != n+10

32 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m

33 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution 2m = n m != n+10 x=1, y=2, z=2x=m, y=n, z=2m

34 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution solve: 2m = n m=1, n=2 solve: 2m = n and m=n+10 m= -10, n= -20 2m = n x=36, y=-7, z=72x=m, y=n, z=2m m != n+10

35 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10t1=m

36 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n

37 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution t1=-10, t2=-20t1=m, t2=n

38 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20x=m, y=n

39 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20x=m, y=n, z=2m

40 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m2m = n

41 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10

42 DART Approach Concrete Execution concrete state symbolic stateconstraintsmain(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2);} int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); int z = double(x); if(z==y){ if(z==y){ if(x != y+10) { printf(“I am fine here”); printf(“I am fine here”); } else { else { printf(“I should not reach here”); printf(“I should not reach here”); abort(); abort();}} Symbolic Execution Program Error x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n m = n+10

43 Properties of DART ► Sound w.r.t errors found (no false positives) ► Complete (in a limited sense) If DART terminates without reporting a bug, no bug exists If DART terminates without reporting a bug, no bug exists

44 Advantages: ► Dynamic Data Analysis ► Sound Errors

45 Dynamic Data Analysis struct foo { int i; char c; } bar (struct foo *a) { if (a->c == 0) { *((char *)a + sizeof(int)) = 1; if (a->c != 0) abort(); }

46 Dynamic Data Analysis ► Static Analyzers would not be able to detect the change in the value of a->c and declare the program to be safe ► DART locates the error by means of satisfying the constraint a->c==0

47 Sound Errors foobar(int x, int y){ if (x*x*x > 0){ if (x>0 && y==10) abort(); } else { if (x>0 && y==20) abort(); }

48 Sound Errors ► Static analysis using predicate abstraction state that both aborts may be reachable ► Test-generation using symbolic execution get stuck at first conditional ► DART detects first abort with high probability

49 DART For C ► Interface Extraction done by light weight static parsing of source code ► Test Driver Generation involves initialization of external variables with random values, stubs for external functions ► Directed Search comprises of implementation of constraint satisfaction, parsing and analysis of C code

50 Discussion


Download ppt "DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel."

Similar presentations


Ads by Google