1. 1 Today Another approach to “coverage” Cover “everything” – within a well-defined, feasible limit Bounded Exhaustive Testing

2. 2 Coverage Revisited With some kinds of coverage, we expect to be able to reach 100% coverage if (x < y) { y = 0; x = x + 1; } else { x = y; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 Statement coverage: Cover every reachable statement

3. 3 Coverage Revisited With some kinds of coverage, we expect to be able to reach 100% coverage if (x < y) { y = 0; x = x + 1; } 3 1 2 x >= y x < y y = 0 x = x + 1 Branch coverage

4. 4 Coverage Revisited With some kinds of coverage, we expect to be able to reach 100% coverage b1b1 b2b2 b3b3 Input space partitioning

5. 5 Coverage Revisited With other kinds of coverage, we know that reaching 100% is very difficult, perhaps completely infeasible if (x < y) { y = 0; x = x + 1; } else { x = y; } if (x < y) { y = 0; x = x + 1; } 4 1 23 x >= yx < y x = y y = 0 x = x + 1 6 4 5 x >= y x < y y = 0 x = x + 1 Path coverage: exponential in the number of conditional branches!

6. 6 Coverage Revisited With other kinds of coverage, we know that reaching 100% is very difficult, perhaps completely infeasible State space coverage

7. 7 Coverage Revisited With other kinds of coverage, we know that reaching 100% is very difficult, perhaps completely infeasible Complete input coverage 4 operations x 0, 1, 2, 3, 4… 2 32 x 0, 1, 2, 3, 4… 2 32 = TOO MUCH

8. 8 Coverage Revisited What if we aim for exhaustive coverage, but arbitrarily limit the size somehow? Daniel Jackson: “small scope hypothesis” “Our approach is simply to truncate the state space artificially, checking only within some finite bounds.” - Jackson and Damon, “Elements of Style, Analyzing a Software Design Feature with a Counterexample Detector

9. 9 Small Scope Hypothesis Remember “downward scalability”? Idea related (in a hand-waving fashion) to small model property in logics: For some logical formulas, though in principle the variables may have infinite domains, it can be shown that you only have to consider a finite set of individuals when checking for satisfiability Bounded by length of the formula

10. 10 Small Scope Hypothesis The idea in a nutshell: Most faults can be exposed by some “short” failing trace Short may mean in # of operations Short may mean in complexity of input structures How many nodes in the red-black tree? Short may mean something else entirely Number of voluntary thread context switches Small flash device Small # of different pathname components Bounded pathname length

11. 11 Small Scope Hypothesis Obvious dangerous exception: resource bound violations Can be handled by “shrinking” the resource bound May require “shrinking” types in a program E.g., converting some ints to chars May be difficult, depending on program and language

12. 12 “All” Binary Trees of Size 3 N0N1N2 left N0N1N2 right N0N1N2 left right N0N1N2 left right N0 N1N2 leftright

13. 13 “All” Binary Trees of Size 3 N0N1N2 left N0N1N2 right N0N1N2 left right N0N1N2 left right N0 N1N2 leftright Do we care about the actual elements?

14. 14 “All” Binary Trees of Size 3 N0N1N2 left N0N1N2 right N0N1N2 left right N0N1N2 left right N0 N1N2 leftright What if these were red-black trees? In general: exploit isomorphisms

15. 15 Enumerating “All” Inputs For more complex data structures, enumerating all valid inputs may be a complex problem E.g., all “programs” (random ASCII sequences) of length 100 vs. All parseable C programs of length 100 May require enumeration by a constraint solver

16. 16 Enumerating “All” Inputs Enumeration may require staging Constraint solver generates a set of abstract inputs, satisfying the constraints A postprocessor then concretizes each abstract input into a (large) set of concrete inputs

17. 17 Bounded (Depth) Model Checking All states reachable with a path of no more than k transitions Or in which no loop executes more than k times…

18. 18 Bounded (Depth) Model Checking Can’t quite do the naïve thing: pan –m1000 Won’t guarantee reaching every state reachable in 1,000 steps Why not?

19. 19 Iterative Bounding Typical approach is iterative Start with a small bound If you find a failing trace, fix the software or the specification and repeat If you can show no faults with bound k Increase the bound and repeat until you can’t exhaustively test for the given bound Traditional search technique, iterative deepening

20. 20 Limitations to Scope/Bound Stop increasing scope for one of two reasons: Difficulty of generating all inputs Clever approaches can often deal with this Difficulty of executing all the test cases! This one is more fundamental With large enough k, exhaustive coverage become “exhausting”

21. 21 Evaluation How does bounded exhaustive testing stack up against random testing? Hard to say in general Marinov, Andoni, Daniliuc, Khurshid, and Rinard at MIT tried to look at this question for some Java programs “An Evaluation of Exhaustive Testing for Data Structures”

22. 22 Evaluation Marinov et al.: let’s use mutation testing and kill rates to compare Generated all tests for limit k Generated all tests for limit k-1 Compared the mutation kill rate for the complete k-1 tests to the rate for a random subset of the k tests Subset same size as complete k-1 tests

23. 23 Evaluation K Tests K-1 Tests Random subset

24. 24 Evaluation Cases where bounded exhaustive testing beat random testing

25. 25 Evaluation More interesting facts: Scope correlated highly with statement coverage Even after statement coverage was complete, increasing scope increased the mutant kill rate

26. 26 CHESS Now we’ll look at one of the most interesting bounded exhaustive testing approaches Context bounding Idea: limit number of pre-emptive context switches by threads

27. 27 Review Before I hand over to Klaus for good…

28. 28 Black box (Finite State Machine) testing Design for testability Coverage measures Random testing Constraint-based testing Debugging and test case minimization Using model checkers for testing Coverage revisited (“small model property”) Topics in Testing We’ve Covered

29. 29 Black box (Finite State Machine) testing There “are no Turing machines” Vasilevskii and Chow algorithm for conformance testing based on spanning trees and distinguishing sets Exhaustive testing that cannot miss bugs is often computationally intractable Topics in Testing We’ve Covered a ab d

30. 30 Design for testability Controllability and observability Simulation and stubbing, assertions, downward scalability, etc. Topics in Testing We’ve Covered

31. 31 Coverage measures Not necessarily correlated with fault detection! Still useful! Graph coverage: node and edge (statement and branch coverage) Logic coverage Input space partitioning Syntax-based coverage Topics in Testing We’ve Covered 4 1 23 x >= yx < y x = y y = 0 x = x + 1 ((a = y)

32. 32 Random testing Generate inputs at random Explore very large numbers of executions Relies on a good automatic test oracle Feedback to bias choices away from redundant and irrelevant inputs is useful Good baseline for evaluating other methods, and often very effective Topics in Testing We’ve Covered

33. 33 Constraint-based testing Addresses weaknesses of random testing E.g., finding needles in haystacks, such as where hash(x) = y Combines concrete and symbolic execution to generate inputs Concrete execution helps where symbolic solvers choke Topics in Testing We’ve Covered

34. 34 Debugging and test case minimization Automatic minimization of test cases is very valuable for debugging and reducing regression suite size Debugging can be considered as an application of the scientific method Various techniques exist for using test cases to localize faults Topics in Testing We’ve Covered

35. 35 Using model checkers for testing Testing based on states, rather than on executions or paths Use abstractions to reduce state space Use automatic instrumentation to handle the engineering difficulties Topics in Testing We’ve Covered

36. 36 Any Questions???