Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Today Executions, paths, and states Testing via model checking.

Similar presentations

Presentation on theme: "1 Today Executions, paths, and states Testing via model checking."— Presentation transcript:

1 1 Today Executions, paths, and states Testing via model checking

2 2 Executions, Paths, and States Control flow graph (CFG): structure of a program’s code An execution is simply a run of the program – the result of some set of commands Some executions, with different operations: A path is a particular way through the CFG – ignoring the commands – where the program “goes” Same path, different executions A state is the program’s location + memory contents at a point in time Same state, different executions, different paths (in the executions, color only shows the location in the CFG, not the contents of memory, which may be different even if colors are the same) Same paths, different states

3 3 A Question Testing can focus on exploring lots of Executions Random testing Paths Concolic testing States? Model checking But first: does this gain us anything?

4 4 A Question First question: does it make the problem more manageable That is, are there cases where exploring all states is more feasible than exploring all paths?

5 5 Paths vs. States int G = 0; int foo (int x, int y) { if (x < y) G++; if (x == 10) G++; } int bar (int x, int y) { if (x 8) G++; } int main () { foo(x, y); bar(x, y); if (x + y == 40) G++; } Once more, how many paths through this program?

6 6 Paths vs. States int G = 0; int foo (int x, int y) { if (x < y) G++; if (x == 10) G++; } int bar (int x, int y) { if (x 8) G++; } int main () { foo(x, y); bar(x, y); if (x + y == 40) G++; } What if we consider the number of states? If we include x and y’s values in the state, the number is much larger than the number of paths… so it’s certainly not always an improvement!

7 7 Paths vs. States pick(chunk, 0, 10); pick(nbytes, 0, 10); write(chunk, wbuf, nbytes); pick(chunk, 0, 10); pick(nbytes, 0, 10); read(chunk, rbuf, nbytes); pick(chunk, 0, 10); pick(nbytes, 0, 10); write(chunk, wbuf, nbytes); Less clear which is better in this case: lots of states, but also lots of paths Paths lets us consider inputs that produce the same sequence of writes as “the same” States means we don’t have to worry about error returns in the past, if that error return doesn’t change the system state

8 8 Paths vs. States So, considering states rather than paths seems to be a trade-off without a clear winner – often at least as un-manageable as complete path exploration Which is better for error detection? Hard to say, but for a class of errors (reachability), we can say that complete state coverage would definitely expose them

9 9 Model Checking A model checker is a tool for exploring a state space Basic idea: generate every reachable state of a transition system Think of states as nodes in a graph Directed edges mean “from this state, this is a possible next state of the program” Multiple outgoing edges where there is input/thread scheduling/other nondeterminism

10 10 Model Checking Model checking is a huge research field Clarke, Emerson, and Sifakis received this year’s Turing Award for coming up with the basic idea Some major subfields: Hardware and software Symbolic vs. explicit-state

11 11 Model Checking Symbolic model checkers “explore” a state space by manipulating logical expressions representing many states In explicit-state model checking (part of) the graph is actually enumerated Especially useful for software, where complex data structures are hard to handle symbolically Closer to testing Gerard’s SPIN is the most popular explicit-state model checker

12 12 Model Checking We will be looking at one particular kind of explicit-state model checking Using SPIN To explore the state spaces of C programs Other model checkers used for this approach to testing Java PathFinder 2 (NASA Ames) Bogor (U Kansas/Nebraska)

13 13 Model Checking: State-Based Testing Model-checking by executing the program Backtracking search for all states State already visited! Backtrack and try a different operation Done with test! Backtrack and try a different operation CFG State already visited! Backtrack and try a different operation Will explore, as a side-effect, many executions and many paths, but the goal is to explore states

14 14 Many Software Model Checkers CBMC BLAST SLAM JPF2 SPIN CMC CRunner MAGIC VeriSoft Bogor

15 15 Two Approaches CBMC BLAST SLAM JPF2 SPIN CMC CRunner MAGIC Analysis of derived transition system Execution of actual code VeriSoft CRunner Our focus in this class (dynamic: like testing) (static: like a complete analysis) Bogor

16 16 SPIN and Model-Driven Verification SPIN 4.0 introduced model-driven verification Can embed C code in a PROMELA model Code is executed as part of a transition during depth-first search SPIN compiles a PROMELA model into a C program: it’s a model checker generator Embed C code in transitions by executing the compiled C code – at native speed, with real compiled behavior (including optimizations) Take advantage of all SPIN features – hashing, multicore exploration, etc.

17 17 SPIN and Model-Driven Verification Used to “model check” significant pieces of implementation code at JPL Very useful for checking rich properties that automated abstraction engines often fail to handle Functional correctness (differential checking with a reference implementation) Invariants of rich data structures Using SPIN for testing C programs

18 18 SPIN and Model-Driven Verification When SPIN backtracks, it uses information on how to restore the state of the C program: Tracked memory is restored on backtrack Matched memory is also used to determine if a state has been visited before All other memory is not restored on backtrack or stored in state table Execute C code until control returns to SPIN Push tracked & matched state on stack Has state been visited before? Store matched state in state table Backtrack: pop stack & restore tracked & matched state Y N

19 19 SPIN and Model-Driven Verification Use state that is tracked but not matched to introduce abstractions E.g., symmetry reduction: a property that does not depend on the order of items in a linked list Track the original version of the list Copy and sort the list after each transition Match on the sorted list only SPIN will backtrack when it visits a state with same list contents, even in a different order

20 20 Testing via Model Checking We do not expect to cover all states There are far too many (trillions, at least) Use abstraction (“consider this state and that state to be the same”) to limit the exploration – may miss errors Use techniques that have a small probability of missing some states (hashing schemes)

21 21 Testing via Model Checking Unsound abstractions: For example, in the flight flash file systems we’ve checked, the state of the flash device Used page Free page Dirty page Bad block live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

22 22 Testing via Model Checking Does not guarantee that states (or errors) won’t be missed, but provides a better understanding of which states have been explored, based on the program’s purpose Manages state space size Used page Free page Dirty page Bad block live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)

23 23 Unsound Abstractions “Abstraction” can mean a lot of things What I’ll be talking about in this class: One operation: from a concrete state, c, we can compute an abstract state A(c) Backtrack according to abstract state matching Goal: we would like to explore every reachable abstract state of a C program

24 24 Unsound Abstractions For some abstractions, the goal is reached if we simply match on abstract state – if  c 1, c 2. A(c 1 ) = A(c 2 )   c 3. T(c 1, c 3 )   c 4. T(c 2, c 4 )  A(c 3 ) = A(c 4 )

25 25 Unsound Abstractions Otherwise: These states abstract the same So do these And these These, however, do not abstract equivalently, and can only be reached through their parents (and are the only concrete representatives of their respective abstract states). Whichever of these concrete states we explore first, we will miss one of the children.

26 26 Unsound Abstractions We usually don’t have soundness for any abstractions that reduce the state-space to a manageable size If not, unsound in our context (underapproximating abstract successors by abstracting concrete states) That’s ok It’s just testing, we don’t expect to be complete

27 27 Testing via Model Checking Requires the ability to restore a running program to an earlier execution state Difficult engineering problem, handled by applying automatic code instrumentation Rewrite the program to check that the backtracking works properly As a side-benefit, can simulate flight software resets and hardware faults

28 28 Simple PROMELA Code int x; int y; active proctype main () { if :: x = 1 :: x = 2 fi; assert (x == y); } Start simple This model has 7 states What are they? State = (PC, x, y) SPIN’s nondeterministic choice construct Picks any one of the choices that is enabled How do we guard a choice? if :: (x y = 1 :: (x y = 3 :: (x > 1) -> y = 4 fi; Not mutually exclusive!

29 29 Simple PROMELA Code int x; int y; active proctype main () { if :: x = 1 :: x = 2 fi; if :: y = 1 :: y = 2 fi; if :: x > y -> x = y :: y > x -> y = x :: else -> skip fi; assert (x == y); } This model has 17 states What are they? State = (PC, x, y) Er… Don’t worry about state-counting too much – SPIN has various automatic reductions and atomicity choices that can make that difficult

30 30 Simple PROMELA Code int x; active proctype main () { do :: (x x++ :: break od Only a couple more PROMELA constructs to learn for building test harnesses: the do loop Like if, except it introduces a loop to the top – break choice can exit the loop This nondeterministically assigns x a value in the range 0…9

31 31 Simple PROMELA Code int x; inline pick (var, MAX) var = 0; do :: (var var++ :: break od inline gives us a macro facility As you can imagine, this is a useful macro for building a test harness!

32 32 Testing via Model Checking Basic idea: We’ll write a test harness in PROMELA Use SPIN to backtrack and explore all input choices Call C code we’re testing Use abstraction to limit the number of states we consider We can even “trick” SPIN into doing pure random testing!

33 33 Testing via Model Checking Rest of class: We’ll look at the idea of “model-driven verification” that drives all this I’ll show how to write a simple PROMELA harness to test C code Thursday: we’ll investigate automatic code instrumentation, some novel coverage-based code strategies, and see how model checking and random testing compare Download SPIN yourself, and play with it! Examples available on class website

Download ppt "1 Today Executions, paths, and states Testing via model checking."

Similar presentations

Ads by Google