Presentation is loading. Please wait.

Presentation is loading. Please wait.

Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES,

Similar presentations


Presentation on theme: "Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES,"— Presentation transcript:

1 Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES, USA

2 Multicore is tricky (users suspicious too) “I haven't looked at this new framework, and I don't think I will (just because I'm a c# kind of guy) but one thing I've noticed with thread-safe frameworks is that a lot of them are not actually thread safe. I'm not at all trying to undermine Moran's framework, I'm just raising this as a cautionary note to any/everyone attempting to create thread-safe API's You must ensure thread safety, mostly by invoking a set of tests meant to stress the limits of your thread-safety, including attempts to recreate "implausible" but not impossible scenarios.

3 Multicore is tricky (users suspicious too) For example, running a test where 16 threads attempt to (randomly) add, remove, or inspect … over a few billion iterations is generally what I consider the minimum for stress testing, which includes some form of journalizing so that you can determine the state your thread-safe collection should be in at the end of the test, and then compare it to the actual state. And I bring this up, because too many frameworks … failed to meet this quality standard, [one] example suffers from concurrency issues, and ends with deadlocks and other nasty side effects, which only occurs perhaps 0.001% of the time in the critical sections.” Jason Swearingen NovaLeaf.com

4 Multicore is tricky (more) “…and writing such tests is often harder than writing the code being tested in the first place, since the tests are concurrent programs too… Further, running tests on concurrent code means running on a variety of architectures, since some CPU architectures (e.g., IA32) have stronger memory models than others, and therefore certain re-orderings will not occur (at least not due to the hardware) on such systems.” Brian Goetz

5 State of the Art  Stress Testing  Dynamic Analysis  Runtime monitoring  Static Analysis  Model Checking  Symbolic Execution  Software Model Checking Guided-test [SPIN 2009] Slice And Dice [ICSE-NIER 2010] Guided-test [SPIN 2009] Slice And Dice [ICSE-NIER 2010]

6 Belief Sea of choices Guided by belief Guided-test and Slice-and-dice

7 Guided-test wants to find real errors Find an error or you do not have much to say Not intended to be exhaustive Thus not completeYet sound

8 Our Solution Possible errors locations from imprecise analysis tools Generate backward slices from possible error locations Runtime Environment Sound error detection Add inter-thread dependence information Heuristics Guided Symbolic Execution

9 Input Possible errors locations from imprecise analysis tools Possible errors locations from imprecise analysis tools Generate backward slices from possible error locations Runtime Environment Sound error detection Add inter-thread dependence information Heuristics Guided Symbolic Execution

10 Small example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

11 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

12 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

13 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

14 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

15 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

16 Small Example Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

17 Could it really be? Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Is it possible to arrive at this location on some execution?

18 Initial Abstraction to Follow (not explore) Possible errors locations from imprecise analysis tools Generate backward slices to possible error locations Generate backward slices to possible error locations Runtime Environment Sound error detection Add inter-thread dependence information Heuristics Guided Symbolic Execution

19 Slice along sequential execution Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

20 Target Locations & Start of Programs A.run l start Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

21 Call Sites and Start Locs of the callee A.run l start check(elem) A.check l start Exception Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception }

22 Conditional Statements A.run l start check(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

23 Data Definitions A.run l start check(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

24 Synchronization Operations A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

25 Abstract System A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

26 Abstract Trace A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

27 Abstract Trace A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } One trace of many we could have selected: fair coin or farm in parallel

28 Our Solution Possible errors locations from imprecise analysis tools Generate backward slices to possible error locations Runtime Environment Sound error detection Add inter-thread dependence information Heuristics Guided Symbolic Execution

29 Extract current program location of the most recently executed thread Map Concrete state to a location Thread-0 Stack LocalVars Thread-1 Stack LocalVars Thread-n Stack LocalVars Heap GlobalVars Data Input Variables Symbolic Representations Data Input Variables Symbolic Representations ϕ path Constraint s1s1 s1s1 l1l1 l1l1 Goal is to match a concrete state to the next program location in our trace Follow the trace! Goal is to match a concrete state to the next program location in our trace Follow the trace!

30 Guidance Strategy s0s0 s0s0 s2s2 s2s2 s3s3 s3s3 s1s1 s1s1 l1l1 l2l2 lnln

31 s0s0 s0s0 s2s2 s2s2 s3s3 s3s3 s1s1 s1s1 l1l1 l2l2 lnln Flip a fair coin and move forward

32 Guidance Strategy s0s0 s0s0 s2s2 s2s2 s3s3 s3s3 s1s1 s1s1 s5s5 s5s5 s6s6 s6s6 s4s4 s4s4 l1l1 l2l2 lnln Distance heuristic to choose nearest to next location on the current trace

33 Our Solution Possible errors locations from imprecise analysis tools Generate backward slices to possible error locations Runtime Environment Sound error detection Add inter-thread dependence information (refine) Heuristics Guided Symbolic Execution

34 Followed trace to predicate… A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception False in all successors (zut!)

35 Where else is elem.e defined? Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

36 Redefine a global variable B.run l start e := 11 A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception

37 Generate a new backward slice B.run l start e := 11 if(x > 18) lock(elem) unlock(elem) Element.reset l start elem.reset() Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

38 Updated Abstract System B.run l start e := 11 if(x > 18) lock(elem) unlock(elem) Element.reset l start elem.reset() A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception

39 Update Abstract Trace B.run l start e := 11 if(x > 18) lock(elem) Element.reset l start elem.reset() A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception

40 Check Lock Dependencies B.run l start e := 11 if(x > 18) lock(elem) Element.reset l start elem.reset() A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception

41 Update Abstract Trace B.run l start e := 11 if(x > 18) lock(elem) Element.reset l start elem.reset() unlock(elem) A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) } Thread B { void run (Element elem) { int x; // input variable if( x > 18) { lock(elem) elem.reset(); unlock(elem) }

42 Update Abstract State B.run l start e := 11 if(x > 18) lock(elem) Element.reset l start elem.reset() unlock(elem) A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception Restart with this new trace from the initial state

43 Slice and dice wants to find real errors and prove correct… Don’t find an error yet have much to say Goal is still to find errorsso optimize searchyet useful in no error

44 Abstract Trace A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Thread A { void run (Element elem) { lock(elem) check(elem) unlock(elem) } void check(Element elem) { if(elem.e > 9) Throw exception } Object Element { int e; Element() { e = 1; } void reset() { e = 11; } Object Element { int e; Element() { e = 1; } void reset() { e = 11; }

45 Slice and Dice  Keep entire abstraction  Schedule only on node in the abstraction A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception

46 When it is time to refine… B.run l start e := 11 if(x > 18) lock(elem) Element.reset l start elem.reset() unlock(elem) A.run l start lock(elem) check(elem) A.check l start if(elem.e > 9) Exception …rather than create a single trace…

47 Keep the entire abstraction B.run l start e := 11 if(x > 18) lock(elem) unlock(elem) Element.reset l start elem.reset() A.run l start lock(elem) check(elem) unlock(elem) A.check l start if(elem.e > 9) Exception

48 Visualization Demonstration

49 Belief Sea of choices Guided by belief Guided-test and Slice-and-dice

50 jpf-guided-test 50 Verification & Validation Lab Brigham Young University Provo, UT 84606, USA http://vv.cs.byu.edu/ VV Lab, Brigham Young University, USA

51 This slide left intentionally blank

52 Experimental Results  A new GuidedSymbolic extension in Java Pathfinder  Uses the engine from the symbolic extension (Pasareanu et. al. ISSTA 2008)  Dynamic partial order reduction turned on  Abstraction, refinement and heuristic computation all performed on Java bytecode  Libraries are part of the multi-threaded system

53 Multi-threaded Programs  Reorder Benchmark SLOC: 44, Reachability  Airline Benchmark SLOC: 31, Reachability  VecDeadlock Real JDK 1.4 concurrent library SLOC: 7267, Deadlock in Vector class  VecDeadlock Real JDK 1.4 concurrent library SLOC: 7169, Deadlock in Vector class  VecRace Real JDK 1.4 concurrent library SLOC: 7151, Race in ArrayList class

54 Error Discovery Time in minutes 60 30 20 10 Out of Patience Guided DFS VecDeadlock0 VecDeadlock1 VecRace

55 Comparison Time in seconds 60 30 20 10 Guided DFS VecDeadlock0 VecDeadlock1 VecRace

56 Error Discovery 56 SubjectGuided Search Thread #StatesTime (secs)Memory Reorder(9,1)112051.677MB Reorder(10,1)122391.677MB Airline(15,3)1612103.235MB Airline(20,2)2132797.465MB Airline(20,1)2136097.465MB VecDeadlock0213704.5666MB VecDeadlock1229486.8966MB VecRace231207.9865MB

57 Error Discovery 57 SubjectAbstraction-Refinement Total trace length Refinements Reorder(9,1)131 Reorder(10,1)131 Airline(15,3)313 Airline(20,2)319 Airline(20,1)320 VecDeadlock0141 VecDeadlock1152 VecRace121

58 Our Solution Possible errors locations from imprecise analysis tools Generate backward slices to possible error locations Runtime Environment Sound error detection Add inter-thread dependence information (refine) Heuristics (a word on symbolic) Heuristics (a word on symbolic) Guided Symbolic Execution

59 Ranking Data Non-Determinism T o sym s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 sjsj sjsj null new T() existing objects not of type X o sym.foo() X.foo() X extends T … … h(s 3 …s j-1 ) = 1 Less preferred states

60 Ranking Data Non-Determinism T o sym s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 sjsj sjsj null new T() subtypes of T existing objects … Use the abstract trace to prune the choice set …

61 Ranking Data Non-Determinism T o sym s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 sjsj sjsj null new T() subtypes of T existing objects o sym.foo() X.foo() X extends T … …

62 Ranking Data Non-Determinism T o sym s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 sjsj sjsj null new T() new X() Or Existing with type X existing objects o sym.foo() X.foo() X extends T … …

63 Ranking Data Non-Determinism T o sym s1s1 s1s1 s2s2 s2s2 s3s3 s3s3 sjsj sjsj null new T() existing objects o sym.foo() X.foo() X extends T h(s 3 …s j-1 ) = 0 … … new X() Or Existing with type X Preferred states


Download ppt "Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES,"

Similar presentations


Ads by Google