1. CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George

2. Outline  Introduction  Using CHESS  Capturing nondeterminism  Exploring nondeterminism  Evaluation  Related work

3. Introduction  Concurrent programming is difficult  Testing concurrent program is difficult  Famous concurrency bug: “Heisenbugs”  Most strategy used in finding concurrency bugs  Stress testing

4. Introduction  What is “Concurrency testing”  Concurrency scenario testing  Model checking techniques  Three key challengers by applying model checking  Perturbation problem  Rich and complex concurrency APIs in system  State-space explosion.

5. Introduction  What is “CHESS”  Concurrency scenario testing  Model checking techniques  Scale to large concurrent program  Reproduce an erroneous execution  Been integrated into the test framework inside Microsoft

6. Introduction  How does CHESS solve the previous key problems?  Testing methodology of CHESS reduce the perturbation  Only one perturbation in CHESS  Wrappers to provides enough hooks  Variety of techniques to address the state-explosion

7. Contributions of this paper  The first system for integrating model checking into concurrent systems and test frameworks with minimal perturbation;  Techniques for systematic exploration of systems code for fine-grained concurrency with shared memory and multithreading;

8. Contributions of this paper  Validation of the CHESS tool and its accompany testing and wrapper methodology on three different platforms;  A substantial number of previously unknown bugs, even in well-tested systems;  The ability to consistently reproduce crashing bugs with unknown cause.

9. Using CHESS  Traditional testing methodology  Disadvantages of this testing methodology

10. Using CHESS  CHESS architecture

11. Using CHESS  Testing methodology in CHESS

12. Using CHESS  To guarantee the previous advantages, CHESS has to control the scheduling of the tasks  Two ways to control it  Modify the scheduler  Severely limited the deployment

13. Using CHESS  Testing methodology in CHESS

14. Using CHESS  Two assumption of about the testing program  Quiescence  Definition of “Quiescence State”  How does CHESS handle “Quiescence State”

15. Capturing nondeterminism  Tasks in CHESS  Threads  Timers  Asynchronous I/Os  Wrappers in CHESS  Goal of wrappers

16. Capturing nondeterminism  Design of Wrappers in CHESS

17. Capturing nondeterminism  Synchronization wrappers

18. Capturing nondeterminism  Synchronization wrappers

19. Capturing nondeterminism  Hooking the wrappers  use various mechanisms to dynamically intercept calls to the real API functions and forward them to the wrappers.  Programs in this paper  Win32 .NET  Singularity

20. Exploring nondeterminism  How CHESS systematically drives the test  Basic scheduler operation  Allows only one thread to execute at a time  Repeatedly executes the same test driving each iteration of the test through a different schedule  Three phases in each iteration

21. Exploring nondeterminism  Replay phase  Replays a sequence of scheduling choices from a trace file  Record phase  Schedules a thread till the thread yields the processor  Search phase  Uses the enabled information at each schedule point to determine the scheduler for the next iteration

22. Exploring nondeterminism  Imperfect replay  Not rely on perfect replay capability  Common sources of nondeterminism  Lazy-initialization  Interference from environment  Nondeterministic calls

23. Exploring nondeterminism  Ensuring fair schedules  Can not enumerate all fair schedules  State-explosion  Definition of State-explosion  Inserting preemptions  Capturing states

24. Evaluation  Systems on which CHESS has been run on

25. Evaluation  Test scenarios and findings

26. Evaluation  Validation CHESS against stress-testing  Common objection of the CHESS  Failure and bug in CHESS  Succeeded in reproducing every stress-test failure

27. Evaluation  Description of two bugs  PLINQ bug

28. Evaluation  Description of two bugs  Singularity bug

29. Related work  Repeatable deterministic testing  Systematic generation of thread schedules  Applying state exploration directly to executing concurrent programs  Replay a concurrent execution