CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George
Outline Introduction Using CHESS Capturing nondeterminism Exploring nondeterminism Evaluation Related work
Introduction Concurrent programming is difficult Testing concurrent program is difficult Famous concurrency bug: “Heisenbugs” Most strategy used in finding concurrency bugs Stress testing
Introduction What is “Concurrency testing” Concurrency scenario testing Model checking techniques Three key challengers by applying model checking Perturbation problem Rich and complex concurrency APIs in system State-space explosion.
Introduction What is “CHESS” Concurrency scenario testing Model checking techniques Scale to large concurrent program Reproduce an erroneous execution Been integrated into the test framework inside Microsoft
Introduction How does CHESS solve the previous key problems? Testing methodology of CHESS reduce the perturbation Only one perturbation in CHESS Wrappers to provides enough hooks Variety of techniques to address the state-explosion
Contributions of this paper The first system for integrating model checking into concurrent systems and test frameworks with minimal perturbation; Techniques for systematic exploration of systems code for fine-grained concurrency with shared memory and multithreading;
Contributions of this paper Validation of the CHESS tool and its accompany testing and wrapper methodology on three different platforms; A substantial number of previously unknown bugs, even in well-tested systems; The ability to consistently reproduce crashing bugs with unknown cause.
Using CHESS Traditional testing methodology Disadvantages of this testing methodology
Using CHESS CHESS architecture
Using CHESS Testing methodology in CHESS
Using CHESS To guarantee the previous advantages, CHESS has to control the scheduling of the tasks Two ways to control it Modify the scheduler Severely limited the deployment
Using CHESS Testing methodology in CHESS
Using CHESS Two assumption of about the testing program Quiescence Definition of “Quiescence State” How does CHESS handle “Quiescence State”
Capturing nondeterminism Tasks in CHESS Threads Timers Asynchronous I/Os Wrappers in CHESS Goal of wrappers
Capturing nondeterminism Design of Wrappers in CHESS
Capturing nondeterminism Synchronization wrappers
Capturing nondeterminism Synchronization wrappers
Capturing nondeterminism Hooking the wrappers use various mechanisms to dynamically intercept calls to the real API functions and forward them to the wrappers. Programs in this paper Win32 .NET Singularity
Exploring nondeterminism How CHESS systematically drives the test Basic scheduler operation Allows only one thread to execute at a time Repeatedly executes the same test driving each iteration of the test through a different schedule Three phases in each iteration
Exploring nondeterminism Replay phase Replays a sequence of scheduling choices from a trace file Record phase Schedules a thread till the thread yields the processor Search phase Uses the enabled information at each schedule point to determine the scheduler for the next iteration
Exploring nondeterminism Imperfect replay Not rely on perfect replay capability Common sources of nondeterminism Lazy-initialization Interference from environment Nondeterministic calls
Exploring nondeterminism Ensuring fair schedules Can not enumerate all fair schedules State-explosion Definition of State-explosion Inserting preemptions Capturing states
Evaluation Systems on which CHESS has been run on
Evaluation Test scenarios and findings
Evaluation Validation CHESS against stress-testing Common objection of the CHESS Failure and bug in CHESS Succeeded in reproducing every stress-test failure
Evaluation Description of two bugs PLINQ bug
Evaluation Description of two bugs Singularity bug
Related work Repeatable deterministic testing Systematic generation of thread schedules Applying state exploration directly to executing concurrent programs Replay a concurrent execution