1. CHESS : Systematic Testing of Concurrent Programs
Madan Musuvathi
Shaz Qadeer
Microsoft Research

2. Testing multithreaded programs is HARD
Specific thread interleavings expose subtle errors
Testing often misses these errors
Even when found, errors are hard to debug
No repeatable trace
Source of the bug is far away from where it manifests

3. Concurrency is a real problem
Windows 2000 hot fixes
Concurrency errors most common defects among “detectable errors”
Incorrect synchronization and protocol errors most common defects among all coding errors
Windows Server 2003 late cycle defects
Synchronization errors second in the list, next to buffer overruns
Race conditions can result in security exploits

4. Current practice Concurrency testing == Stress testing
Example: testing a concurrent queue
Create 100 threads performing queue operations
Run for days/weeks
Pepper the code with sleep ( random() )
Stress increases the likelihood of rare interleavings
Makes any error found hard to debug
The way people do concurrency testing is some form stress testing

5. CHESS: Unit testing for concurrency
Example: testing a concurrent queue
Create 1 reader thread and 1 writer thread
Exhaustively try all thread interleavings
Run the test repeatedly on a specialized scheduler
Explore a different thread interleaving each time
Use model checking techniques to avoid redundancy
Check for assertions and deadlocks in every run
The error-trace is repeatable
we believe that we should have unit-testing for concurrency for precisely the same reasons we have unit-testing for sequential programs.

6. Systematic Stress Testing Using CHESS
Program
Tester Provides a Test Scenario
While(not done) {
TestScenario() }
CHESS
TestScenario() { … }
CHESS runs the scenario in a loop
Every run takes a different interleaving
Every run is repeatable
Win32 API
T: In order for this to work, we have to design TestScenario in a particular way
Kernel:
Threads, Scheduler,
Synchronization Objects

7. Conditions on Test Scenario
Test scenario should terminate in all interleavings
Test scenario should be idempotent
Free all resources (handles, memory, …)
Clear the hardware state
Key observation:
Existing stress tests already have these properties
Because they repeatedly run for ever
In fact, we got the core idea when we were talking with test-lead for networking
T: The tester should have a slightly different mindset

8. Perturb the System as Little as Possible
Run the system as is
On the actual OS, hardware
Using system threads, synchronization
Program
While(not done){
TestScenario() }
CHESS
TestScenario(){ … }
Detour Win32 API calls
To control and introduce nondeterminism
Win32 API
Advantages
Avoid reporting false errors
Easy to add to existing test frameworks
Use existing debuggers
Kernel:
Threads, Scheduler,
Synchronization Objects

9. Implementation details
Handle all the Win32 synchronization mechanisms
Critical sections, locks, semaphores, events,…
Threadpools
Asynchronous procedure calls
Timers
IO Completions
No modification to the kernel scheduler / Win32 library
CHESS drives the system along a desired by interleaving by ‘hijacking’ the scheduler

10. Controlling the Scheduling Nondeterminism
Nondeterministic choices for the scheduler
Determine when to context switch
On context switch, pick the next runnable thread to run
On resource release, wake up one of the waiting threads
Hijack these choices from the scheduler
Ensure at most one thread is runnable
No thread is waiting on a resource
At chosen schedule points, block the current thread while waking the next thread
Emulate program execution on a uniprocessor with context switches only at synchronization points

11. Partial-order reduction
Many thread interleavings are equivalent
Accesses to separate memory locations by different threads can be reordered
Avoid exploring equivalent thread interleavings
T1: x := 1
T2: y := 2
T2: y := 2
T1: x := 1

12. Partial-order reduction in CHESS
Algorithm:
Assume the program is data-race free
Context switch only at synchronization points
Check for data-races in each execution
Theorem:
If the algorithm terminates without reporting races,
then the program has no assertion failures

13. Executions on Multi-cores
CHESS checks for data-races
If a Test Scenario manifests a bug on a multi-core machine, then CHESS will
Either report a data-race
Or the bug
CHESS systematically enumerates all sequentially consistent executions
Any data-race free multi-core execution is equivalent to a sequentially consistent execution

14. State space explosion Thread 1 Thread 2 0,0 1,0 2,0 1,1 2,0 1,0 2,2
y = 1;
x = 2;
y = 2;
0,0
1,0
2,0
x = 1;
y = 1;
1,1
2,0
1,0
2,2
x = 2;
2,1
2,1
2,2
1,1
1,2
1,2
y = 2;
2,2
2,2
2,1
1,2
1,1
1,1

15. State space explosion … Number of executions = O( nnk )
Thread 1
Thread n
Number of executions
= O( nnk )
Exponential in both n and k
Typically: n < 10 k > 100
Limits scalability to large programs (large k)
x = 1; …
y = 1;
x = 2; …
y = 2; …
k steps
each
n threads

16. Bounding execution depth
Works very well for message-passing programs
Limit the number of message exchanges
Message processing code executed atomically
Can go ‘deep’ in the state space
Does not work for multithreaded programs
Even toy programs can have large number of steps (shared-variable accesses)

17. Iterative context bounding
Prioritize executions with small number of preemptions
Two kinds of context switches:
Preemptions – forced by the scheduler
e.g. Time-slice expiration
Non-preemptions – a thread voluntarily yields
e.g. Blocking on an unavailable lock, thread end
Thread 1
Thread 2
x = 1;
if (p != 0) {
x = 1;
if (p != 0) {
x = p->f; }
p = 0;
preemption
x = p->f; }
non-preemption

18. Iterative context-bounding algorithm
The scheduler has a budget of c preemptions
Nondeterministically choose the preemption points
Resort to non-preemptive scheduling after c preemptions
Once all executions explored with c preemptions
Try with c+1 preemptions
Iterative context-bounding has desirable properties
Property 0: Easy to implement

19. Property 1: Polynomial state space
Terminating program with fixed inputs and deterministic threads
n threads, k steps each, c preemptions
Number of executions <= nkCc . (n+c)!
= O( (n2k)c. n! )
Exponential in n and c, but not in k
Thread 1
Thread 2
Choose c preemption points
x = 1; …
x = 1; …
y = 1;
x = 2; …
y = 2;
x = 2; …
Permute n+c atomic blocks … …
y = 1;
y = 2;

20. Property 2: Deep exploration possible with small bounds
A context-bounded execution has unbounded depth
a thread may execute unbounded number of steps within each context
Event a context-bound of zero yields complete terminating executions

21. Property 3: Finds the ‘simplest’ error trace
Finds smallest number of preemptions to the error
Number of preemptions better metric of error complexity than execution length

22. Property 4: Coverage metric
If search terminates with context-bound of c, then any remaining error must require at least c+1 preemptions
Intuitive estimate for
The complexity of the bugs remaining in the program
The chance of their occurrence in practice

23. Property 5: Lots of bugs with small number of preemptions
A non-blocking implementation of the work- stealing queue algorithm
bounded circular buffer accessed concurrently by readers and stealers
Developer provided
test harness
three buggy variations of the program
Each bug found with at most 2 preemptions
executions with 35 preemptions are possible!

24. Context-bounding + Partial-order reduction
Algorithm:
Assume the program is data-race free
Context switch only at synchronization points
Explore executions with c preemptions
Check for data-races in each execution
Theorem:
If the algorithm terminates without reporting races,
Then the program has no assertion failures reachable with c preemptions
Requires that a thread can block only at synchronization points
Proof (Musuvathi-Q, PLDI 2007)

25. Bugs found Program KLOC Max Num Threads Bugs Reachable with
Preemption Count 1 2 3
Total
Bluetooth
0.4
Work-Stealing Queue
1.3
Transaction Manager
7.0
APE
18.9 4 -
Dryad Channels
16.0 5 7

26. // Function called by the main thread
void TestChannel(WorkQueue* workQueue, ...) {
// Creating a channel
// allocates worker threads
RChannelReader* channel =
new RChannelReaderImpl(..., workQueue);
// ... do work here
channel->Close();
// wrong assumption that channel->Close()
// waits for worker threads to be finished
delete channel;
// BUG: deleting the channel when
// worker threads still have a valid
// reference to the channel }
// Function called by a worker thread
// of RChannelReaderImpl
void RChannelReaderImpl::
AlertApplication(RChannelItem* item) {
// Notify Application
// XXX: Preempt here for the bug
EnterCriticalSection(&m_baseCS);
// process before exit
LeaveCriticalSection(&m_baseCS); }

27. // Function called by the main thread
void TestChannel(WorkQueue* workQueue, ...) {
// Creating a channel
// allocates worker threads
RChannelReader* channel =
new RChannelReaderImpl(..., workQueue);
// ... do work here
channel->Close();
// wrong assumption that channel->Close()
// waits for worker threads to be finished
delete channel;
// BUG: deleting the channel when
// worker threads still have a valid
// reference to the channel }
// Function called by a worker thread
// of RChannelReaderImpl
void RChannelReaderImpl::
AlertApplication(RChannelItem* item) {
// Notify Application
// XXX: Preempt here for the bug
EnterCriticalSection(&m_baseCS);
// process before exit
LeaveCriticalSection(&m_baseCS); }

28. // Function called by the main thread
void TestChannel(WorkQueue* workQueue, ...) {
// Creating a channel
// allocates worker threads
RChannelReader* channel =
new RChannelReaderImpl(..., workQueue);
// ... do work here
channel->Close();
// wrong assumption that channel->Close()
// waits for worker threads to be finished
delete channel;
// BUG: deleting the channel when
// worker threads still have a valid
// reference to the channel }
// Function called by a worker thread
// of RChannelReaderImpl
void RChannelReaderImpl::
AlertApplication(RChannelItem* item) {
// Notify Application
// XXX: Preempt here for the bug
EnterCriticalSection(&m_baseCS);
// process before exit
LeaveCriticalSection(&m_baseCS); }

29. // Function called by the main thread
void TestChannel(WorkQueue* workQueue, ...) {
// Creating a channel
// allocates worker threads
RChannelReader* channel =
new RChannelReaderImpl(..., workQueue);
// ... do work here
channel->Close();
// wrong assumption that channel->Close()
// waits for worker threads to be finished
delete channel;
// BUG: deleting the channel when
// worker threads still have a valid
// reference to the channel }
// Function called by a worker thread
// of RChannelReaderImpl
void RChannelReaderImpl::
AlertApplication(RChannelItem* item) {
// Notify Application
// XXX: Preempt here for the bug
EnterCriticalSection(&m_baseCS);
// process before exit
LeaveCriticalSection(&m_baseCS); }

30. // Function called by the main thread
void TestChannel(WorkQueue* workQueue, ...) {
// Creating a channel
// allocates worker threads
RChannelReader* channel =
new RChannelReaderImpl(..., workQueue);
// ... do work here
channel->Close();
// wrong assumption that channel->Close()
// waits for worker threads to be finished
delete channel;
// BUG: deleting the channel when
// worker threads still have a valid
// reference to the channel }
// Function called by a worker thread
// of RChannelReaderImpl
void RChannelReaderImpl::
AlertApplication(RChannelItem* item) {
// Notify Application
// XXX: Preempt here for the bug
EnterCriticalSection(&m_baseCS);
// process before exit
LeaveCriticalSection(&m_baseCS); }

31. Facts about Dryad error trace
Long error trace but requires only one preemption
Depth-bounding cannot find it without a lot of luck
The error trace has 6 non-preempting context switches
It is important to leave unbounded the number of non- preempting context switches
This (and the other 6 errors) in Dryad remained in spite of careful regression testing and months of production use

32. Bugs found Program KLOC Max Num Threads Bugs Reachable with
Preemption Count 1 2 3
Total
Bluetooth
0.4
Work-Stealing Queue
1.3
Transaction Manager
7.0
APE
18.9 4 -
Dryad Channels
16.0 5 7

33. Coverage vs. Context-bound

34. Dryad (coverage vs. time)

35. Current CHESS applications (work in progress)
Dryad (library for distributed dataflow programming)
Singularity/Midori (OS in managed code)
User-mode drivers
Cosmos (distributed file system)
SQL database

36. Conclusion Concurrency is important
Building robust concurrent software is still a challenge
Lack of debugging and testing tools
CHESS: Concurrency unit-testing
Exhaustively try all interleavings
Attempt to seamlessly integrate with existing test frameworks
Provide replay capability
Iterative context-bounding algorithm key to the design