Taming Concurrency: A Program Verification Perspective Shaz Qadeer Microsoft Research

Reliable concurrent software? Correctness problem –does program behave correctly for all inputs and all interleavings? Bugs due to concurrency are insidious –non-deterministic, timing dependent –data corruption, crashes –difficult to detect, reproduce, eliminate

P satisfies S Undecidable problem! Why is verification of concurrent programs more difficult than verification of sequential programs?

P satisfies S Assertions: Provide contracts to decompose problem into a collection of decidable problems pre-condition and post-condition for each procedure loop invariant for each loop Abstractions: Provide an abstraction of the program for which verification is decidable Finite-state systems finite automata Infinite-state systems pushdown automata, counter automata, timed automata, Petri nets

Interference int t; t := x; t := t + 1; x := t; Correct pre x = 0; post x = 1;

Interference int t; t := x; t := t + 1; x := t; Incorrect! pre x = 0; post x = 2; int t; t := x; t := t + 1; x := t; AB

Controlling interference int t; acquire(l); t := x; t := t + 1; x := t; release(l); Correct! pre x = 0; post x = 2; int t; acquire(l); t := x; t := t + 1; x := t; release(l); AB

Interference makes program verification difficult Annotation explosion with the assertion approach State explosion with the abstraction approach

Annotation explosion For sequential programs –assertion for each loop –assertion refers only to variables in scope For concurrent programs –assertion for each control location –assertion may need to refer to private state of other threads

State explosion SequentialConcurrent Finite-state systems Pushdown systems P.G (P.G) 3 P = # of program locations G = # of global states PSPACE complete Undecidable n = # of threads

Taming interference Atomicity for combating annotation explosion Interference-bounding for combating state explosion

Critical_Section l; /*# guarded_by l */ int balance; Bank account /*# atomic */ void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l); } /*# atomic */ int read( ) { int r; acquire(l); r = balance; release(l); return r; } /*# atomic */ void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l); }

Atomicity violation in StringBuffer (Flanagan-Q, PLDI 03) public final class StringBuffer { private int count; private char[ ] value;. public synchronized StringBuffer append (StringBuffer sb) { if (sb == null) sb = NULL; int len = sb.length( ); int newcount = count + len; if (newcount > value.length) expandCapacity(newcount); sb.getChars(0, len, value, count); //use of stale len !! count = newcount; return this; } public synchronized int length( ) { return count; } public synchronized void getChars(...) {... } }

Inadequate atomicity is a good predictor of undesirable interference! compared to data races Numerous tools for detecting atomicity violations static (Type systems, ESPC, QED) dynamic (Atomizer, AVIO, AtomAid, Velodrome, …) Significant effort cutting across communities architecture and operating systems programming languages and compilers testing and formal methods

Definition of atomicity deposit is atomic if for every non-serialized execution, there is a serialized execution with the same behavior         acq(l)r=balbal=r+nrel(l)xyz  Serialized execution of deposit         acq(l)r=balbal=r+nrel(l)xyz         acq(l)r=balbal=r+nrel(l)xyz  Non-serialized executions of deposit

S0S0 S1S1 S2S2 S3S3 S4S4 S7S7 S6S6 S5S5 acq(l)r=balbal=r+nrel(l)xyz S0S0 T1T1 T2T2 T3T3 S4S4 S7S7 T6T6 S5S5 r=balbal=r+nrel(l)xyz acq(l) S0S0 T1T1 S2S2 T3T3 S4S4 S7S7 S6S6 S5S5 yr=balbal=r+nrel(l)xacq(l)z S0S0 T1T1 T2T2 T3T3 S4S4 S7S7 S6S6 S5S5 r=balbal=r+nrel(l)xyz S0S0 S1S1 S2S2 T3T3 S4S4 S7S7 S6S6 S5S5 acq(l)r=balbal=r+nrel(l)yz x Reduction (Lipton, CACM 75)

 B: both right + left commutes –variable access holding lock  A: atomic action, non-commuting –access unprotected variable Four atomicities  L: left commutes –lock release  R: right commutes –lock acquire

S0S0. S5S5 R* A L*xY... S0S0. S5S5 R* A L* x Y... Sequential composition  Theorem: Sequence (R+B)*;(A+  ); (L+B)* is atomic CCCCCC CCCAAA CCCLLL CARARR CARLBB CARLB; ; A R;A;L ; R;A;L C A ; A R; B ; A; L A R

Critical_Section l; /*# guarded_by l */ int balance; Bank account /*# atomic */ void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l); } /*# atomic */ int read( ) { int r; acquire(l); r = balance; release(l); return r; } /*# atomic */ void withdraw(int x) { int r = read(); acquire(l); balance = r – x; release(l); } RBBLRBBL A RBLBRBLB A ARBLARBL C Incorrect!

Critical_Section l; /*# guarded_by l */ int balance; Bank account /*# atomic */ void deposit (int x) { acquire(l); int r = balance; balance = r + x; release(l); } /*# atomic */ int read( ) { int r; acquire(l); r = balance; release(l); return r; } /*# atomic */ void withdraw(int x) { acquire(l); int r = balance; balance = r – x; release(l); } RBBLRBBL A RBLBRBLB A RBBLRBBL A Correct!

Taming interference Atomicity for combating annotation explosion Interference-bounding for combating state explosion

Interference bounding An approach to the state-explosion problem Explore all executions with a bounded amount of interference –increase the interference bound iteratively Good idea if low interference bound –can be exploited by algorithms –is enough to expose bugs

Context-bounded verification [Wu-Q, PLDI 04] [Rehof-Q, TACAS 05]           Context Context switch Interference proportional to number of context switches Explore all executions with few context switches Unbounded computation within each context –Different from bounded model checking

Context-bounding today CHESS [Musuvathi-Q, PLDI 07] JMoped [Suwimonteerabuth et al., SPIN 08] SPIN for multithreaded C programs [Zaks- Joshi, SPIN 08] CBA [Lal-Reps, CAV 08] Static Driver Verifier

Testing concurrent programs is HARD Bugs hidden in rare thread interleavings Today, concurrency testing = stress testing –Poor coverage of interleavings –Unpredictable coverage results in “Heisenbugs” The mark of reliability of the system still remains its ability to withstand stress

CHESS in a nutshell Replace the OS scheduler with a demonic scheduler Systematically explore all scheduling choices Concurrent Program Win32 API Kernel Scheduler Kernel Scheduler Demonic Scheduler Demonic Scheduler

CHESS architecture Kernel: Threads, Scheduler, Synchronization Objects Kernel: Threads, Scheduler, Synchronization Objects While(not done) { TestScenario() } While(not done) { TestScenario() } TestScenario() { … } Program CHESS CHESS runs the scenario in a loop Each run takes different interleaving Each run is repeatable Win32 API Intercept synchronization and threading calls To control and introduce nondeterminism Detect Assertion violations Deadlocks Dataraces Livelocks

CHESS methodology generalizes CHESS works for –Unmanaged programs (written in C, C++) –Managed programs (written in C#,…) –Singularity applications With appropriate wrappers, can work for Java, Linux applications.NET Program.NET Program.NET CLR CHESS Win32 Program Win32 Program Win32 / OS CHESS Singularity Program Singularity Program Singularity CHESS

CHESS: Systematic testing for concurrency Kernel: Threads, Scheduler, Synchronization Objects While(not done){ TestScenario() } TestScenario(){ … } Program CHESS Win32 API CHESS runs the scenario in a loop Each run is a different interleaving Each run is repeatable

x = 1; … x = k; State-space explosion x = 1; … x = k; … n threads k steps each Number of executions = O( n nk ) Exponential in both n and k –Typically: n 100 Limits scalability to large programs Goal: Scale CHESS to large programs (large k) Thread 1Thread n

x = 1; if (p != 0) { x = p->f; } Preemption-bounding Prioritize executions with small number of preemptions Two kinds of context switches: Preemptions – forced by the scheduler e.g. Time-slice expiration Non-preemptions – a thread voluntarily yields e.g. Blocking on an unavailable lock, thread end x = p->f; } x = 1; if (p != 0) { p = 0; preemption non-preemption Thread 1Thread 2

Preemption-bounding in CHESS The scheduler has a budget of c preemptions –Nondeterministically choose the preemption points Resort to non-preemptive scheduling after c preemptions Once all executions explored with c preemptions –Try with c+1 preemptions

x = 1; … x = 1; … x = k; Property 1: Polynomial bound Terminating program with fixed inputs and deterministic threads –n threads, k steps each, c preemptions Number of executions <= nk C c. (n+c)! = O( (n 2 k) c. n! ) Exponential in n and c, but not in k x = 1; … x = k; x = 1; … x = k; … x = k; Choose c preemption points Permute n+c atomic blocks Thread 1Thread 2

Property 2: Simple error traces Finds smallest number of preemptions to the error Number of preemptions better metric of error complexity than execution length

Property 3: Coverage metric If search terminates with preemption- bound of c, then any remaining error must require at least c+1 preemptions Intuitive estimate for –The complexity of the bugs remaining in the program –The chance of their occurrence in practice

Property 4: Many bugs with 2 preemptions ProgramLines of code Bugs Work Stealing Q4K4 CDS6K1 CCR9K3 ConcRT16K4 Dryad18K7 APE19K4 STM20K2 TPL24K9 PLINQ24K1 Singularity175K2 37 (total) Acknowledgement: testers from PCP team

Coverage vs. Preemption-bound

CHESS status Being used by testers in many Microsoft product groups Demo and session at Professional Developer Conference 2008 CHESS for Win32 – CHESS for.NET –Coming soon

So far … Interference makes program verification difficult Taming interference –atomicity –interference-bounding

Whither next? Concurrent programs as a composition of modules implementing atomic actions Inteference-bounding for general concurrent systems Symbolic context-bounded verification

Structuring concurrent programs In addition to atomicity, we need –behavioral abstraction analogous to pre/post- conditions for sequential code fragments –intuitive and simple contract language A calculus of atomic actions (Elmas- Tasiran-Q, POPL 08) –QED verifier

Interference-bounding for general concurrent systems Task = unit-of-work Shared-memory systems –Task (usually) corresponds to a thread Message-passing systems –Task corresponds to a sequence of messages Need linguistic support for the task abstraction in software models and implementations

Context-bounded reachability is NP-complete UnboundedContext-bounded Finite-state systems Pushdown systems PSPACE complete NP-complete UndecidableNP-complete P = # of program locations G = # of global states n = # of threads c = # of context switches

Symbolic context-bounded verification The transition relations of tasks are usually encoded symbolically –reachability analysis is PSPACE-complete even for a single task Scalable tools for single symbolic task –Bebop, Moped, Getafix Can we extend these techniques to deal with multiple tasks? –Lal and Reps (TACAS 2008, CAV 2008) –Suwimonteerabuth et al. (SPIN 2008)

Questions?