Poirot – A Concurrency Sleuth Shaz Qadeer Research in Software Engineering Microsoft Research

Concurrent programming is difficult

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

… if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation Fatal error! IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

Concurrent programming is difficult Multiple loci of control resulting in non-local control flow Code difficult to understand and review

What about verification? Assertion-based modular reasoning becomes complicated due to non-local interactions – Floyd-Hoare morphs into Owicki-Gries Even with simple (finite) abstractions, the presence of concurrency makes the analysis computationally very expensive SequentialConcurrent Single Procedure P-timePSPACE-complete Multi Procedure P-timeUndecidable

What about testing? Number of executions = O( n nk ) Exponential in both n and k x = 1; … x = k; x = 1; … x = k; … Thread 1Thread n Scheduling nondeterminism Uncontrollable Unobservable Exponential

Concurrency is important More than ever before Increasing importance of communicating systems – networked devices – cyber-physical systems Distributed programs running on the cloud – EC2, Azure, AppEngine, … Parallel programs running on multicores and GPUs – TBB, TPL, CUDA, AMP, …

Concurrency testing with CHESS Deterministic scheduling – make scheduling choices observable and controllable Search prioritization – combating the combinatorial explosion of possible schedules

Deterministic scheduling Kernel: Threads, Scheduler, Synchronization Objects While(not done){ TestScenario() } TestScenario(){ … } Program CHESS Win32 API Tester Provides a Test Scenario CHESS runs the scenario in a loop Each run is a different interleaving Each run is repeatable

Search prioritization (I) Given p ≥ 0, generate all schedules with up to p preemptions Pseudo-polynomial number of schedules – polynomial in preemption bound and schedule points – exponential in number of threads Many bugs with fewer than 2 preemptions Simple error traces for easier debugging

Search prioritization (II) Given p ≥ 0 and deterministic schedulers S 0, …, S p-1, schedule according to S 0, …, S p-1 in sequence moving from one to next nondeterministically – e.g., round-robin non-preemptive scheduling with p different round-robin orders Polynomial number of schedules Testers can innovate by designing domain- specific deterministic schedulers

CHESS is available Used internally by Microsoft product groups and externally by Microsoft customers Binary and source code available at: –

Limitations of CHESS Exposing and gaining control of scheduling choices is difficult – most implementation effort and user frustration due to this problem Testing components that interact extensively with the environment is difficult Input coverage is not addressed

Static program exploration with Poirot Symbolic instead of concrete execution C: Source code for software component E: Model for environment and scheduler Explore behaviors of C+E – for all symbolic inputs – for all scheduling choices

Disk AsyncRead(…) { } DiskReader(…) { } DiskReader(…) { } headtail Request queueIn-memory cache cache cacheSize Demo: Asynchronous File I/O

Poirot architecture Trace Viewer Concurrent.NET Program Concurrent Boogie Program Coverage Report.NET  Boogie Concurrent C Program C  Boogie Corral

Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

Abstraction Set of global variables G Set of tracked variables T Drop writes to variables in G-T Replace reads to variables in G-T with nondeterministic values

Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

Refinement Path p – feasible if only variables in T are tracked – infeasible if all variables in G are tracked Expand tracked set T to U such that p infeasible while tracking only variables in U Naïve algorithm: linear scan of G-T New divide-and-conquer algorithm – best case log(|G-T|) – worst case 2*|G-T|

Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

Sequentialization (I) Given a concurrent program P, construct a sequential program Q such that Q  P Drop each occurrence of async-call Convert each occurrence of async-call to call

Sequentialization (II) Given a concurrent program P, construct a family of programs Q i such that – Q 0  Q 1  Q 2  …  P –  i Q i = P Even better if interesting behaviors of P manifest in Q i for low values of i

Context-bounding Under-approximation parameterized by K ≥ 0 – executions in which each thread gets at most K contexts to execute As K  , we get all behaviors Can we create sequentializations for context- bounding?

Sequentializing context switches Shared Memory T1T1 T2T2 Local Memory Execution: T1T1 T2T2 T1T1 T2T2 T1T1 T1T1 T2T2 T1T1 (s 1, l 1 )(s 2, l 2 ) s2s2 l2l2

Guess and verify T1T1 T2T2 T1T1 (s 1, l 1 ) (s 2, l 2 )(s 3, l 2 ) Guess the effect of T 2 Verify the guess Make copies of global variables Source-to-source translation – linear in program size and K Generalizes to dynamically-created threads

Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

Stratified search main … Call tree given recursion bound r VC(T) assert no bug Summaries(L) T L assert no bug VC(p) Convert loops to recursive calls

Poirot status Medium-sized C programs – up to 20K low-level systems code – reports precise traces at scale Small.NET programs – bytecode to Boogie translator in progress Try: Download available

Why bounded search? Data: Boolean, Integers, Arrays Control: Sequencing, Choice, Iteration, Call, Async-Call Sequencing Choice NP-complete Sequencing Choice Iteration Call Async-call Undecidable Sequencing Choice Iteration Call Async-call + bound Decidable PSPACE-hard Advances in SAT/SMT-solvers have made this problem tractable HAVOC verifier deployed for security analysis in Windows/IE Rationale: It is better to fail at the simpler problem!

Poirot collaborators Akash Lal, MSR Bangalore Shuvendu Lahiri, MSR Redmond

Questions