Presenter: Godmar Back CS 5204 Eraser: A Dynamic Data Race Detector for Multithreaded Programs Savage et al (SOSP 1997, ACM TOCS) Presenter: Godmar Back

16 Years Back Parallel Computing Wave Emergence of low-cost SMP Several massively parallel designs Emergence of low-cost SMP OS Support for multi-threaded programs Multi-threaded OS kernels Emergence of the Internet CS 5204 Fall 2013

Why Threads Are A Bad Idea 1996 Usenix talk by John Ousterhout: “Why threads are a bad idea” Threads are hard to program (synchronization error-prone, prone to deadlock, convoying, etc.); experts only Threads can make modularization difficult Thread implementations are hard to make fast Threads aren’t well-supported (as of 1996) (Ousterhout’s) Conclusion: use threads only when their power is needed - for true CPU concurrency on an SMP – else use single-threaded event-based model CS 5204 Fall 2013

How to write save programs Build safety into the language! Hoare’s Monitors (see separate slide set) CS 5204 Fall 2013

Concurrency Errors vs Data Races Unintended sharing Atomicity violations Order violations Deadlocks/Livelocks Source: Freund et al Likely result in data race CS 5204 Fall 2013

Data Race Conflicting access to the same memory location, at least one of them being a write Read/write Write/read Write/write A race is a pair of conflicting accesses that happens concurrently CS 5204 Fall 2013

“Happens concurrently”? A & B are said to happen concurrently if, in a sequentially consistent execution, they could have occur in either order. That is, there exist sequentially consistent executions in which they could have occurred one after the other. Sequential Consistency Updates are seen in program order by each thread All updates to shared memory are seen in same order by all threads CS 5204 Fall 2013

Race Detection Approaches Happens-Before Relationship Lamport 1978 Distributed Systems Idea First applied by Dinning and Schonberg 1991 CS 5204 Fall 2013

HB in Distributed Systems a b c d f P1 [1 0 0] [2 0 0] [3 0 0] [4 3 0] [5 5 3] [2 0 0] [3 0 0] [2 3 0] [2 5 3] g i j k l m P2 [0 1 0] [2 2 0] [2 3 0] [2 4 3] [2 5 3] [3 6 5] [0 1 0] [0 1 3] [3 1 5] n o p q r s P3 [0 0 1] [0 1 2] [0 1 3] [3 1 4] [3 1 5] [3 1 6] a  f b  s c  m [1 0 0] < [5 5 3] [2 0 0] < [3 1 6] [3 0 0] < [3 6 5] CS 5204 Fall 2013

Vector Clocks a b c d f P1 [1 0 0] [2 0 0] [3 0 0] [4 3 0] [5 5 3] [2 3 0] [2 5 3] g i j k l m P2 [0 1 0] [2 2 0] [2 3 0] [2 4 3] [2 5 3] [3 6 5] [0 1 0] [0 1 3] [3 1 5] n o p q r s P3 [0 0 1] [0 1 2] [0 1 3] [3 1 4] [3 1 5] [3 1 6] d || s q || i k || r [4 3 0] < [3 1 6] [3 1 4] < [2 2 0] [2 4 3] < [3 1 5] CS 5204 Fall 2013

Vector Clocks Vector timestamps: Each node keeps track of logical time of other nodes (as far as it’s seen messages from them) in Vi[i] Send vector timestamp vt along with each message Reconcile vectors timestamp with own vectors upon receipt using MAX(vt[k], Vi [k]) for all k CS 5204 Fall 2013

HB for Race Detection Happens-Before relationships between synchronization and thread creation/join events Code leading to thread create happens before thread body Thread body happens before code subsequent to join() Unlock() happens before lock() CS 5204 Fall 2013

Happens Before Race Detection Monitor thread’s accesses to shared variables Find concurrent events (for instance, using vector clocks) Output warning Sound: never reports a false race Does not require races to manifest in execution, but may miss some races depending on execution CS 5204 Fall 2013

HB relationship is spurious – ‘mu’ doesn’t protect y Missed Race in HB Thread 1 y := y+1; Lock(mu); v:= v+1; Unlock(mu); HB relationship is spurious – ‘mu’ doesn’t protect y Thread 2 Lock(mu); v:= v+1; Unlock(mu); y := y+1; CS 5204 Fall 2013

Cost of HB (1997) Vector clocks Need one per memory location Of size O(t) where t is number of threads ever created in system; all operations on them are of size t. CS 5204 Fall 2013

Eraser Intuition HB too expensive Observation: properly written programs follow lock discipline Lock l protects variable v. If v is ever accessed while l isn’t held -> race How to find out which l protects which v? Infer it from the program CS 5204 Fall 2013

Lock Set Algorithm (Simple Version) Let locks_held(t) be the set of locks held by thread t For each shared memory location v, initialize C(v) to the set of all locks On each access to v by thread t, Set C(v) := C(v) ∩ locks_held(t) If C(v) := {}, then issue a warning CS 5204 Fall 2013

Example Program locks_held C(v) int v; {} {mu1, mu2} lock(mu1); {mu1} v := v + 1; unlock(mu1); lock(mu2); {mu2} {} Warning! unlock(mu2); CS 5204 Fall 2013

Refinement Simple version flags false positives for Variable initialization without locks held Read sharing Read-write locking mechanisms Handle 1., 2. by introducing states Handle 3. by changing algorithm CS 5204 Fall 2013

Refinement State Machine Virgin wr wr new thread Exclusive Shared Modified rd/wr first thread rd new thread Shared wr rd CS 5204 Fall 2013

Lock Set Algorithm (Extended) Let locks_held(t) be the set of locks held in any mode by thread t Let write_locks_held(t) be the set of locks held in write mode by thread t For each shared memory location v, initialize C(v) to the set of all locks On each read of v by thread t, Set C(v) := C(v) ∩ locks_held(t) If C(v) = {}, then issue a warning On each write of v by thread t, Set C(v) := C(v) ∩ write_locks_held(t) CS 5204 Fall 2013

Implementation Hashtable of locksets, indexed by integer Shadow memory Variable -> (state, lockset index) Fast access using simple offset CS 5204 Fall 2013

Shadow Memory CS 5204 Fall 2013

Missing Races Q: When is this race missed? int[] shared = new int[1]; Thread t = new Thread() { public void synchronized run() { shared[0] = shared[0] + 1; ... } }; shared[0] = 512; t.start(); shared[0] = shared[0] + 256; Q: When is this race missed? CS 5204 Fall 2013

Unhandled Cases Memory Reuse “Benign” races Not really benign, see You Don't Know Jack about Shared Variables or Memory Models  [Boehm/Adve 2011] If (fptr == NULL) { lock(fptr_mu); if (fptr == NULL) { fptr = open(filename); } unlock(fptr_mu); CS 5204 Fall 2013

Race Detection Since Eraser Combined HB + LockSet DJIT+ Google Thread Sanitizer Helgrind+ RaceTrack [Yu 2005] Goldilocks Helgrind <= 3.3 Pure HB Helgrind 3.4 and later Intel Thread Checker Fasttrack CS 5204 Fall 2013

Dynamic Data-Race Detection FastTrack [Flanagan-Freund 09] Vector Clocks [M 88] Goldilocks [EQT 07] DJIT+ [ISZ 99,PS 03] TRaDe [CB 01] ... Happens Before [Lamport 78] Eraser [SBN+ 97] RaceTrack [YRC 05] MultiRace [PS 03] Hybrid Race Detector [OC 03] ... Precision Design Criteria: sound & complete (find at least 1st data race on each var) efficient Insight: HB relation is a partial order But all accesses to a var are almost always totally ordered Barriers [PS 03] Initialization [vPG 01] ... Cost Source: Freund et al

Source: Flanagan 2009 CS 5204 Fall 2013

Source: Flanagan 2009 CS 5204 Fall 2013

In this paper, we focus on online dynamic race detectors, which generally fall into two categories depending on whether they report false alarms. Precise race detectors never produce false alarms.  (…) Precise dynamic race detectors do not reason about all possible traces, however, and may not identify races that occur only when other code paths are taken. (…) A variety of alternative imprecise race detectors have been developed, which may provide improved performance (and sometimes better coverage), but which report false alarms on some race-free programs. For example, Eraser's LockSet algorithm…. Views on precision Unfortunately, tools based on happens-before have two significant drawbacks. First, they are difficult to implement efficiently because they require per-thread information about concurrent accesses to each shared-memory location. More importantly, the effectiveness of tools based on happens before is highly dependent on the interleaving produced by the scheduler. While Eraser is a testing tool and therefore cannot guarantee that a program is free from races, it can detect more races than tools based on happens-before. Savage et al 1997 Flanagan & Freund 2010 CS 5204 Fall 2013

Aside on Helgrind CS 5204 Fall 2013

Project Idea Implement Fasttrack-like algorithm for node.js based on Boris Petrov, Martin Vechev, Manu Sridharan, and Julian Dolby. 2012. Race detection for web applications. In Proceedings of the 33rd ACM SIGPLAN conference on Programming Language Design and Implementation (PLDI '12). ACM, New York, NY, USA, 251-262. DOI=10.1145/2254064.2254095 http:doi.acm.org/10.1145/2254064.2254095 http://researcher.watson.ibm.com/researcher/files/us-msridhar/pldi12-wr.pdf CS 5204 Fall 2013

Conclusion Eraser was influential paper that pioneered Lockset idea Influenced a decade of development and refinement of race detectors CS 5204 Fall 2013