1. A Randomized Dynamic Program Analysis for Detecting Real Deadlocks Koushik Sen CS 265

2. What is a Deadlock? An unintended condition in a shared-memory, multi- threaded program in which: – a set of threads blocks forever – because each thread in the set waits to acquire a lock being held by another thread in the set This work: ignore other causes (e.g., wait/notify) Example // thread t1 sync (l1) { sync (l2) { … } // thread t2 sync (l2) { sync (l1) { … } l1 t1 l2 t2

3. Motivation Today’s concurrent programs are rife with deadlocks – 6,500/198,000 (~ 3%) of bug reports in Sun’s bug database at http://bugs.sun.com are deadlocks Deadlocks are difficult to detect – Usually triggered non-deterministically, on specific thread schedules – Fail-stop behavior not guaranteed (some threads may be deadlocked while others continue to run) Fixing other concurrency bugs like races can introduce new deadlocks – Our past experience with reporting races: developers often ask for deadlock checker

4. Our Goal Build a deadlock detection tool that – Scales to very large programs – Finds deadlocks quickly – Has no false positives Shows a real execution exhibiting a deadlock

5. Related Work: Program Analysis Static program analysis (e.g., Engler & Ashcraft SOSP’03; Williams-Thies-Ernst ECOOP’05, Naik et al. ICSE’09) + Examines all possible program behavior -Often reports many false positives Type systems (e.g., Boyapati-Lee-Rinard OOPSLA’02) - Annotation burden often significant Model checking (e.g., SPIN, Verisoft, Java Pathfinder) - Does not currently scale beyond few KLOC - Not “directed” Dynamic program analysis (e.g. Goodlock Havelund et al., Agrawal et al.) + Usually reports lesser false positives - Has false negatives

6. Related Work: Testing Testing + Easy to deploy + Scales to large programs + No false positives -Same schedule gets tested many times  No effort to control thread scheduler - Often subtle thread schedules that exhibit deadlocks are not tested

7. In Summary Static and dynamic program analyses have false positives Testing is simple – No false positives – But, may miss subtle thread schedules that result in deadlocks

8. In Summary Static and dynamic program analyses have false positives Testing is simple – No false positives – But, may miss subtle thread schedules that result in deadlocks Can we leverage program analysis to make testing quickly find real deadlocks?

9. Our Approach Active Testing Phase 1: Use imprecise static or dynamic program analysis to find “abstract” states where a potential deadlock can happen Phase 2: “Direct” testing (or model checking) based on the “abstract” states obtained from phase 1

10. Example Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ }

11. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } Testing f1() f2() Lock(o2) Lock(o1) Unlock(o1) Unlock(o2)

12. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } Testing Lock(o1) Lock(o2) Unlock(o2) Unlock(o1) Lock(o2) Lock(o1) Unlock(o1) Unlock(o2) f1() f2()

13. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } Lock(o1) Lock(o2) Unlock(o2) Unlock(o1) Lock(o2) Lock(o1) Unlock(o1) Unlock(o2) f1() f2() No deadlock detected Testing

14. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } Deadlock Directed Testing Lock(o2) Lock(o1) Paused f1() f2()

15. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } Deadlock Directed Testing Lock(o2) Lock(o1) Paused f1() f2() Lock(o1) Lock(o2) Paused

16. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } Deadlock Directed Testing Lock(o2) Lock(o1) Paused f1() f2() Lock(o1) Lock(o2) Paused

17. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } Deadlock Directed Testing Lock(o2) Lock(o1) Paused f1() f2() Lock(o1) Lock(o2) Paused Deadlock detected !

18. Preempting threads How do we know where to pause a thread ?

19. Preempting threads How do we know where to pause a thread ? – Use existing static or dynamic analyses to find potential deadlock cycles Note that these analyses may report false deadlock cycles – Use “information” recorded for a deadlock cycle to decide where to pause a thread – We use a modified version of the Goodlock algorithm (iGoodlock) [Havelund et al, Agarwal et al]

20. iGoodlock We consider dynamic instances of the following statements c: Acquire(t,l) c: Release(t,l) c: Call(t,m) c: Release(t,m) c: o = new (t,o’,T)

21. iGoodlock We consider dynamic instances of the following statements c: Acquire(t,l) c: Release(t,l) c: Call(t,m) c: Release(t,m) c: o = new (t,o’,T)

22. Lock Dependency Relation D is a subset of (T X 2 L X L X C*) (t,L,l,C) is in the lock dependency relation if

23. Lock Dependency Relation D is a subset of (T X 2 L X L X C*) (t,L,l,C) is in the lock dependency relation if – thread t acquires lock l while holding the locks in the set L, and C is the sequence of labels of Acquire statements that were executed by t to acquire the locks in L ∪ {l}.

24. Lock Dependency Chain (t 1,L 1,l 1,C 1 ), (t 2,L 2,l 2,C 2 ), …, (t m,L m,l m,C m ) is a lock dependency chain if

25. Lock Dependency Chain (t 1,L 1,l 1,C 1 ), (t 2,L 2,l 2,C 2 ), …, (t m,L m,l m,C m ) is a lock dependency chain if for all distinct i, j ∈ [1, m], t i ≠ t j for all distinct i, j ∈ [1, m], l i ≠ l j for all i ∈ [1,m − 1], l i ∈ L i+1, for all distinct i,j ∈ [1,m], L i ∩ L j = ∅.

26. Lock Dependency Chain (t 1,L 1,l 1,C 1 ), (t 2,L 2,l 2,C 2 ), …, (t m,L m,l m,C m ) is a lock dependency chain if for all distinct i, j ∈ [1, m], t i ≠ t j for all distinct i, j ∈ [1, m], l i ≠ l j for all i ∈ [1,m − 1], l i ∈ L i+1, for all distinct i,j ∈ [1,m], L i ∩ L j = ∅. A chain is a deadlock if

27. Lock Dependency Chain (t 1,L 1,l 1,C 1 ), (t 2,L 2,l 2,C 2 ), …, (t m,L m,l m,C m ) is a lock dependency chain if for all distinct i, j ∈ [1, m], t i ≠ t j for all distinct i, j ∈ [1, m], l i ≠ l j for all i ∈ [1,m − 1], l i ∈ L i+1, for all distinct i,j ∈ [1,m], L i ∩ L j = ∅. A chain is a deadlock if l m ∈ L 1

28. Thread 1Thread 2 Thread1 foo(o1,o2,true) Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } Acquire(o1) Acquire(o2) Release(o2) Release(o1) Acquire(o2) Acquire(o1) Release(o1) Release(o2) Compute D on this trace f1() f2()

29. Computing lock dependency chains Compute D k where D 1 = D for each d ∈ D and τ ∈ D i if d, τ is a lock dependency chain and a deadlock cycle then report a potential deadlock if d, τ is a lock dependency chain and not a deadlock cycle – add d, τ to D i+1

30. Preempting threads T1, o1, T2, o2 are runtime objects How do we identify them across executions ?

31. Preempting threads T1, o1, T2, o2 are runtime objects How do we identify them across executions ? – Runtime addresses?

32. Preempting threads T1, o1, T2, o2 are runtime objects How do we identify them across executions ? – Runtime addresses? No, they change across executions !

33. Preempting threads T1, o1, T2, o2 are runtime objects How do we identify them across executions ? – Runtime addresses? No, they change across executions ! – Allocation sites where they were created ? Yes, but different objects created at the same allocation site will all be treated as the same object

34. Preempting threads T1, o1, T2, o2 are runtime objects How do we identify them across executions ? – Runtime addresses? No, they change across executions ! – Allocation sites where they were created ? Yes, but different objects created at the same allocation site will all be treated as the same object – Can we do better?

35. Abstractions Based on k-object sensitivity [Milanova et al] Based on light-weight execution indexing [Xin et al]

36. Abstractions 7 class A { 8 void bar() { 9 for(int j = 0; j < 5; j++) 10 Object l = new Object(); 11 } 12 } 1 class C { 2 void foo() { 3 A a = new A(); 4 a.bar(); 5} 6 } 13 main() { 14 C c = new C(); 15 for(int i = 0; i < 5; i++) 16 c.foo(); 17 }

37. Abstractions 7 class A { 8 void bar() { 9 for(int j = 0; j < 5; j++) 10 Object l = new Object(); 11 } 12 } 1 class C { 2 void foo() { 3 A a = new A(); 4 a.bar(); 5} 6 } 13 main() { 14 C c = new C(); 15 for(int i = 0; i < 5; i++) 16 c.foo(); 17 } Abstraction of last object created (k- object sensitivity) : [10,3, 14] Abstraction of last object created (execution indexing) : [(10,5),(4,1),(16,5)]

38. Implementation Implemented in a prototype tool called DEADLOCKFUZZER for Java Part of the CALFUZZER framework – Active testing [Sen PLDI 08, Park et al. FSE 08, Joshi et al. CAV 09] of concurrent programs Instrument Java bytecode to – observe events during execution – control scheduler

39. Evaluation Program Name Lines Of Code Average Runtime in milliseconds NormaliGoodlockDeadlockFuzzer cache4j3,8972,0453,409N.A. sor17,718163396N.A. hedc25,0241651,668N.A. jspider10,2524,6225,020N.A. Jigsaw160,338--- Java Logging4,248166272493 Java Swing337,2914,6949,56328,052 DBCP27,1946031,393 Synchronized Lists 17,6332,8623,2447,070 Synchronized Maps 18,9112,2952,5962898

40. Evaluation Program Name Lines Of Code # Deadlock CyclesProbability of Reproduction iGoodlockRealReproduced cache4j3,89700N.A. sor17,71800N.A. hedc25,02400N.A. jspider10,25200N.A. Jigsaw160,338283 ≥ 29 290.214 Java Logging4,2483331.00 Java Swing337,2911111.00 DBCP27,1942221.00 Synchronized Lists 17,63327 0.99 Synchronized Maps 18,91120 0.52

41. Evaluation

42. Limitations If DEADLOCKFUZZER does not reproduce a deadlock, we cannot say if the deadlock is a false positive or not Jigsaw – Deadlocks reported by iGoodlock : 283 – Deadlocks reproduced by DEADLOCKFUZZER : 29 – Deadlocks confirmed as false positives : 18 – Rest : 236

43. Limitations If DEADLOCKFUZZER does not reproduce a deadlock, we cannot say if the deadlock is a false positive or not Jigsaw – Deadlocks reported by iGoodlock : 283 – Deadlocks reproduced by DEADLOCKFUZZER : 29 – Deadlocks confirmed as false positives : 18 – Rest : 236 Cannot say if they are real deadlocks or false warnings

44. Conclusion DEADLOCKFUZZER is a practical deadlock detection tool that – Scales to large programs – Finds deadlock quickly – Finds real deadlocks Complements static and dynamic analyses – Automatically confirms some of the real deadlocks