1. How not to do Java Concurrency And how to find if you did it wrong Mark Winterrowd 2014-09-30

2. Introductions Coverity Me You! Bob Copyright Coverity, Inc., 20142

3. Meet Bob Bob's development philosophy: Speed is everything Clever tweaks are their own reward Trying it out once or twice is plenty testing Your manager has assigned him to help you out! Copyright Coverity, Inc., 20143

4. Your Mission: Keep your code correct! Bob will write code mirroring mistakes seen in the wild For novices: preparation for handling your own Bob For experts: Examples occurring in real code Modeled off of code seen in Tomcat, Eclipse and Android Copyright Coverity, Inc., 20144

5. Patterns discussed here are low level Prefer to use higher level structures e.g., java.util.concurrent Hide uses behind higher level calls or objects Sometimes unavoidable Copyright Coverity, Inc., 20145

6. Multithreaded Lazy Initialization Or, sacrificing correctness for speed

7. Task: Lazily initialize a singleton In a single-threaded program: private static MyObj inst = null; public static MyObj getInst() { if(inst == null) { inst = new MyObj(); } return inst; } This won't work in a multithreaded program. Copyright Coverity, Inc., 20147

8. Simple Fix: private static MyObj inst = null; static final Object initLock = new Object(); public static MyObj getInst() { synchronized(initLock) { if(inst == null) { inst = new MyObj(); } return inst; } Bob objects: "This requires you to wait to get the lock every time!" Copyright Coverity, Inc., 20148

9. Bob's Performance Optimization public static MyObj getInst() { inst = new MyObj(); } return inst; } "Since we only need to set inst the first time, synchronizing to check for null is unnecessary!" Copyright Coverity, Inc., 20149 if(inst == null) { synchronized(initLock) { Is Bob right? Is his code correct?

10. Bob's optimized code in action Copyright Coverity, Inc., 201410 Thread 1Thread 2 if(inst == null) synchronized(initLock) inst = new MyObj(); return inst;synchronized(initLock) inst = new MyObj(); return inst; No longer a singleton!

11. The JVM can start and stop threads at will Holding a lock prevents another thread from executing code guarded by the same lock object Bob says "In that case, I have an idea!" Copyright Coverity, Inc., 201411

12. "Also check in the synchronized block!" private static MyObj inst = null; static final Object initLock = new Object(); public static MyObj getInst() { if(inst == null) synchronized(initLock) { Copyright Coverity, Inc., 201412 "Now, we can't have inst be initialized multiple times!" } return inst; } if(inst == null) inst = new MyObj(); Is Bob right? Is his code correct?

13. Let's take a look at this line Copyright Coverity, Inc., 201413 inst = new MyObj(); MyObj() { this.field1 = someVal1; this.field2 = someVal2; } A raw MyObj is created before running the constructor (raw MyObj).field1 = someVal1; (raw MyObj).field2 = someVal2; inst = (raw MyObj);

14. JVM re-orders code at runtime! Must have same result in current thread Cannot move code out of or through a synchronized block Reordering inside or into a synchronized is fine! Copyright Coverity, Inc., 201414 public static MyObj getInst() { if(inst == null) synchronized(initLock) { if(inst == null) { inst = new MyObj(); } return inst; } inst = (raw MyObj) inst.field1 = someVal inst.field2 = someVal2

15. This instruction ordering in action: Copyright Coverity, Inc., 201415 Thread 1Thread 2 if(inst == null) synchronized(initLock) inst = (raw MyObj); return inst; Returned an uninitialized MyObj! What is the name of this bad pattern? if(inst == null)

16. Preventing reordering bugs Hold the lock while checking the field for null public static MyObj getInst() { Copyright Coverity, Inc., 201416 synchronized(initLock) { if(inst == null) inst = new MyObj(); } return inst; } if(inst == null) Back to our original solution! "This problem seems unlikely. Is fixing it worth the slowdown?"

17. What slowdown? How unlikely? Uncontested lock acquisition is fast in modern JVMs In computing, unlikely can happen often In 2004, a developer opened Eclipse bug 50178 with this comment: Copyright Coverity, Inc., 201417 "I know I'm picky, but [...] getManifest has an example of what is known as "Double-Checked Locking problem [...] BundleLoader.createClassLoader has the same problem"

18. In the wild: Eclipse bug 50178 Copyright Coverity, Inc., 201418 Reply in 2006: "Synchronizing the places where we lazily create ClassLoader/BundleLoaderProxy objects will cause measurable slowdown. To fix this properly would require at least three additional syncs [...] I added todo comments in the code where double checks are currently being done." A month later: "We are currently having a very hard to reproduce bug when loading preference pages of our RCP application [...] I traced the problem to getBundleLoader [...] When looking at [two methods], I saw they both contained the TODO for this bug. Isn't it better to have slightly less performance instead of possible threading bugs?"

19. Bug 50178 as reported in Coverity Connect Copyright Coverity, Inc., 201419

20. Premature Optimization... Especially dangerous in concurrency Rare bugs may be likely in a user's workflow Slow locking may be unnecessary contention Is the lock protecting too much data? Is the lock protecting unrelated critical sections? Can some expensive operations be safely moved outside of a locked region? Copyright Coverity, Inc., 201420

21. Choosing your locks Or rather, what not to choose

22. Make this code thread safe static Object[] items; void update(Object newItem) { Copyright Coverity, Inc., 201422 "Synchronize on items so another thread can't change items." Object[] oldArr = items; items = new Object[items.length + 1] items[items.length - 1] = newItem; for(int i=0; i < oldArr.length; i++){ items[i] = oldArr[i]; } } synchronized(items) {

23. What does synchronizing on a field do? Does not block accesses from other threads to that field Acquires a lock on the field's contents Copyright Coverity, Inc., 201423 Thread 1 synchronized(items) items = new Object[...] synchronized(items) items = new Object[...] Different contents! Object[ ] oldArr = itemsitems[...] = newItem Thread 2

24. Don't lock on mutable fields Copyright Coverity, Inc., 201424 static Object[] items; void update(Object newItem) { synchronized( Object[] newAr = new Object[items.length+1]; newAr[newAr.length - 1] = newItem; for(int i=0; i < items.length; i++){ newArr[i] = items[i] } items = newAr; } ) { private static final Object lock = new Object(); itemslock

25. Tomcat bug 46990: Locking a mutable field Copyright Coverity, Inc., 201425 "Whilst there aren't any explicit bugs caused by this, it may be behind some of the harder to reproduce bugs."

26. Bob sends you the following code for review Copyright Coverity, Inc., 201426 class AppCtx { private static SysCtx sCtx; public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } " synchronized ensures this code is thread-safe!"

27. What does the synchronized modifier do? Copyright Coverity, Inc., 201427 class AppCtx { private static SysCtx sCtx; public synchronized SysCtx getSysCtx() { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } } synchronized(this) {

28. Two AppCtx, each in their own thread Copyright Coverity, Inc., 201428 this AppCtx 1 AppCtx 2 this sCtx null unlocked synchronized(this) Static AppCtx Members AppCtx1AppCtx2 if(sCtx == null) synchronized(this) if(sCtx == null) sCtx = new SysCtx() Initialized return sCtx; sCtx = new SysCtx()return sCtx; No longer a singleton!

29. Guard static members with static locks Copyright Coverity, Inc., 201429 class AppCtx { private static SysCtx sCtx; public SysCtx getSysCtx() { synchronized( ) { if(sCtx == null) { sCtx = new SysCtx(); } return sCtx; } private static final Object lock=new Object(); thislock

30. In the wild: Android bug 12015587 Copyright Coverity, Inc., 201430 Caused loss of display info, user name, and permissions

31. Wait and Notify and threads communicating poorly

32. Thread 1 consumes results from Thread 2 LinkedList jobQueue; // Called in Thread 1 void processItemFromQueue() { QueueItem item = null; synchronized(qLock) { item = jobQueue.remove(); } processItem(item); } // Called in Thread 2 void putItemInQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); } Copyright Coverity, Inc., 201432 What if jobQueue is empty?

33. The wait / notify pattern lock.wait( ) : current thread stops running, releases lock, and is placed on lock 's wait set lock.notifyAll( ) : Starts up all threads in the wait set, which attempt to re-acquire lock. Call wait when your thread needs another thread to satisfy some condition Call notifyAll when your thread has satisfied a condition for another thread Copyright Coverity, Inc., 201433

34. wait and notify in action Copyright Coverity, Inc., 201434 Executing Instructionslock Locked By Unlocked Wait Set synchronized(lock)lock.wait( ) Blocked on lock synchronized(lock)lock.wait( )synchronized(lock)lock.wait( )lock.notifyAll( )release locksynchronized(lock)release lock

35. You see this in a review of Bob's code: void processItemFromQueue() { QueueItem item = null; // Hold the lock as briefly as possible! if(jobQueue.empty()) synchronized(qLock) { qLock.wait(); } item = jobQueue.remove(); processItem(item); } void addItemToQueue(QueueItem item) { synchronized(qLock) { jobQueue.add(item); qLock.notifyAll(); } Copyright Coverity, Inc., 201435 Is Bob's code correct?

36. Bob's code in action: Copyright Coverity, Inc., 201436 Executing Instructions qLock Locked By Unlocked Wait Set if(jobQueue.empty()) Consumer qLock.wait() C Blocked on qLock synchronized(qLock) ConsumerProducer synchronized(qLock) Producer jobQueue.add(item)release qLockqLock.notifyAll() No waiting threads to notify returns true Consumer is waiting while items are in the queue!

37. Waiting on a stale condition is a bug Another thread can change the state between the check and the wait At best, causes unnecessary delays What if future notifications were blocked on this task? Always check wait condition while holding lock Copyright Coverity, Inc., 201437 Bob comes back with a second version...

38. item = jobQueue.remove(); Code review round #2 void processItemFromQueue() { QueueItem item; Copyright Coverity, Inc., 201438 Bob: "Now the wait will only occur when the queue is empty!" if(jobQueue.empty()) { processItem(item); qLock.wait(); synchronized(qLock) { Is Bob right? Is his code correct? " A thread can also wake up without being notified, interrupted, or timing out, a so- called spurious wakeup." -Object.wait() documentation } } }

39. Bob's code in action: Copyright Coverity, Inc., 201439 Executing Instructions qLock Locked By Unlocked Wait Set if(jobQueue.empty()) Consumer qLock.wait() C Blocked on qLock synchronized(qLock) ConsumerProducer A spurious wakeup occurs! jobQueue.remove() jobQueue empty!

40. Check your wait condition inside a loop! Copyright Coverity, Inc., 201440 void processItemFromQueue() { QueueItem item; synchronized(qLock) { qLock.wait(); item = jobQueue.remove(); } processItem(item); } whileif (jobQueue.empty())

41. In the wild: Eclipse bug 366048 Copyright Coverity, Inc., 201441 Deadlock when ctrl-x was used to cut text. Diagnosed as a wait occurring and never waking up. Remains unfixed, timeout to wait was set to 30s.

42. The bug (in 2004 code) In code from 2005 Copyright Coverity, Inc., 201442 This bug was "fixed" with a 30 second wait timeout in 2013.

43. Preventing new concurrency bugs and fixing the ones you have

44. Don't get overly clever! Do you need a multithreaded program? Do you need lazy initialization? Initialize using static{} Is there a higher level structure to replace wait / notify ? Our example: java.util.concurrent.BlockingQueue Sometimes it's unavoidable, or hard to remove. Copyright Coverity, Inc., 201444

45. Standard methods help somewhat Education Code reviews Testing Copyright Coverity, Inc., 201445 Nondeterminism limits effectiveness Is there anything else we can do? Don’t fix pre-existing problems

46. Manual Inspection Responsible for finding many of our sampled fixed defects One tip we found in Bugzilla: Copyright Coverity, Inc., 201446 No guarantee anyone will notice right away Most of the fixed bugs found by inspection had been present for over 3 years. Why not have a machine inspect your code instead? is Slow "Search[ing] for the ' synchronized ' keyword goes a long way"

47. Static Analysis: Automatic code inspection Open source static analysis FindBugs™ finds many issues But Coverity's static analysis finds more real bugs with better explanations and a lower rate of false reports Also, we integrate with FindBugs Copyright Coverity, Inc., 201447

48. Analyze your code for free! Coverity's SCAN program provides free defect reports for open source programs. http://scan.coverity.com Coverity's CodeSpotter: free-to-use SaaS analysis http://www.code-spotter.com Also a commercial enterprise solution for proprietary software http://www.coverity.com Copyright Coverity, Inc., 201448

49. More about Java Concurrency Java Language Specification "Java Concurrency in Practice" by Goetz et al "Java theory and practice: Are all stateful Web applications broken?" by Goetz http://www.ibm.com/developerworks/library/j-jtp09238/ "Double checked locking is broken", many signers. http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheck edLocking.html http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheck edLocking.html Copyright Coverity, Inc., 201449

50. Copyright 2014 Coverity, Inc.