1. Concurrent Queues Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

2. Art of Multiprocessor Programming© Herlihy-Shavit 2007 2 Another Fundamental Problem We told you about –Sets implemented using linked lists Next: queues Next: stacks

3. Art of Multiprocessor Programming© Herlihy-Shavit 2007 3 Queues Pool of items Queue –enq() & deq() –First-in-first-out (FIFO) order

4. Art of Multiprocessor Programming© Herlihy-Shavit 2007 4 Bounded vs Unbounded Bounded –Fixed capacity –Good when resources an issue Unbounded –Holds any number of objects

5. Art of Multiprocessor Programming© Herlihy-Shavit 2007 5 Blocking vs Non-Blocking Problem cases: –Removing from empty pool –Adding to full (bounded) pool Blocking –Caller waits until state changes Non-Blocking –Method throws exception

6. Art of Multiprocessor Programming© Herlihy-Shavit 2007 6 This Lecture Bounded, Blocking, Lock-based Queue Unbounded, Non-Blocking, Lock-free Queue ABA problem Unbounded Non-Blocking Lock-free Stack Elimination-Backoff Stack

7. Art of Multiprocessor Programming© Herlihy-Shavit 2007 7 Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

8. Art of Multiprocessor Programming© Herlihy-Shavit 2007 8 Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

9. Art of Multiprocessor Programming© Herlihy-Shavit 2007 9 Bounded Queue Sentinel head tail

10. Art of Multiprocessor Programming© Herlihy-Shavit 2007 10 Bounded Queue head tail First actual item

11. Art of Multiprocessor Programming© Herlihy-Shavit 2007 11 Bounded Queue head tail Lock out other deq() calls deqLock

12. Art of Multiprocessor Programming© Herlihy-Shavit 2007 12 Bounded Queue head tail Lock out other enq() calls deqLock enqLock

13. Art of Multiprocessor Programming© Herlihy-Shavit 2007 13 Not Done Yet head tail deqLock enqLock Need to tell whether queue is full or empty

14. Art of Multiprocessor Programming© Herlihy-Shavit 2007 14 Not Done Yet head tail deqLock enqLock Permission to enqueue 8 items permits 8

15. Art of Multiprocessor Programming© Herlihy-Shavit 2007 15 Not Done Yet head tail deqLock enqLock Incremented by deq() Decremented by enq() permits 8

16. Art of Multiprocessor Programming© Herlihy-Shavit 2007 16 Enqueuer head tail deqLock enqLock permits 8 Lock enqLock

17. Art of Multiprocessor Programming© Herlihy-Shavit 2007 17 Enqueuer head tail deqLock enqLock permits 8 Read permits OK

18. Art of Multiprocessor Programming© Herlihy-Shavit 2007 18 Enqueuer head tail deqLock enqLock permits 8 No need to lock tail

19. Art of Multiprocessor Programming© Herlihy-Shavit 2007 19 Enqueuer head tail deqLock enqLock permits 8 Enqueue Node

20. Art of Multiprocessor Programming© Herlihy-Shavit 2007 20 Enqueuer head tail deqLock enqLock permits 8 7 getAndDecrement()

21. Art of Multiprocessor Programming© Herlihy-Shavit 2007 21 Enqueuer head tail deqLock enqLock permits 8 Release lock 7

22. Art of Multiprocessor Programming© Herlihy-Shavit 2007 22 Enqueuer head tail deqLock enqLock permits 7 If queue was empty, notify/signal waiting dequeuers

23. Art of Multiprocessor Programming© Herlihy-Shavit 2007 23 Unsuccesful Enqueuer head tail deqLock enqLock permits 0 Uh-oh Read permits

24. Art of Multiprocessor Programming© Herlihy-Shavit 2007 24 Dequeuer head tail deqLock enqLock permits 8 Lock deqLock

25. Art of Multiprocessor Programming© Herlihy-Shavit 2007 25 Dequeuer head tail deqLock enqLock permits 7 Read sentinel’s next field OK

26. Art of Multiprocessor Programming© Herlihy-Shavit 2007 26 Dequeuer head tail deqLock enqLock permits 7 Read value

27. Art of Multiprocessor Programming© Herlihy-Shavit 2007 27 Dequeuer head tail deqLock enqLock permits 7 Make first Node new sentinel

28. Art of Multiprocessor Programming© Herlihy-Shavit 2007 28 Dequeuer head tail deqLock enqLock permits 8 Increment permits

29. Art of Multiprocessor Programming© Herlihy-Shavit 2007 29 Dequeuer head tail deqLock enqLock permits 7 Release deqLock

30. Art of Multiprocessor Programming© Herlihy-Shavit 2007 30 Unsuccesful Dequeuer head tail deqLock enqLock permits 8 Read sentinel’s next field uh-oh

31. Art of Multiprocessor Programming© Herlihy-Shavit 2007 31 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

32. Art of Multiprocessor Programming© Herlihy-Shavit 2007 32 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Create condition to wait on

33. Art of Multiprocessor Programming© Herlihy-Shavit 2007 33 Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

34. Art of Multiprocessor Programming© Herlihy-Shavit 2007 34 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Release lock and wait on condition

35. Art of Multiprocessor Programming© Herlihy-Shavit 2007 35 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up one waiting thread

36. Art of Multiprocessor Programming© Herlihy-Shavit 2007 36 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up all waiting threads

37. Art of Multiprocessor Programming© Herlihy-Shavit 2007 37 Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

38. Art of Multiprocessor Programming© Herlihy-Shavit 2007 38 Signal Awakens one waiting thread –Which will reacquire lock q.signal();

39. Art of Multiprocessor Programming© Herlihy-Shavit 2007 39 Signal All Awakens all waiting threads –Which will each reacquire lock q.signalAll();

40. Art of Multiprocessor Programming© Herlihy-Shavit 2007 40 A Monitor Lock Critical Section waiting room Lock() unLock()

41. Art of Multiprocessor Programming© Herlihy-Shavit 2007 41 Unsuccessful Deq Critical Section waiting room Lock() await() Deq() Oh no, Empty!

42. Art of Multiprocessor Programming© Herlihy-Shavit 2007 42 Another One Critical Section waiting room Lock() await() Deq() Oh no, Empty!

43. Art of Multiprocessor Programming© Herlihy-Shavit 2007 43 Enqueur to the Rescue Critical Section waiting room Lock() signalAll() Enq( ) unLock() Yawn!

44. Art of Multiprocessor Programming© Herlihy-Shavit 2007 44 Yawn! Monitor Signalling Critical Section waiting room Yawn! Awakend thread might still lose lock to outside contender…

45. Art of Multiprocessor Programming© Herlihy-Shavit 2007 45 Dequeurs Signalled Critical Section waiting room Found it Yawn!

46. Art of Multiprocessor Programming© Herlihy-Shavit 2007 46 Yawn! Dequeurs Signalled Critical Section waiting room Still empty!

47. Art of Multiprocessor Programming© Herlihy-Shavit 2007 47 Dollar Short + Day Late Critical Section waiting room

48. Art of Multiprocessor Programming© Herlihy-Shavit 2007 48 Lost Wake-Up Critical Section waiting room Lock() signal () Enq( ) unLock() Yawn!

49. Art of Multiprocessor Programming© Herlihy-Shavit 2007 49 Lost Wake-Up Critical Section waiting room Lock() Enq( ) unLock() Yawn!

50. Art of Multiprocessor Programming© Herlihy-Shavit 2007 50 Lost Wake-Up Critical Section waiting room Yawn!

51. Art of Multiprocessor Programming© Herlihy-Shavit 2007 51 Lost Wake-Up Critical Section waiting room Found it

52. Art of Multiprocessor Programming© Herlihy-Shavit 2007 52 What’s Wrong Here? Critical Section waiting room zzzz….!

53. Solution to Lost Wakeup Always use signalAll and notifyAll Not signal and notify Art of Multiprocessor Programming© Herlihy-Shavit 2007 53

54. Art of Multiprocessor Programming© Herlihy-Shavit 2007 54 The Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

55. Art of Multiprocessor Programming© Herlihy-Shavit 2007 55 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq & deq locks

56. Art of Multiprocessor Programming© Herlihy-Shavit 2007 56 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq lock’s associated condition

57. Art of Multiprocessor Programming© Herlihy-Shavit 2007 57 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Num permits: 0 to capacity

58. Art of Multiprocessor Programming© Herlihy-Shavit 2007 58 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Head and Tail

59. Art of Multiprocessor Programming© Herlihy-Shavit 2007 59 Enq Method Part One public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

60. Art of Multiprocessor Programming© Herlihy-Shavit 2007 60 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Lock and unlock enq lock

61. Art of Multiprocessor Programming© Herlihy-Shavit 2007 61 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue is full, patiently await further instructions …

62. Art of Multiprocessor Programming© Herlihy-Shavit 2007 62 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Add new node

63. Art of Multiprocessor Programming© Herlihy-Shavit 2007 63 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue was empty, wake frustrated dequeuers

64. Art of Multiprocessor Programming© Herlihy-Shavit 2007 64 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

65. Art of Multiprocessor Programming© Herlihy-Shavit 2007 65 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Are there dequeuers to be signaled?

66. Art of Multiprocessor Programming© Herlihy-Shavit 2007 66 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Lock and unlock deq lock

67. Art of Multiprocessor Programming© Herlihy-Shavit 2007 67 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Signal dequeuers that queue no longer empty

68. Art of Multiprocessor Programming© Herlihy-Shavit 2007 68 The Enq() & Deq() Methods Share no locks –That’s good But do share an atomic counter –Accessed on every method call –That’s not so good Can we alleviate this bottleneck?

69. Art of Multiprocessor Programming© Herlihy-Shavit 2007 69 A Lock-Free Queue Sentinel head tail

70. Art of Multiprocessor Programming© Herlihy-Shavit 2007 70 Compare and Set CAS

71. Art of Multiprocessor Programming© Herlihy-Shavit 2007 71 Enqueue head tail Enq( )

72. Art of Multiprocessor Programming© Herlihy-Shavit 2007 72 Enqueue head tail

73. Art of Multiprocessor Programming© Herlihy-Shavit 2007 73 Logical Enqueue head tail CAS

74. Art of Multiprocessor Programming© Herlihy-Shavit 2007 74 Physical Enqueue head tail Enqueue Node CAS

75. Art of Multiprocessor Programming© Herlihy-Shavit 2007 75 Enqueue These two steps are not atomic The tail field refers to either –Actual last Node (good) –Penultimate Node (not so good) Be prepared!

76. Art of Multiprocessor Programming© Herlihy-Shavit 2007 76 Enqueue What do you do if you find –A trailing tail? Stop and help fix it –If tail node has non-null next field –CAS the queue’s tail field to tail.next As in the universal construction

77. Art of Multiprocessor Programming© Herlihy-Shavit 2007 77 When CASs Fail During logical enqueue –Abandon hope, restart –Still lock-free (why?) During physical enqueue –Ignore it (why?)

78. Art of Multiprocessor Programming© Herlihy-Shavit 2007 78 Enq Method public void enq (T value) { Node node = new Node(value); while ( true ) { Node last = tail.get(); Node next = last.next.get(); if (last == tail.get()) { if (next == null ) { if (last.next.compareAndSet(next, node)) { tail.compareAndSet(last, node); return ; } } else { tail.compareAndSet(last, next); }

79. Art of Multiprocessor Programming© Herlihy-Shavit 2007 79 Dequeuer head tail Read value

80. Art of Multiprocessor Programming© Herlihy-Shavit 2007 80 Dequeuer head tail Make first Node new sentinel CAS

81. Art of Multiprocessor Programming© Herlihy-Shavit 2007 81 Deq Method public T deq() throws EmptyException { while (true) { Node first = head.get(); Node last = tail.get(); Node next = first.next.get(); if (first == head.get()) { if (first == last) { if (next == null) { throw new EmptyException(); } tail.compareAndSet(last, next); } else { T value = next.value; if (head.compareAndSet(first, next)) return value; } }}}

82. Art of Multiprocessor Programming© Herlihy-Shavit 2007 82 Memory Reuse? What do we do with nodes after we dequeue them? Java: let garbage collector deal? Suppose there is no GC, or we prefer not to use it?

83. Art of Multiprocessor Programming© Herlihy-Shavit 2007 83 Dequeuer head tail CAS Can recycle

84. Art of Multiprocessor Programming© Herlihy-Shavit 2007 84 Simple Solution Each thread has a free list of unused queue nodes Allocate node: pop from list Free node: push onto list Deal with underflow somehow …

85. Art of Multiprocessor Programming© Herlihy-Shavit 2007 85 Why Recycling is Hard Free pool headtail Want to redirect tail from grey to red zzz…

86. Art of Multiprocessor Programming© Herlihy-Shavit 2007 86 Both Nodes Reclaimed Free pool zzz headtail

87. Art of Multiprocessor Programming© Herlihy-Shavit 2007 87 One Node Recycled Free pool Yawn! headtail

88. Art of Multiprocessor Programming© Herlihy-Shavit 2007 88 Why Recycling is Hard Free pool CAS headtail OK, here I go!

89. Art of Multiprocessor Programming© Herlihy-Shavit 2007 89 Final State Free pool zOMG what went wrong? headtail Bad news

90. Art of Multiprocessor Programming© Herlihy-Shavit 2007 90 The Dreaded ABA Problem Head pointer has value A Thread reads value A headtail

91. Art of Multiprocessor Programming© Herlihy-Shavit 2007 91 Dreaded ABA continued zzz Head pointer has value B Node A freed headtail

92. Art of Multiprocessor Programming© Herlihy-Shavit 2007 92 Dreaded ABA continued Yawn! Head pointer has value A again Node A recycled & reinitialized headtail

93. Art of Multiprocessor Programming© Herlihy-Shavit 2007 93 Dreaded ABA continued CAS succeeds because pointer matches even though pointer’s meaning has changed CAS headtail

94. Art of Multiprocessor Programming© Herlihy-Shavit 2007 94 The Dreaded ABA Problem Is a result of CAS() semantics –I blame Sun, Intel, AMD, … Not with Load-Locked/Store- Conditional –Good for IBM?

95. Art of Multiprocessor Programming© Herlihy-Shavit 2007 95 Dreaded ABA – A Solution Tag each pointer with a counter Unique over lifetime of node Pointer size vs word size issues Overflow? –Don’t worry be happy? –Bounded tags? AtomicStampedReference class

96. Art of Multiprocessor Programming© Herlihy-Shavit 2007 96 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package address S Stamp Reference Can get reference and stamp atomically

97. Art of Multiprocessor Programming© Herlihy-Shavit 2007 97 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to –http://creativecommons.org/licenses/by-sa/3.0/. Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.