1. Concurrent Queues and Stacks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

2. Art of Multiprocessor Programming2 2 The Five-Fold Path Coarse-grained locking Fine-grained locking Optimistic synchronization Lazy synchronization Lock-free synchronization

3. Art of Multiprocessor Programming3 3 Another Fundamental Problem We told you about –Sets implemented by linked lists –Hash Tables Next: queues Next: stacks

4. Art of Multiprocessor Programming4 4 Queues & Stacks pool of items

5. Art of Multiprocessor Programming5 5 Queues deq()/ enq( ) Total order First in First out

6. Art of Multiprocessor Programming6 6 Stacks pop()/ push( ) Total order Last in First out

7. Art of Multiprocessor Programming7 7 Bounded Fixed capacity Good when resources an issue

8. Art of Multiprocessor Programming8 8 Unbounded Unlimited capacity Often more convenient …

9. Art of Multiprocessor Programming9 Blocking zzz … Block on attempt to remove from empty stack or queue

10. Art of Multiprocessor Programming10 Blocking zzz … Block on attempt to add to full bounded stack or queue

11. Art of Multiprocessor Programming11 Non-Blocking Ouch! Throw exception on attempt to remove from empty stack or queue

12. Art of Multiprocessor Programming12 This Lecture Queue –Bounded, blocking, lock-based –Unbounded, non-blocking, lock-free Stack –Unbounded, non-blocking lock-free –Elimination-backoff algorithm

13. Art of Multiprocessor Programming13 Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

14. Art of Multiprocessor Programming14 Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

15. Art of Multiprocessor Programming15 Bounded Queue Sentinel head tail

16. Art of Multiprocessor Programming16 Bounded Queue head tail First actual item

17. Art of Multiprocessor Programming17 Bounded Queue head tail Lock out other deq() calls deqLock

18. Art of Multiprocessor Programming18 Bounded Queue head tail Lock out other enq() calls deqLock enqLock

19. Art of Multiprocessor Programming 1919 Not Done Yet head tail deqLock enqLock Need to tell whether queue is full or empty

20. Art of Multiprocessor Programming20 Not Done Yet head tail deqLock enqLock Max size is 8 items size 1

21. Art of Multiprocessor Programming21 Not Done Yet head tail deqLock enqLock Incremented by enq() Decremented by deq() size 1

22. Art of Multiprocessor Programming22 Enqueuer 1 Lock enqLock head tail deqLock enqLock size

23. Art of Multiprocessor Programming23 Enqueuer 1 Read size OK head tail deqLock enqLock size

24. Art of Multiprocessor Programming24 Enqueuer 1 No need to lock tail head tail deqLock enqLock size

25. Art of Multiprocessor Programming25 Enqueuer 1 Enqueue Node head tail deqLock enqLock size

26. Art of Multiprocessor Programming26 Enqueuer size 1 2 getAndincrement() head tail deqLock enqLock

27. Art of Multiprocessor Programming27 Enqueuer 8 Release lock 2 head tail deqLock enqLock size

28. Art of Multiprocessor Programming28 Enqueuer 2 If queue was empty, notify waiting dequeuers head tail deqLock enqLock size

29. Art of Multiprocessor Programming29 Unsuccesful Enqueuer 8 Uh-oh Read size … head tail deqLock enqLock size

30. Art of Multiprocessor Programming30 Dequeuer 2 Lock deqLock head tail deqLock enqLock size

31. Art of Multiprocessor Programming31 Dequeuer 2 Read sentinel’s next field OK head tail deqLock enqLock size

32. Art of Multiprocessor Programming32 Dequeuer siz 2 Read value head tail deqLock enqLock size

33. Art of Multiprocessor Programming33 Dequeuer 2 Make first Node new sentinel head tail deqLock enqLock size

34. Art of Multiprocessor Programming34 Dequeuer 1 Decrement size head tail deqLock enqLock size

35. Art of Multiprocessor Programming35 Dequeuer 1 Release deqLock head tail deqLock enqLock size

36. Art of Multiprocessor Programming36 Unsuccesful Dequeuer 0 Read sentinel’s next field uh-oh head tail deqLock enqLock size

37. Art of Multiprocessor Programming37 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

38. Art of Multiprocessor Programming38 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } enq & deq locks

39. Art of Multiprocessor Programming39 Digression: Monitor Locks Java synchronized objects and ReentrantLocks are monitors Allow blocking on a condition rather than spinning Threads: –acquire and release lock –wait on a condition

40. Art of Multiprocessor Programming40 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Acquire lock

41. Art of Multiprocessor Programming41 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Release lock

42. Art of Multiprocessor Programming42 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Try for lock, but not too hard

43. Art of Multiprocessor Programming43 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Create condition to wait on

44. Art of Multiprocessor Programming44 The Java Lock Interface public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } Never mind what this method does

45. Art of Multiprocessor Programming45 Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

46. Art of Multiprocessor Programming46 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Release lock and wait on condition

47. Art of Multiprocessor Programming47 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up one waiting thread

48. Art of Multiprocessor Programming48 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up all waiting threads

49. Art of Multiprocessor Programming49 Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

50. Art of Multiprocessor Programming50 Signal Awakens one waiting thread –Which will reacquire lock q.signal();

51. Art of Multiprocessor Programming51 Signal All Awakens all waiting threads –Which will each reacquire lock q.signalAll();

52. Art of Multiprocessor Programming52 A Monitor Lock Critical Section waiting room lock() unlock()

53. Art of Multiprocessor Programming53 Unsuccessful Deq Critical Section waiting room lock() await() deq() Oh no, empty!

54. Art of Multiprocessor Programming54 Another One Critical Section waiting room lock() await() deq() Oh no, empty!

55. Art of Multiprocessor Programming55 Enqueuer to the Rescue Critical Section waiting room lock() signalAll() enq( ) unlock() Yawn!

56. Art of Multiprocessor Programming56 Yawn! Monitor Signalling Critical Section waiting room Yawn! Awakened thread might still lose lock to outside contender…

57. Art of Multiprocessor Programming57 Dequeuers Signalled Critical Section waiting room Found it Yawn!

58. Art of Multiprocessor Programming58 Yawn! Dequeuers Signaled Critical Section waiting room Still empty!

59. Art of Multiprocessor Programming59 Dollar Short + Day Late Critical Section waiting room

60. Art of Multiprocessor Programming60 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods

61. Art of Multiprocessor Programming61 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Each object has an implicit lock with an implicit condition

62. Art of Multiprocessor Programming62 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Lock on entry, unlock on return

63. Art of Multiprocessor Programming63 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; this.notifyAll(); return result; } … }} Java Synchronized Methods Wait on implicit condition

64. Art of Multiprocessor Programming64 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) this.wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Signal all threads waiting on condition

65. Art of Multiprocessor Programming65 (Pop!) The Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

66. Art of Multiprocessor Programming66 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq & deq locks

67. Art of Multiprocessor Programming67 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq lock’s associated condition

68. Art of Multiprocessor Programming68 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } size: 0 to capacity

69. Art of Multiprocessor Programming69 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Head and Tail

70. Art of Multiprocessor Programming70 Enq Method Part One public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == Capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

71. Art of Multiprocessor Programming71 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Lock and unlock enq lock

72. Art of Multiprocessor Programming72 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Wait while queue is full …

73. Art of Multiprocessor Programming73 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One when await() returns, you might still fail the test !

74. Art of Multiprocessor Programming74 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Be Afraid After the loop: how do we know the queue won’t become full again?

75. Art of Multiprocessor Programming75 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Add new node

76. Art of Multiprocessor Programming76 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue was empty, wake frustrated dequeuers

77. Art of Multiprocessor Programming77 Beware Lost Wake-Ups Critical Section waiting room lock() Queue empty so signal () enq( ) unlock() Yawn!

78. Art of Multiprocessor Programming78 Lost Wake-Up Critical Section waiting room lock() enq( ) unlock() Yawn! Queue not empty so no need to signal

79. Art of Multiprocessor Programming79 Lost Wake-Up Critical Section waiting room Yawn!

80. Art of Multiprocessor Programming80 Lost Wake-Up Critical Section waiting room Found it

81. Art of Multiprocessor Programming81 What’s Wrong Here? Critical Section waiting room Still waiting ….!

82. Art of Multiprocessor Programming82 Solution to Lost Wakeup Always use –signalAll() and notifyAll() Not –signal() and notify() 82

83. Art of Multiprocessor Programming83 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

84. Art of Multiprocessor Programming84 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Are there dequeuers to be signaled?

85. Art of Multiprocessor Programming85 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Lock and unlock deq lock

86. Art of Multiprocessor Programming86 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Signal dequeuers that queue is no longer empty

87. Art of Multiprocessor Programming87 The enq() & deq() Methods Share no locks –That’s good But do share an atomic counter –Accessed on every method call –That’s not so good Can we alleviate this bottleneck?

88. Art of Multiprocessor Programming88 Split the Counter The enq() method –Increments only –Cares only if value is capacity The deq() method –Decrements only –Cares only if value is zero

89. Art of Multiprocessor Programming89 Split Counter Enqueuer increments enqSize Dequeuer increments deqSize When enqueuer hits capacity –Locks deqLock –Sets size = enqSize - DeqSize Intermittent synchronization –Not with each method call –Need both locks! (careful …)

90. Art of Multiprocessor Programming90 A Lock-Free Queue Sentinel head tail

91. Art of Multiprocessor Programming91 Compare and Set CAS

92. Art of Multiprocessor Programming92 Enqueue head tail enq( )

93. Art of Multiprocessor Programming93 Enqueue head tail

94. Art of Multiprocessor Programming94 Logical Enqueue head tail CAS

95. Art of Multiprocessor Programming95 Physical Enqueue head tail CAS

96. Art of Multiprocessor Programming96 Enqueue These two steps are not atomic The tail field refers to either –Actual last Node (good) –Penultimate Node (not so good) Be prepared!

97. Art of Multiprocessor Programming97 Enqueue What do you do if you find –A trailing tail? Stop and help fix it –If tail node has non-null next field –CAS the queue’s tail field to tail.next As in the universal construction

98. Art of Multiprocessor Programming98 When CASs Fail During logical enqueue –Abandon hope, restart –Still lock-free (why?) During physical enqueue –Ignore it (why?)

99. Art of Multiprocessor Programming99 Dequeuer head tail Read value

100. Art of Multiprocessor Programming100 Dequeuer head tail Make first Node new sentinel CAS

101. Art of Multiprocessor Programming101 Memory Reuse? What do we do with nodes after we dequeue them? Java: let garbage collector deal? Suppose there is no GC, or we prefer not to use it?

102. Art of Multiprocessor Programming102 Dequeuer head tail CAS Can recycle

103. Art of Multiprocessor Programming103 Simple Solution Each thread has a free list of unused queue nodes Allocate node: pop from list Free node: push onto list Deal with underflow somehow …

104. Art of Multiprocessor Programming104 Why Recycling is Hard Free pool headtail Want to redirect head from gray to red zzz…

105. Art of Multiprocessor Programming105 Both Nodes Reclaimed Free pool zzz headtail

106. Art of Multiprocessor Programming106 One Node Recycled Free pool Yawn! headtail

107. Art of Multiprocessor Programming107 Why Recycling is Hard Free pool CAS headtail OK, here I go!

108. Art of Multiprocessor Programming108 Recycle FAIL Free pool zOMG what went wrong? headtail

109. Art of Multiprocessor Programming109 The Dreaded ABA Problem headtail Head reference has value A Thread reads value A Head reference has value A Thread reads value A

110. Art of Multiprocessor Programming110 Dreaded ABA continued zzz headtail Head reference has value B Node A freed Head reference has value B Node A freed

111. Art of Multiprocessor Programming111 Dreaded ABA continued Yawn! headtail Head reference has value A again Node A recycled and reinitialized Head reference has value A again Node A recycled and reinitialized

112. Art of Multiprocessor Programming112 Dreaded ABA continued CAS headtail CAS succeeds because references match, even though reference’s meaning has changed CAS succeeds because references match, even though reference’s meaning has changed

113. Art of Multiprocessor Programming113 The Dreaded ABA FAIL Is a result of CAS() semantics –Oracle, Intel, AMD, … Not with Load-Locked/Store-Conditional –IBM …

114. Art of Multiprocessor Programming114 Dreaded ABA – A Solution Tag each pointer with a counter Unique over lifetime of node Pointer size vs word size issues Overflow? –Don’t worry be happy? –Bounded tags? AtomicStampedReference class

115. Art of Multiprocessor Programming115 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package address S Stamp Reference Can get reference & stamp atomically

116. Art of Multiprocessor Programming116 Concurrent Stack Methods –push(x) –pop() Last-in, First-out (LIFO) order Lock-Free!

117. Art of Multiprocessor Programming117 Empty Stack Top

118. Art of Multiprocessor Programming118 Push Top

119. Art of Multiprocessor Programming119 Push Top CAS

120. Art of Multiprocessor Programming120 Push Top

121. Art of Multiprocessor Programming121 Push Top

122. Art of Multiprocessor Programming122 Push Top

123. Art of Multiprocessor Programming123 Push Top CAS

124. Art of Multiprocessor Programming124 Push Top

125. Art of Multiprocessor Programming125 Pop Top

126. Art of Multiprocessor Programming126 Pop Top CAS

127. Art of Multiprocessor Programming127 Pop Top CAS mine!

128. Art of Multiprocessor Programming128 Pop Top CAS

129. Art of Multiprocessor Programming129 Pop Top

130. Art of Multiprocessor Programming130 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff(); }} Lock-free Stack

131. Art of Multiprocessor Programming131 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public Boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack tryPush attempts to push a node

132. Art of Multiprocessor Programming132 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Read top value

133. Art of Multiprocessor Programming133 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack current top will be new node’s successor

134. Art of Multiprocessor Programming134 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Try to swing top, return success or failure

135. Art of Multiprocessor Programming135 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Push calls tryPush

136. Art of Multiprocessor Programming136 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Create new node

137. Art of Multiprocessor Programming137 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack If tryPush() fails, back off before retrying

138. Art of Multiprocessor Programming138 Lock-free Stack Good –No locking Bad –Without GC, fear ABA –Without backoff, huge contention at top –In any case, no parallelism

139. Art of Multiprocessor Programming139 Big Question Are stacks inherently sequential? Reasons why –Every pop() call fights for top item Reasons why not –Stay tuned …

140. Art of Multiprocessor Programming140 Elimination-Backoff Stack How to –“turn contention into parallelism” Replace familiar –exponential backoff With alternative –elimination-backoff

141. Art of Multiprocessor Programming141 Observation Push( ) Pop() linearizable stack After an equal number of pushes and pops, stack stays the same After an equal number of pushes and pops, stack stays the same Yes!

142. Art of Multiprocessor Programming142 Idea: Elimination Array Push( ) Pop() stack Pick at random Pick at random Elimination Array

143. Art of Multiprocessor Programming143 Push Collides With Pop Push( ) Pop() stack continue No need to access stack No need to access stack Yes!

144. Art of Multiprocessor Programming144 No Collision Push( ) Pop() stack If no collision, access stack If no collision, access stack If pushes collide or pops collide access stack

145. Art of Multiprocessor Programming145 Elimination-Backoff Stack Lock-free stack + elimination array Access Lock-free stack, –If uncontended, apply operation –if contended, back off to elimination array and attempt elimination

146. Art of Multiprocessor Programming146 Elimination-Backoff Stack Push( ) Pop() Top CAS If CAS fails, back off

147. Art of Multiprocessor Programming147 Dynamic Range and Delay Push( ) Pick random range and max waiting time based on level of contention encountered

148. 50-50, Random Slots

149. Asymmetric Rendevous pop 1 () pop 2 () Push( ) Pops find first vacant slot and spin. Pushes hunt for pops.

150. Asymmetric vs. Symmetric

152. Effect of Backoff and Slot Choice

153. Effect of Slot Choice Random choice Sequential choice Darker shades mean more exchanges

154. Effect of Slot Choice

155. Art of Multiprocessor Programming155 Linearizability Un-eliminated calls –linearized as before Eliminated calls: –linearize pop() immediately after matching push() Combination is a linearizable stack

156. Art of Multiprocessor Programming156 Un-Eliminated Linearizability push(v 1 ) time linearizable push(v 1 ) pop(v 1 )

157. Art of Multiprocessor Programming157 Eliminated Linearizability pop(v 2 )push(v 1 ) push(v 2 ) time push(v 2 ) pop(v 2 ) push(v 1 ) pop(v 1 ) Collision Point Red calls are eliminated pop(v 1 ) linearizable

158. Art of Multiprocessor Programming158 Backoff Has Dual Effect Elimination introduces parallelism Backoff to array cuts contention on lock- free stack Elimination in array cuts down number of threads accessing lock-free stack

159. Art of Multiprocessor Programming159 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array

160. Art of Multiprocessor Programming160 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array An array of Exchangers

161. Art of Multiprocessor Programming161 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); Digression: A Lock-Free Exchanger

162. Art of Multiprocessor Programming162 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); A Lock-Free Exchanger Atomically modifiable reference + status

163. Art of Multiprocessor Programming163 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package In C or C++: address S reference stamp

164. Art of Multiprocessor Programming164 Extracting Reference & Stamp public T get(int[] stampHolder);

165. Art of Multiprocessor Programming165 Extracting Reference & Stamp public T get(int[] stampHolder); Returns reference to object of type T Returns stamp at array index 0!

166. Art of Multiprocessor Programming166 Exchanger Status enum Status {EMPTY, WAITING, BUSY};

167. Art of Multiprocessor Programming167 Exchanger Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet

168. enum Status {EMPTY, WAITING, BUSY}; Art of Multiprocessor Programming168 Exchange Status Nothing yet One thread is waiting for rendez-vous

169. Art of Multiprocessor Programming169 Exchange Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet One thread is waiting for rendez-vous Other threads busy with rendez-vous

170. Art of Multiprocessor Programming170 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange

171. Art of Multiprocessor Programming171 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Item and timeout

172. Art of Multiprocessor Programming172 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Array holds status

173. Art of Multiprocessor Programming173 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Loop until timeout

174. Art of Multiprocessor Programming174 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Get other’s item and status

175. Art of Multiprocessor Programming175 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } }} The Exchange An Exchanger has three possible states

176. Art of Multiprocessor Programming176 Lock-free Exchanger

177. Art of Multiprocessor Programming177 Lock-free Exchanger CAS

178. Art of Multiprocessor Programming178 Lock-free Exchanger

179. Art of Multiprocessor Programming179 Lock-free Exchanger In search of partner …

180. Art of Multiprocessor Programming180 Lock-free Exchanger Slot Still waiting … Try to exchange item and set status to BUSY CAS

181. Art of Multiprocessor Programming181 Lock-free Exchanger Slot Partner showed up, take item and reset to EMPTY item status

182. Art of Multiprocessor Programming182 EMPTY Lock-free Exchanger Slot item status Partner showed up, take item and reset to EMPTY

183. Art of Multiprocessor Programming183 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY

184. Art of Multiprocessor Programming184 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Try to insert myItem and change state to WAITING

185. Art of Multiprocessor Programming185 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Spin until either myItem is taken or timeout Spin until either myItem is taken or timeout

186. Art of Multiprocessor Programming186 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY myItem was taken, so return herItem that was put in its place myItem was taken, so return herItem that was put in its place

187. Art of Multiprocessor Programming187 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Otherwise we ran out of time, try to reset status to EMPTY and time out

188. Art of Multiprocessor Programming188 Art of Multiprocessor Programming© Herlihy-Shavit 2007 188 case EMPTY: // slot is free if (slot.compareAndSet(herItem, myItem, WAITING, BUSY)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.compareAndSet(myItem, null, WAITING, EMPTY)){throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If reset failed, someone showed up after all, so take that item If reset failed, someone showed up after all, so take that item

189. Art of Multiprocessor Programming189 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Clear slot and take that item

190. Art of Multiprocessor Programming190 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start

191. Art of Multiprocessor Programming191 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY

192. Art of Multiprocessor Programming192 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY

193. Art of Multiprocessor Programming193 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If successful, return other’s item, otherwise someone else took it, so try again from start If successful, return other’s item, otherwise someone else took it, so try again from start

194. Art of Multiprocessor Programming194 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If BUSY, other threads exchanging, so start again If BUSY, other threads exchanging, so start again

195. Art of Multiprocessor Programming195 The Exchanger Slot Exchanger is lock-free Because the only way an exchange can fail is if others repeatedly succeeded or no-one showed up The slot we need does not require symmetric exchange

196. Art of Multiprocessor Programming196 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Back to the Stack: the Elimination Array

197. Art of Multiprocessor Programming197 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array visit the elimination array with fixed value and range visit the elimination array with fixed value and range

198. Art of Multiprocessor Programming198 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Pick a random array entry

199. Art of Multiprocessor Programming199 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Exchange value or time out

200. Art of Multiprocessor Programming200 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push

201. Art of Multiprocessor Programming201 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push First, try to push

202. Art of Multiprocessor Programming202 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push If I failed, backoff & try to eliminate

203. Art of Multiprocessor Programming203 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Value pushed and range to try

204. Art of Multiprocessor Programming204 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Only pop() leaves null, so elimination was successful Only pop() leaves null, so elimination was successful

205. Art of Multiprocessor Programming205 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Otherwise, retry push() on lock-free stack

206. Art of Multiprocessor Programming206 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if (otherValue != null) { return otherValue; } }} Elimination Stack Pop

207. Art of Multiprocessor Programming207 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if ( otherValue != null) { return otherValue; } }} Elimination Stack Pop If value not null, other thread is a push(), so elimination succeeded If value not null, other thread is a push(), so elimination succeeded

208. Art of Multiprocessor Programming208 Summary We saw both lock-based and lock-free implementations of queues and stacks Don’t be quick to declare a data structure inherently sequential –Linearizable stack is not inherently sequential (though it is in the worst case) ABA is a real problem, pay attention

209. Art of Multiprocessor Programming209 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to –http://creativecommons.org/licenses/by-sa/3.0/. Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.