1. Lecture 6-2 : Concurrent Queues and Stacks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

2. pool Data Structure similar to Set –Does not necessarily provide contains() method –Allows the same item to appear more than once –get() and set() Art of Multiprocessor Programming© Herlihy-Shavit 2007 2 public interface Pool { void put(T item); T get(); }

3. Art of Multiprocessor Programming© Herlihy-Shavit 2007 3 Queues & Stacks Both: pool of items Queue –enq() & deq() –First-in-first-out (FIFO) order Stack –push() & pop() –Last-in-first-out (LIFO) order

4. Art of Multiprocessor Programming© Herlihy-Shavit 2007 4 Bounded vs Unbounded Bounded –Fixed capacity –Good when resources an issue Unbounded –Holds any number of objects

5. Art of Multiprocessor Programming© Herlihy-Shavit 2007 5 Blocking vs Non-Blocking Problem cases: –Removing from empty pool –Adding to full (bounded) pool Blocking –Caller waits until state changes Non-Blocking –Method throws exception

6. Art of Multiprocessor Programming© Herlihy-Shavit 2007 6 Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

7. Art of Multiprocessor Programming© Herlihy-Shavit 2007 7 Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

8. Art of Multiprocessor Programming© Herlihy-Shavit 2007 8 A Bounded Lock-Based Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; public BoundedQueue(int capacity) { this.capacity = capacity; this.head = new Node(null); this.tail = head; this.size = new AtomicInteger(0); this.enqLock = new ReentrantLock(); this.notFullCondition = enqLock.newCondition(); this.deqLock = new ReentrantLock(); this.notEmptyCondition = deqLock.newCondition(); } protected class Node { public T value; public Node next; public Node(T x) { value = x; next = null; }

9. 9 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (size.getAndIncrement() == 0) { mustWakeDequeuers = true; } } finally { enqLock.unlock(); } if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } public T deq() { T result; boolean mustWakeEnqueuers = false; deqLock.lock(); try { while (size.get() == 0) { notEmptyCondition.await(); result = head.next.value; head = head.next; if (size.getAndDecrement() == capacity) { mustWakeEnqueuers = true; } } finally { deqLock.unlock(); } if (mustWakeEnqueuers) { enqLock.lock(); try { notFullCondition.signalAll(); } finally { enqLock.unlock(); } return result; }

10. lock enqLock/deqLock –At most one enqueuer/dequeuer at a time can manipulate the queue’s fields Two locks –Enqueuer does not lock out dequeuer –vice versa Association –enqLock associated with notFullCondition –deqLock associated with notEmptyCondition Art of Multiprocessor Programming© Herlihy-Shavit 2007 10

11. enqueue 1.Acquires enqLock 2.Reads the size field 3.If full, enqueuer must wait until dequeuer makes room 4.enqueuer waits on notFullCondition field, releasing enqLock temporarily, and blocking until that condition is signaled. 5.Each time the thread awakens, it checks whether there is a room, and if not, goes back to sleep 6.Insert new item into tail 7.Release enqLock 8.If queue was empty, notify/signal waiting dequeuers Art of Multiprocessor Programming© Herlihy-Shavit 2007 11

12. dequeue 1.Acquires deqLock 2.Reads the size field 3.If empty, dequeuer must wait until item is enqueued 4.dequeuer waits on notEmptyCondition field, releasing deqLock temporarily, and blocking until that condition is signaled. 5.Each time the thread awakens, it checks whether item was enqueued, and if not, goes back to sleep 6.Assigne the value of head’s next node to “result” and reset head to head’s next node 7.Release deqLock 8.If queue was full, notify/signal waiting enqueuers 9.Return “result” Art of Multiprocessor Programming© Herlihy-Shavit 2007 12

13. Art of Multiprocessor Programming13 Bounded Queue Sentinel head tail

14. Art of Multiprocessor Programming14 Bounded Queue head tail First actual item

15. Art of Multiprocessor Programming15 Bounded Queue head tail Lock out other deq() calls deqLock

16. Art of Multiprocessor Programming16 Bounded Queue head tail Lock out other enq() calls deqLock enqLock

17. Art of Multiprocessor Programming17 Not Done Yet head tail deqLock enqLock Need to tell whether queue is full or empty

18. Art of Multiprocessor Programming18 Not Done Yet head tail deqLock enqLock Max size is 8 items size 1

19. Art of Multiprocessor Programming19 Not Done Yet head tail deqLock enqLock Incremented by enq() Decremented by deq() size 1

20. Art of Multiprocessor Programming20 Enqueuer head tail deqLock enqLock size 1 Lock enqLock

21. Art of Multiprocessor Programming21 Enqueuer head tail deqLock enqLock size 1 Read size OK

22. Art of Multiprocessor Programming22 Enqueuer head tail deqLock enqLock size 1 No need to lock tail

23. Art of Multiprocessor Programming23 Enqueuer head tail deqLock enqLock size 1 Enqueue Node

24. Art of Multiprocessor Programming24 Enqueuer head tail deqLock enqLock size 1 2 getAndincrement()

25. Art of Multiprocessor Programming25 Enqueuer head tail deqLock enqLock size 8 Release lock 2

26. Art of Multiprocessor Programming26 Enqueuer head tail deqLock enqLock size 2 If queue was empty, notify waiting dequeuers

27. Art of Multiprocessor Programming27 Unsuccesful Enqueuer head tail deqLock enqLock size 8 Uh- oh Read size …

28. Art of Multiprocessor Programming28 Dequeuer head tail deqLock enqLock size 2 Lock deqLock

29. Art of Multiprocessor Programming29 Dequeuer head tail deqLock enqLock size 2 Read sentinel’s next field OK

30. Art of Multiprocessor Programming30 Dequeuer head tail deqLock enqLock size 2 Read value

31. Art of Multiprocessor Programming31 Dequeuer head tail deqLock enqLock size 2 Make first Node new sentinel

32. Art of Multiprocessor Programming32 Dequeuer head tail deqLock enqLock size 1 Decrement size

33. Art of Multiprocessor Programming33 Dequeuer head tail deqLock enqLock size 1 Release deqLock

34. Art of Multiprocessor Programming© Herlihy-Shavit 2007 34 Monitor Locks Java ReentrantLocks are monitors Allow blocking on a condition rather than spinning Threads: –acquire and release lock –wait on a condition

35. Art of Multiprocessor Programming© Herlihy-Shavit 2007 35 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Create condition to wait on

36. Art of Multiprocessor Programming© Herlihy-Shavit 2007 36 Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

37. Art of Multiprocessor Programming© Herlihy-Shavit 2007 37 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Release lock and wait on condition

38. Art of Multiprocessor Programming© Herlihy-Shavit 2007 38 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up one waiting thread

39. Art of Multiprocessor Programming© Herlihy-Shavit 2007 39 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up all waiting threads

40. Art of Multiprocessor Programming© Herlihy-Shavit 2007 40 Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

41. Art of Multiprocessor Programming© Herlihy-Shavit 2007 41 Signal Awakens one waiting thread –Which will reacquire lock q.signal();

42. Art of Multiprocessor Programming© Herlihy-Shavit 2007 42 Signal All Awakens all waiting threads –Which will each reacquire lock q.signalAll();

43. Unbounded Lock-Free Queue (Nonblocking) Unbounded –No need to count the number of items Lock-free –Use AtomicReference An object reference that may be updated atomically. –boolean compareAndSet(V expect, V update) Atomically sets the value to the given updated value if the current value == the expected value. Nonblocking –No need to provide conditions on which to wait 43

44. Art of Multiprocessor Programming44 A Lock-Free Queue Sentinel head tail

45. Art of Multiprocessor Programming45 Enqueue head tail Enq( )

46. Art of Multiprocessor Programming46 Enqueue head tail

47. Art of Multiprocessor Programming47 Logical Enqueue head tail CAS

48. Art of Multiprocessor Programming48 Physical Enqueue head tail CAS

49. Art of Multiprocessor Programming49 Enqueue These two steps are not atomic The tail field refers to either –Actual last Node (good) –Penultimate Node (not so good) Be prepared!

50. Art of Multiprocessor Programming50 Enqueue What do you do if you find –A trailing tail? Stop and help fix it –If tail node has non-null next field –CAS the queue’s tail field to tail.next As in the universal construction

51. Art of Multiprocessor Programming51 When CASs Fail During logical enqueue –Abandon hope, restart –Still lock-free (why?) During physical enqueue –Ignore it (why?)

52. Art of Multiprocessor Programming52 Dequeuer head tail Read value

53. Art of Multiprocessor Programming53 Dequeuer head tail Make first Node new sentinel CAS

54. Art of Multiprocessor Programming© Herlihy-Shavit 2007 54 Unbounded Lock-Free Queue (Nonblocking) public class LockFreeQueue { private AtomicReference head; private AtomicReference tail; public LockFreeQueue() { this.head = new AtomicReference (null); this.tail = new AtomicReference (null); } public class Node { public T value; public AtomicReference next; public Node(T value) { this.value = value; this.next = new AtomicReference (null); }

55. 55 public void enq(T item) { Node node = new Node(item); while (true) { Node last = tail.get(); Node next = last.next.get(); if (last == tail.get()) { if (next == null) { if (last.next.compareAndSet(next, node)) { tail.compareAndSet(last, node); return; } } else { tail.compareAndSet(last, next); } public T deq() throws EmptyException { while (true) { Node first = head.get(); Node last = tail.get(); Node next = first.next.get(); if (first == head.get()) { if (first == last) { if (next = null) { throw new EmptyException(); } tail.compareAndSet(last, next); } else { T value = next.value; if (head.compareAndSet(first,next)) return value; } Unbounded Lock-Free Queue (Nonblocking)

56. Art of Multiprocessor Programming56 Concurrent Stack Methods –push(x) –pop() Last-in, First-out (LIFO) order Lock-Free!

57. Art of Multiprocessor Programming57 Empty Stack Top

58. Art of Multiprocessor Programming58 Push Top

59. Art of Multiprocessor Programming59 Push Top CAS

60. Art of Multiprocessor Programming60 Push Top

61. Art of Multiprocessor Programming61 Push Top

62. Art of Multiprocessor Programming62 Push Top

63. Art of Multiprocessor Programming63 Push Top CAS

64. Art of Multiprocessor Programming64 Push Top

65. Art of Multiprocessor Programming65 Pop Top

66. Art of Multiprocessor Programming66 Pop Top CAS

67. Art of Multiprocessor Programming67 Pop Top CAS mine !

68. Art of Multiprocessor Programming68 Pop Top CAS

69. Art of Multiprocessor Programming69 Pop Top

70. Art of Multiprocessor Programming70 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff(); }} Lock-free Stack

71. Art of Multiprocessor Programming71 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public Boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack tryPush attempts to push a node

72. Art of Multiprocessor Programming72 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Read top value

73. Art of Multiprocessor Programming73 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack current top will be new node’s successor

74. Art of Multiprocessor Programming74 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Try to swing top, return success or failure

75. Art of Multiprocessor Programming75 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Push calls tryPush

76. Art of Multiprocessor Programming76 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Create new node

77. Art of Multiprocessor Programming77 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack If tryPush() fails, back off before retrying

78. 78 protected boolean tryPush(Node node) { Node oldTop = top.get(); node.next = oldTop; return (top.compareAndSet(oldTop, node)); } public void push( T value ) { Node node = new Node( value ); while (true) { if (tryPush(node)) { return; } else { backoff.backoff( ); } } protected Node tryPop( ) throws EmptyException { Node oldTop = top.get(); if ( oldTop == null ) { throw new EmptyException( ); } Node newTop = oldTop.next; if ( top.compareAndSet( oldTop, newTop ) ) { return oldTop; } else { return null; } } public T pop() throws EmptyException { while (true) { Node returnNode = tryPop( ); if ( returnNode != null ) { return returnNode.value; } else { backoff.backoff( ); } } Unbounded Lock-Free Stack

79. Art of Multiprocessor Programming79 Lock-free Stack Good –No locking Bad –Without GC, fear ABA –Without backoff, huge contention at top –In any case, no parallelism

80. Art of Multiprocessor Programming80 Question Are stacks inherently sequential? Reasons why –Every pop() call fights for top item Reasons why not –Think about it!