1. שירן חליבה Concurrent Queues

2. Outline: Some definitions 3 queue implementations : A Bounded Partial Queue An Unbounded Total Queue An Unbounded Lock-Free Queue

3. Introduction and some definitions: Pools show up in many places in concurrent systems. For example, in many applications, one or more producer threads produce items to be consumed by one or more consumer threads. To allow consumers to keep up, we can place a buffer between the producers and the consumers. Often, pools act as producer–consumer buffers. A pool allows the same item to appear more than once.

4. Introduction and some definitions cont. A queue is a special kind of pool with FIFO fairness. It provides an enq(x) method that puts item x at one end of the queue, called the tail, and a deq() method that removes and returns the item at the other end of the queue, called the head.

5. Bounded vs. Unbounded A pool can be bounded or unbounded. Bounded Fixed capacity Good when resources an issue Unbounded Holds any number of objects

6. Blocking vs. Non-Blocking Problem cases: Removing from empty pool Adding to full (bounded) pool Blocking Caller waits until state changes Non-Blocking Method throws exception

7. Total vs. Partial Pool methods may be total or partial. A method is total if calls do not wait for certain conditions to become true. For example, a get() call that tries to remove an item from an empty pool immediately returns a failure code or throws an exception. A total interface makes sense when the producer (or consumer) thread has something better to do than wait for the method call to take effect.

8. Total vs. Partial A method is partial if calls may wait for conditions to hold. For example, a partial get() call that tries to remove an item from an empty pool blocks until an item is available to return. A partial interface makes sense when the producer (or consumer) has nothing better to do than to wait for the pool to become nonempty (or non full).

9. Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

10. Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

11. A Bounded Partial Queue head tail deqLock enqLock Permission to enqueue 8 items permits 8 Lock out other enq() calls Lock out other deq() calls First actual item Sentinel

12. Enqueuer head tail deqLock enqLock permits 8 Lock enqLock Read permits OK No need to lock tail?

13. Enqueuer head tail deqLock enqLock permits 8 Enqueue Node 7 getAndDecrement() (Why atomic?)

14. Enqueuer head tail deqLock enqLock permits 8 Release lock 7 If queue was empty, notify waiting dequeuers

15. Unsuccesful Enqueuer head tail deqLock enqLock permits 0 Uh-oh Read permits

16. Dequeuer head tail deqLock enqLock permits 7 Lock deqLock Read sentinel’s next field OK

17. Dequeuer head tail deqLock enqLock permits 7 Read value

18. Dequeuer head tail deqLock enqLock permits 7 Make first Node new sentinel

19. Dequeuer head tail deqLock enqLock permits 7 Release deqLock

20. Dequeuer head tail deqLock enqLock permits 8 Increment permits (no need lock?) Answer: we had to hold the lock while enqueuing to prevent lots of enqueuers from proceeding without noticing that the capacity had been exceeded. Dequeuers will notice the queue is empty when they observe that the sentinel’s next field is null

21. Unsuccesful Dequeuer head tail deqLock enqLock permits 8 Read sentinel’s next field uh-oh

22. Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

23. The ReentrantLock is a monitor ( The mechanism that Java uses to support synchronization ). Allows blocking on a condition rather than spinning. How do we use it? (* More on monitors: http://www.artima.com/insidejvm/ed2/threadsynch.html )

24. Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

25. Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

26. Signal Awakens one waiting thread Which will reacquire lock q.signal();

27. A Monitor Lock Critical Section waiting room Lock() unLock()

28. Unsuccessful Deq Critical Section waiting room Lock() await() Deq() Oh no, Empty!

29. Another One Critical Section waiting room Lock() await() Deq() Oh no, Empty!

30. Enqueur to the Rescue Critical Section waiting room Lock() signalAll() Enq( ) unLock() Yawn!

31. Monitor Signalling Critical Section waiting room Yawn! Awakend thread might still lose lock to outside contender…

32. Dequeurs Signalled Critical Section waiting room Found it Yawn!

33. Dequeurs Signalled Critical Section waiting room Still empty!

34. Dollar Short + Day Late Critical Section waiting room

35. Why not signal()?

36. Lost Wake-Up Critical Section waiting room Lock() signal () Enq( ) unLock() Yawn!

37. Lost Wake-Up Critical Section waiting room Lock() Enq( ) unLock() Yawn!

38. Lost Wake-Up Critical Section waiting room Yawn!

39. Lost Wake-Up Critical Section waiting room Found it

40. What’s Wrong Here? Critical Section waiting room zzzz….!

41. Enq Method public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

42. Cont… public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

43. The Enq() & Deq() Methods Share no locks That’s good But do share an atomic counter Accessed on every method call That’s not so good Can we alleviate this bottleneck? What is the problem?

44. Split the Counter The enq() method Decrements only Cares only if value is zero The deq() method Increments only Cares only if value is capacity

45. Split Counter Enqueuer decrements enqSidePermits Dequeuer increments deqSidePermits When enqueuer runs out Locks deqLock Transfers permits (dequeuer doesn't need permits- check head.next) Intermittent( תקופתי ) synchronization Not with each method call Need both locks! (careful …)

46. An Unbounded Total Queue Queue can hold an unbounded number of items. The enq() method always enqueues its item. The deq() throws EmptyException if there is no item to dequeue. No deadlock- each method acquires only one lock. Both the enq() and deq() methods are total as they do not wait for the queue to become empty or full.

47. An Unbounded Total Queue

48. A Lock-Free Queue Sentinel head tail Extension of the unbounded total queue Quicker threads help the slower threads Each node’s next field is an: AtomicReference The queue itself consists of two AtomicReference fields: head and tail

49. Compare and Set CAS

50. LockFreeQueue class

51. Enqueue head tail Enq( )

52. Enqueue head tail

53. Logical Enqueue head tail CAS

54. Physical Enqueue head tail Enqueue Node CAS

55. Enqueue These two steps are not atomic The tail field refers to either Actual last Node (good) Penultimate* Node (not so good) Be prepared! (*Penultimate :next to the last)

56. Enqueue What do you do if you find A trailing tail? Stop and fix it If tail node has non-null next field CAS the queue’s tail field to tail.next

57. When CASs Fail During logical enqueue Abandon hope, restart During physical enqueue Ignore it (why?)

58. LockFreeQueue class

59. Enq() Creates a new node with the new value to be enqueued reads tail, and finds the node that appears to be last checks whether that node has a successor If not - appends the new node by calling compareAndSet() If the compareAndSet() succeeds, the thread uses a second compareAndSet() to advance tail to the new node second compareAndSet() call fails, the thread can still return successfully If the tail node has a successor, then the method tries to “help”other threads by advancing tail to refer directly to the successor before trying again to insert its own node.

60. Dequeuer head tail Read value

61. Dequeuer head tail Make first Node new sentinel CAS

62. What is the problem here?

63. LockFreeQueue class

64. Deq() If the queue is nonempty(the next field of the head node is not null), the dequeuer calls compareAndSet() to change head from the sentinel node to its successor before advancing head one must make sure that tail is not left referring to the sentinel node which is about to be removed from the queue test: if head equals tail and the (sentinel) node they refer to has a non-null next field, then the tail is deemed to be lagging behind. deq() then attempts to help make tail consistent by swinging it to the sentinel node’s successor, and only then updates head to remove the sentinel

65. Summary A thread fails to enqueue or dequeue a node only if another thread’s method call succeeds in changing the reference, so some method call always completes. As it turns out, being lock-free substantially enhances the performance of queue implementations, and the lock-free algorithms tend to outperform the most efficient blocking ones.