שירן חליבה Concurrent Queues

Outline: Some definitions 3 queue implementations : A Bounded Partial Queue An Unbounded Total Queue An Unbounded Lock-Free Queue

Introduction and some definitions: Pools show up in many places in concurrent systems. For example, in many applications, one or more producer threads produce items to be consumed by one or more consumer threads. To allow consumers to keep up, we can place a buffer between the producers and the consumers. Often, pools act as producer–consumer buffers. A pool allows the same item to appear more than once.

Introduction and some definitions cont. A queue is a special kind of pool with FIFO fairness. It provides an enq(x) method that puts item x at one end of the queue, called the tail, and a deq() method that removes and returns the item at the other end of the queue, called the head.

Bounded vs. Unbounded A pool can be bounded or unbounded. Bounded Fixed capacity Good when resources an issue Unbounded Holds any number of objects

Blocking vs. Non-Blocking Problem cases: Removing from empty pool Adding to full (bounded) pool Blocking Caller waits until state changes Non-Blocking Method throws exception

Total vs. Partial Pool methods may be total or partial. A method is total if calls do not wait for certain conditions to become true. For example, a get() call that tries to remove an item from an empty pool immediately returns a failure code or throws an exception. A total interface makes sense when the producer (or consumer) thread has something better to do than wait for the method call to take effect.

Total vs. Partial A method is partial if calls may wait for conditions to hold. For example, a partial get() call that tries to remove an item from an empty pool blocks until an item is available to return. A partial interface makes sense when the producer (or consumer) has nothing better to do than to wait for the pool to become nonempty (or non full).

Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

A Bounded Partial Queue head tail deqLock enqLock Permission to enqueue 8 items permits 8 Lock out other enq() calls Lock out other deq() calls First actual item Sentinel

Enqueuer head tail deqLock enqLock permits 8 Lock enqLock Read permits OK No need to lock tail?

Enqueuer head tail deqLock enqLock permits 8 Enqueue Node 7 getAndDecrement() (Why atomic?)

Enqueuer head tail deqLock enqLock permits 8 Release lock 7 If queue was empty, notify waiting dequeuers

Unsuccesful Enqueuer head tail deqLock enqLock permits 0 Uh-oh Read permits

Dequeuer head tail deqLock enqLock permits 7 Lock deqLock Read sentinel’s next field OK

Dequeuer head tail deqLock enqLock permits 7 Read value

Dequeuer head tail deqLock enqLock permits 7 Make first Node new sentinel

Dequeuer head tail deqLock enqLock permits 7 Release deqLock

Dequeuer head tail deqLock enqLock permits 8 Increment permits (no need lock?) Answer: we had to hold the lock while enqueuing to prevent lots of enqueuers from proceeding without noticing that the capacity had been exceeded. Dequeuers will notice the queue is empty when they observe that the sentinel’s next field is null

Unsuccesful Dequeuer head tail deqLock enqLock permits 8 Read sentinel’s next field uh-oh

Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

The ReentrantLock is a monitor ( The mechanism that Java uses to support synchronization ). Allows blocking on a condition rather than spinning. How do we use it? (* More on monitors: )

Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

Signal Awakens one waiting thread Which will reacquire lock q.signal();

A Monitor Lock Critical Section waiting room Lock() unLock()

Unsuccessful Deq Critical Section waiting room Lock() await() Deq() Oh no, Empty!

Another One Critical Section waiting room Lock() await() Deq() Oh no, Empty!

Enqueur to the Rescue Critical Section waiting room Lock() signalAll() Enq( ) unLock() Yawn!

Monitor Signalling Critical Section waiting room Yawn! Awakend thread might still lose lock to outside contender…

Dequeurs Signalled Critical Section waiting room Found it Yawn!

Dequeurs Signalled Critical Section waiting room Still empty!

Dollar Short + Day Late Critical Section waiting room

Why not signal()?

Lost Wake-Up Critical Section waiting room Lock() signal () Enq( ) unLock() Yawn!

Lost Wake-Up Critical Section waiting room Lock() Enq( ) unLock() Yawn!

Lost Wake-Up Critical Section waiting room Yawn!

Lost Wake-Up Critical Section waiting room Found it

What’s Wrong Here? Critical Section waiting room zzzz….!

Enq Method public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

Cont… public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

The Enq() & Deq() Methods Share no locks That’s good But do share an atomic counter Accessed on every method call That’s not so good Can we alleviate this bottleneck? What is the problem?

Split the Counter The enq() method Decrements only Cares only if value is zero The deq() method Increments only Cares only if value is capacity

Split Counter Enqueuer decrements enqSidePermits Dequeuer increments deqSidePermits When enqueuer runs out Locks deqLock Transfers permits (dequeuer doesn't need permits- check head.next) Intermittent( תקופתי ) synchronization Not with each method call Need both locks! (careful …)

An Unbounded Total Queue Queue can hold an unbounded number of items. The enq() method always enqueues its item. The deq() throws EmptyException if there is no item to dequeue. No deadlock- each method acquires only one lock. Both the enq() and deq() methods are total as they do not wait for the queue to become empty or full.

An Unbounded Total Queue

A Lock-Free Queue Sentinel head tail Extension of the unbounded total queue Quicker threads help the slower threads Each node’s next field is an: AtomicReference The queue itself consists of two AtomicReference fields: head and tail

Compare and Set CAS

LockFreeQueue class

Enqueue head tail Enq( )

Enqueue head tail

Logical Enqueue head tail CAS

Physical Enqueue head tail Enqueue Node CAS

Enqueue These two steps are not atomic The tail field refers to either Actual last Node (good) Penultimate* Node (not so good) Be prepared! (*Penultimate :next to the last)

Enqueue What do you do if you find A trailing tail? Stop and fix it If tail node has non-null next field CAS the queue’s tail field to tail.next

When CASs Fail During logical enqueue Abandon hope, restart During physical enqueue Ignore it (why?)

LockFreeQueue class

Enq() Creates a new node with the new value to be enqueued reads tail, and finds the node that appears to be last checks whether that node has a successor If not - appends the new node by calling compareAndSet() If the compareAndSet() succeeds, the thread uses a second compareAndSet() to advance tail to the new node second compareAndSet() call fails, the thread can still return successfully If the tail node has a successor, then the method tries to “help”other threads by advancing tail to refer directly to the successor before trying again to insert its own node.

Dequeuer head tail Read value

Dequeuer head tail Make first Node new sentinel CAS

What is the problem here?

LockFreeQueue class

Deq() If the queue is nonempty(the next field of the head node is not null), the dequeuer calls compareAndSet() to change head from the sentinel node to its successor before advancing head one must make sure that tail is not left referring to the sentinel node which is about to be removed from the queue test: if head equals tail and the (sentinel) node they refer to has a non-null next field, then the tail is deemed to be lagging behind. deq() then attempts to help make tail consistent by swinging it to the sentinel node’s successor, and only then updates head to remove the sentinel

Summary A thread fails to enqueue or dequeue a node only if another thread’s method call succeeds in changing the reference, so some method call always completes. As it turns out, being lock-free substantially enhances the performance of queue implementations, and the lock-free algorithms tend to outperform the most efficient blocking ones.