1. Distributed Algorithms (22903)
Lock-free stack algorithms
Lecturer: Danny Hendler

2. Wait-freedom vs. lock-freedom
Wait-freedom – each process completes its operation in a finite number of its own steps
Lock-freedom – some process completes its operation after a finite number of steps is taken

3. The compare&swap operation
Comare&swap(b,old,new)
atomically
v read from b
if (v = old) {
b new
return success }
else
return failure;
Supported by most modern multiprocessor architectures 3

4. Treiber’s stack algorithm…
val
val
val
Top
next
next
next
Push(int v, Stack S)
n := new NODE ;create node for new stack item
n.val := v ;write item value
do forever ;repeat until success
node top := S.top
n.next := top ;next points to current (LIFO order)
if compare&swap(S, top, n) ; try to add new item
return ; return if succeeded
end do

5. Treiber’s stack algorithm (cont’d)…
val
val
val
Top
next
next
next
Pop(Stack S)
do forever
top := S.top
if top = null
return empty
if compare&swap(S, top, top.next)
return-val=top.val
free top
return return-val
end do

6. Treiber’s stack algorithm (cont’d)
It is easily seen that the alg is linearizable and lock-free
A disadvantage of the algorithms is that…
It has a sequential bottleneck.

7. … An elimination backoff stack algorithm
(Hendler, Shavit and Yerushalmi, 2004)
Key idea: pairs of push/pop operations may collide and eliminate each other without accessing a central stack.
Central stack …
val
val
val
Top
next
next
next
collision array

8. … Collision scenarios Collision array Central stack push pop push pop
val
val
val
Top
next
next
next

9. Elimination: challenges
Prevent elimination chains: e.g., A collides with B, which collides with C…
Prevent race conditions: e.g., A collides with B, which is already gone…
Collision array
push
pop
Top
val
next …
Central stack

10. Data structures Location array collision array
Each stack operation is represented by a ThreadInfo structure
struct ThreadInfo { id ;the identifier of the thread performing the operation op ;a PUSH/POP opcode cell ;a cell structure spin ; duration to spin }
Struct cell { ;a representation of stack item as in Treiber pnext ;pointer to the next cell
pdata ;stack item }
Location array p1 p2 p3 p4 pn
Thread Info
Thread Info
collision array p1 p7

11. Elimination-backoff stack code
void StackOp(ThreadInfo* p) {
if (TryPerformStackOP(p) == FALSE) ;if operation not applied to central stack
LesOp(p) ;try to eliminate operation by colliding with an opposite-type operation
return
void LesOP(ThreadInfo * p)
while (1)
location[mypid]=p ;announce arrival
pos=GetPosition(p) ;get a random position at the collision array
him=collision[pos] ;read current value of that position
while (!compare&swap(&collision[pos],him,mypid);try to write own ID
him=collision[pos] ;continue till success
if (him != empty) ;if read an ID of another thread
q=location[him] ;read a pointer to the other thread’s info
if (q!=NULL && q->id=him && q->op != p->op) ;if may collide
if (compare&swap(&location[mypid],p,NULL) ;try to prevent unwanted collisions
if (TryCollision(p,q)==true) ;if collided successfully
return ;return code is already at ThreadInfo structure
else goto stack ;try to apply operation to central stack
else FinishCollision(p), return ;extract information and finish
delay (p->spin) ;Wait for other thread to collide with me
if (!compare&swap(&location[mypid],p,NULL) ;if someone collided with me
FinishCollision(p), return;Extract information and finish
stack: if (TryPerformStackOp(p)==TRUE) return ;try to apply operation to central stack

12. Elimination-backoff stack code (cont’d)
void TryCollision(ThreadInfo* p, ThreadInfo *q)
if (p->op==PUSH)
if (compare&swap(&location[him],q,p)) ;give my record to other thread
return TRUE
else
return FALSE
if (compare&swap(&location[him],q,NULL))
p->cell=q->cell ;get pointer to PUSH operation’s cell
void FinishCollision(ThreadInfo* p)
if (p->op==POP)
p->pcell=location[mypid]->pcell
location[mypid]=NULL

13. Elimination-backoff stack code (cont’d)
Why is this implementation linearizable?
Can a record be recycled once popped from stack?

14. Recycling: Simple Solution
Each thread has a free list of unused queue nodes
Allocate node: pop from list
Free node: push onto list
Use CAS for atomicity
Deal with underflow somehow …
The most reasonable solution is to have each thread manage its own pool of unused nodes. When a thread needs a new Node, it pops one from the list. No need for synchronization. When a thread fees A Node, it pushes the newly-freed Node onto the list. What do we do when a thread runs out of nodes? Perhaps we could just malloc() more, or perhaps we can devise a shared pool. Never mind.
© Herlihy-Shavit 2007

15. Why Recycling is Hard Free pool Want to rediret head from grey to red
tail
Want to rediret head
from grey to red
zzz…
Now the green thread goes to sleep, and other threads dequeue the red object and the green object, sending the prior sentinel and prior red Node to the respective free pools of the dequeuing threads.
Free pool
© Herlihy-Shavit 2007

16. Why Recycling is Hard zzz Free pool head tail © Herlihy-Shavit 2007
Despite what you might think, we are perfectly safe, because any thread that tries to CAS the head field must fail, because that field has changed. Now assume that the enough deq() and enq() calls occur such that the original sentinel Node is recyled, and again becomes a sentinel Node for an empty queue.
Free pool
© Herlihy-Shavit 2007

17. Why Recycling is Hard Yawn! Free pool head tail © Herlihy-Shavit 2007
Now the green thread wakes up. It applies CAS to the queue’s head field ..
Free pool
© Herlihy-Shavit 2007

18. Why Recycling is Hard CAS OK, here I go! Free pool head tail
Surprise! It works! The problem is that the bit-wise value of the head field is the same as before, even though its meaning has changed. This is a problem with the way the CAS operation is defined, nothing more.
Free pool
© Herlihy-Shavit 2007

19. Final State What went wrong? Free pool head tail © Herlihy-Shavit 2007
In the end, the tail pointer points to a sentinel node, while the head points to node in some thread’s free list. What went wrong?
What went wrong?
Free pool
© Herlihy-Shavit 2007

20. The Dreaded ABA Problem
head
tail
Head pointer has value A
Thread reads value A
© Herlihy-Shavit 2007

21. Head pointer has value B
Dreaded ABA continued
head
tail
zzz
Head pointer has value B
Node A freed
© Herlihy-Shavit 2007

22. Head pointer has value A again Node A recycled & reinitialized
Dreaded ABA continued
head
tail
Yawn!
Head pointer has value A again
Node A recycled & reinitialized
© Herlihy-Shavit 2007

23. Dreaded ABA continued CAS CAS succeeds because pointer matches
head
tail
CAS succeeds because pointer matches
even though pointer’s meaning has changed
© Herlihy-Shavit 2007

24. The Dreaded ABA Problem
Is a result of CAS() semantics (Sun, Intel, AMD)
Does not arise with Load-Locked/Store-Conditional (IBM)
© Herlihy-Shavit 2007

25. Dreaded ABA – A Solution
Tag each pointer with a counter
Unique over lifetime of node
Pointer size vs word size issues
Overflow?
Don’t worry be happy?
Bounded tags
Other solutions exist (e.g. hazard pointers)
© Herlihy-Shavit 2007