1. The Relative Power of Synchronization Methods
Nir Shavit
Multiprocessor Synchronization
Spring 2003

2. Wait-Free Implementation
Every method call completes in finite number of steps
Implies no mutual exclusion
4-Feb-19
(2)
© 2003 Herlihy and Shavit

3. Wait-Free Constructions
Wait-free atomic registers
From safe registers
Two-threaded FIFO queue
From atomic registers
And indirectly from safe registers
4-Feb-19
© 2003 Herlihy and Shavit

4. Rationale We wanted atomic registers to implement mutual exclusion
So we couldn’t use mutual exclusion to implement atomic registers
But wait, there’s more!
4-Feb-19
© 2003 Herlihy and Shavit

5. Unexpected Delay ??? ??? Swapped out back at 4-Feb-19
(2)
© 2003 Herlihy and Shavit

6. Heterogeneous Architectures
???
???
yawn
286
Pentium
Pentium
4-Feb-19
(1)
© 2003 Herlihy and Shavit

7. Fault-Tolerance
???
???
4-Feb-19
(2)
© 2003 Herlihy and Shavit

8. Basic Questions Might be a good idea in principle But how do you do it
Systematically?
Correctly?
Efficiently?
4-Feb-19
© 2003 Herlihy and Shavit

9. Concurrent FIFO Queue
4-Feb-19
© 2003 Herlihy and Shavit

10. Two-Thread Wait-Free Queue
public class WaitFreeQueue {
int head = 0, tail = 0;
Item[QSIZE] items;
public void enq(Item x) {
while (tail-head == QSIZE); // busy-wait
items[tail++ % QSIZE] = x; }
public Item deq() {
while (tail == head); // busy-wait
return items[head++ % QSIZE]; }}
4-Feb-19
© 2003 Herlihy and Shavit

11. What About Multiple Dequeuers?
4-Feb-19
© 2003 Herlihy and Shavit

12. Grand Challenge Only new aspect Implement a FIFO queue Wait-free
Linearizable
From atomic read-write registers
Multiple dequeuers
4-Feb-19
(1)
© 2003 Herlihy and Shavit

13. Consensus While you are ruminating on the grand challenge…
We will give you another puzzle
Consensus
Pretty important …
4-Feb-19
© 2003 Herlihy and Shavit

14. Consensus: Each Thread has a Private Input19 32 21
4-Feb-19
© 2003 Herlihy and Shavit

15. They Communicate
4-Feb-19
© 2003 Herlihy and Shavit

16. They Agree on Some Thread’s Input19 19 19
4-Feb-19
© 2003 Herlihy and Shavit

17. Formally: Consensus Consistent: all threads decide the same value
Valid: the common decision value is some thread's input
Wait-free: each thread decides after a finite number of steps
4-Feb-19
© 2003 Herlihy and Shavit

18. Theorem: No Consensus from Registers if any Thread can Halt
???
???
4-Feb-19
© 2003 Herlihy and Shavit

19. Proof Strategy Assume otherwise
Reason about the properties of any such protocol
Derive a contradiction
Quod Erat Demonstrandum
4-Feb-19
© 2003 Herlihy and Shavit

20. Wait-Free Computation
A moves
B moves
Either A or B “moves”
Moving means
Register read
Register write
4-Feb-19
© 2003 Herlihy and Shavit

21. The Two-Move Tree Final states Initial state 4-Feb-19
(2)
© 2003 Herlihy and Shavit

22. Decision Values 1 1 1 1
4-Feb-19
© 2003 Herlihy and Shavit

23. Bivalent: Both Possible1 1 1 1
4-Feb-19
© 2003 Herlihy and Shavit

24. Univalent: Single Value Possible1 1 1 1
4-Feb-19
© 2003 Herlihy and Shavit

25. 1-valent: Only 1 Possible1 1 1
4-Feb-19
© 2003 Herlihy and Shavit

26. 0-valent: Only 0 possible1 1 1 1
4-Feb-19
© 2003 Herlihy and Shavit

27. Summary Wait-free computation is a tree Bivalent system states
Outcome not fixed
Univalent states
Outcome is fixed
May not be “known” yet
1-Valent and 0-Valent states
4-Feb-19
© 2003 Herlihy and Shavit

28. Claim Some initial system state is bivalent
If A starts with input 0 and B starts with input 1, then initially A should be able to decide 0 and B should be able to decide 1. Lets see why?
4-Feb-19
© 2003 Herlihy and Shavit

29. A 0-Valent Initial State
All executions lead to decision of 0
4-Feb-19
(2)
© 2003 Herlihy and Shavit
(1)

30. A 0-Valent Initial State
Solo execution by A also decides 0
4-Feb-19
(1)
© 2003 Herlihy and Shavit
(1)

31. A 1-Valent Initial State
All executions lead to decision of 1
4-Feb-19
(2)
© 2003 Herlihy and Shavit
(1)

32. A 1-Valent Initial State
Solo execution by B also decides 1
4-Feb-19
(1)
© 2003 Herlihy and Shavit
(1)

33. A Univalent Initial State?1
Can all executions lead to the same decision?
4-Feb-19
(2)
© 2003 Herlihy and Shavit

34. State is Bivalent 1 Solo execution by A must decide 01
Solo execution by A must decide 0
Solo execution by B must decide 1
4-Feb-19
© 2003 Herlihy and Shavit

35. Critical States critical 0-valent 1-valent 4-Feb-19
(3)
© 2003 Herlihy and Shavit
(3)

36. Critical States Starting from a bivalent initial state
The protocol can reach a critical state
Otherwise we could stay bivalent forever
And the protocol is not wait-free
4-Feb-19
© 2003 Herlihy and Shavit

37. From a Critical State c 0-valent 1-valent
If A goes first, protocol decides 0
If B goes first, protocol decides 1
4-Feb-19
© 2003 Herlihy and Shavit

38. Model Dependency So far, memory-independent! True for Registers
Message-passing
Carrier pigeons
Any kind of asynchronous computation
4-Feb-19
© 2003 Herlihy and Shavit

39. What are the Threads Doing?
Reads and/or writes
To same/different registers
4-Feb-19
© 2003 Herlihy and Shavit

40. Possible Interactions
A reads x
x.read()
y.read()
x.write()
y.write() ?
A reads y
4-Feb-19
© 2003 Herlihy and Shavit

41. States look the same to A
Reading Registers
A runs solo, decides 0 c
B reads x
A runs solo, decides 1 1
Contradiction
States look the same to A
4-Feb-19
© 2003 Herlihy and Shavit

42. Possible Interactions
x.read()
y.read()
x.write()
y.write() no ?
4-Feb-19
© 2003 Herlihy and Shavit

43. Writing Distinct Registers
A writes y
B writes x
A writes y
B writes x
Contradiction 1
The song remains the same
4-Feb-19
© 2003 Herlihy and Shavit

44. Possible Interactions
x.read()
y.read()
x.write()
y.write() no ?
4-Feb-19
© 2003 Herlihy and Shavit

45. Writing Same Registersc
B writes x
A writes x
A runs solo, decides 0
A writes x
Contradiction
A runs solo, decides 1 1
States look the same to A
4-Feb-19
© 2003 Herlihy and Shavit

46. That’s All, Folks! no x.read() y.read() x.write() y.write() 4-Feb-19
© 2003 Herlihy and Shavit

47. Theorem
It is impossible to solve consensus using read/write atomic registers
Assume protocol exists
It has a bivalent initial state
Must be able to reach a critical state
Case analysis of interactions
Reads vs others
Writes vs writes
4-Feb-19
© 2003 Herlihy and Shavit

48. What Does Consensus have to do with Concurrent Objects?
4-Feb-19
© 2003 Herlihy and Shavit

49. Always announce intentions Figure out which one won
The Consensus Object
public abstract class Consensus
implements Consensus {
private Object[] announce;
public ConsensusImpl(int n) {
announce = new Object[n]; }
public void propose(Object value) {
announce[Thread.myIndex()] = value;
abstract public Object decide(); }}
Always announce intentions
Figure out which one won
4-Feb-19
(4)
© 2003 Herlihy and Shavit
(4)

50. Can FIFO Queue Implement A Consensus Object?
proposed array
FIFO Queue
with red and black balls 8
4-Feb-19
© 2003 Herlihy and Shavit

51. FIFO Queue with red and black balls
A Consensus Protocol
Preferred array
FIFO Queue with red and black balls 8
Coveted red ball
Dreaded black ball
4-Feb-19
© 2003 Herlihy and Shavit

52. Protocol: Write Value to Array1
4-Feb-19
© 2003 Herlihy and Shavit

53. Protocol: Take Next Item from Queue8 1
4-Feb-19
© 2003 Herlihy and Shavit

54. Protocol: Take Next Item from Queue
I got the dreaded black ball, so I will decide the other’s value from the array
I got the coveted red ball, so I will decide my value 1 8
4-Feb-19
© 2003 Herlihy and Shavit

55. Consensus Using FIFO Queue
public class QueueConsensus extends Consensus { Queue queue;
public QueueConsensus() {
queue = new Queue();
queue.enq(“red ball");
queue.enq(“black ball");}
public Object decide() {
String status = queue.deq();
int i = Thread.myIndex();
if (status == “red ball")
return proposed[i];
else
return proposed[1-i];}
Initialize Queue
i won
i lost
4-Feb-19
© 2003 Herlihy and Shavit

56. Why does this Work? If one thread gets the red ball
Then the other gets the black ball
Winner can take her own value
Loser can find winner’s value in array
Because threads write array
Before dequeueing from queue
4-Feb-19
© 2003 Herlihy and Shavit

57. Implication We can solve 2-thread consensus using only
A two-dequeuer queue, and
A couple of atomic registers
4-Feb-19
© 2003 Herlihy and Shavit

58. Implications contradiction Given Assume there exists
A consensus protocol from queue and registers
Assume there exists
A queue implementation from atomic registers
Substitution yields:
A wait-free consensus protocol from atomic registers
contradiction
4-Feb-19
(1)
© 2003 Herlihy and Shavit
(1)

59. Corollary It is impossible to implement
a two-dequeuer wait-free FIFO queue
from read/write memory.
4-Feb-19
© 2003 Herlihy and Shavit

60. Consensus Numbers An object has consensus number n If it can be used
Together with atomic read/write registers
To implement n-thread consensus
But not (n+1)-thread consensus
4-Feb-19
© 2003 Herlihy and Shavit

61. Consensus Numbers Theorem
Atomic read/write registers have consensus number 1
4-Feb-19
© 2003 Herlihy and Shavit

62. Consensus Numbers
Consensus numbers are a useful way of measuring synchronization power
Theorem
If you can implement X from Y
And X has consensus number c
Then Y has consensus number at least c
4-Feb-19
© 2003 Herlihy and Shavit

63. Synchronization Speed Limit
Conversely
If X has consensus number c
And Y has consensus number d < c
Then there is no way to construct a wait-free implementation of X by Y
This theorem will be very useful
Unforeseen practical implications!
4-Feb-19
© 2003 Herlihy and Shavit

64. Earlier Grand Challenge
Snapshot means
Write any array element
Read multiple array elements atomically
What about the dual problem:
Write multiple array elements atomically
Scan any array elements
Call this problem multiple assignment
4-Feb-19
© 2003 Herlihy and Shavit

65. Multiple Assignment Theorem
No protocol
Using only read/write registers
Implements multiple assignment
Weird or what?
Single write/multiple read OK
Multi write/multiple read impossible
In fact: Multi write/single read also impossible!!!
4-Feb-19
(1)
© 2003 Herlihy and Shavit
(1)

66. Proof Strategy Given a 3-element array Then we can solve 2-consensus
A writes atomically to slots 0 and 1
B writes atomically to slots 1 and 2
Any thread can scan any set of locations
Then we can solve 2-consensus
So multiple assignment has consensus number at least 2
No read/write register implementation
4-Feb-19
© 2003 Herlihy and Shavit
(1)

67. Initially A B Writes to 0 and 1 Writes to 1 and 2 4-Feb-19
© 2003 Herlihy and Shavit

68. Thread A wins if it sees A B
4-Feb-19
© 2003 Herlihy and Shavit
(1)

69. Thread A wins if it sees A B
4-Feb-19
(1)
© 2003 Herlihy and Shavit

70. Thread A loses if it seesB
4-Feb-19
(1)
© 2003 Herlihy and Shavit

71. Figure out which one won
Multi-Consensus
class MultiConsensus {
MultiAssign multi;
public Object decide() {
int i = Thread.myIndex(); // my index
int j = 1 - i; // other index
multi.assign(i, input, i+1, input);
Object[] result = multi.scan();
if (result[(j + 1) % 3] == null ||
(result[j] == result[(j + 1) % 3]))
return proposed[i];
else
return proposed[j]; }
Write/Scan locations
Figure out which one won
4-Feb-19
(4)
© 2003 Herlihy and Shavit
(4)

72. Summary
If a thread can assign atomically to 2 out of 3 array locations
Then we can solve 2-consensus
Implying
No wait-free implementation
From read/write registers
4-Feb-19
© 2003 Herlihy and Shavit

73. Read-Modify-Write Method takes 2 arguments: Method call: Variable x
Function f
Method call:
Returns value of x
Replaces x with f(x)
4-Feb-19
© 2003 Herlihy and Shavit

74. Read-Modify-Write Return prior value Apply function f()
public abstract class RMW {
private int value;
public void synchronized
rmw(function f) {
int prior = this.value;
this.value = f(this.value);
return prior; }
Return prior value
Apply function f()
4-Feb-19
(1)
© 2003 Herlihy and Shavit

75. f(v)=v is the identity function
Example: Read
public abstract class RMW {
private int value;
public void synchronized read() {
int prior = this.value;
this.value = this.value;
return prior; }
f(v)=v is the identity function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

76. F(v)=1 is constant function
Example: test&set
public abstract class RMW {
private int value;
public synchronized void TAS() {
int prior = this.value;
this.value = 1;
return prior; }
F(v)=1 is constant function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

77. f(v)=v+1 is increment function
Example: fetch&inc
public abstract class RMW {
private int value;
public synchronized void FAI() {
int prior = this.value;
this.value = this.value+1;
return prior; }
f(v)=v+1 is increment function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

78. F(v,x)=v+x is the addition function
Example: fetch&add
public abstract class RMW {
private int value;
public synchronized void FAA(int x) {
int prior = this.value;
this.value = this.value+x;
return prior; }
F(v,x)=v+x is the addition function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

79. F(v,x) = x is a constant function
Example: swap
public abstract class RMW {
private int value;
public synchronized void swap(int x) {
int prior = this.value;
this.value = x;
return prior; }
F(v,x) = x is a constant function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

80. Example: compare&swap
public abstract class RMW {
private int value;
public synchronized void
CAS(int old, int new) {
int prior = this.value;
if (this.value == old)
this.value = new;
return prior; }
f(old,new) is a complex function
4-Feb-19
(1)
© 2003 Herlihy and Shavit

81. Definition
A RMW is non-trivial if there exists a value v Such that v ≠ f(v)
read is trivial
FAA is not
4-Feb-19
© 2003 Herlihy and Shavit

82. Theorem Any non-trivial RMW object has consensus number at least 2
Implies no wait-free implementation of RMW registers from read/write registers
Hardware RMW instructions not just a convenience
4-Feb-19
© 2003 Herlihy and Shavit

83. Reminder All objects that inherit from Consensus have propose method
which just stores input into this.announce[i]
4-Feb-19
© 2003 Herlihy and Shavit

84. No, return other’s input
Proof
Initialized to v
public class RMWConsensus
implements Consensus {
private RMW r;
public Object decide() {
int i = Thread.myIndex();
if (r.rmw(f) == v)
return this.announce[i];
else
return this.announce[1-i]; }}
Am I first?
Yes, return my input
No, return other’s input
4-Feb-19
(4)
© 2003 Herlihy and Shavit

85. Proof We have displayed A two-thread consensus protocol
Using any non-trivial RMW object
4-Feb-19
© 2003 Herlihy and Shavit

86. Interfering RMW
Let F be a set of functions such that for all fi and fj, either
They commute: fi(fj(v))=fj(fi(v))
They overwrite: fi(fj(v))=fi(v)
Claim: Any such set of RMW objects has consensus number exactly 2
4-Feb-19
© 2003 Herlihy and Shavit

87. Examples Test-and-Set f(v)=1 Swap f(v,x)=x Fetch-and-inc f(v)=v+1
Overwrite fi(fj(v))=fi(v)
Overwrite fi(fj(v))=fi(v)
Commute fi(fj(v))= fj(fi(v))
4-Feb-19
© 2003 Herlihy and Shavit

88. Meanwhile Back at the Critical State
A about to apply fA
B about to apply fB c
0-valent
1-valent
4-Feb-19
© 2003 Herlihy and Shavit

89. Maybe the Functions Commute
A applies fA
B applies fB
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
4-Feb-19
© 2003 Herlihy and Shavit

90. Maybe the Functions Commute
A applies fA
B applies fB
These states look the same to C
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
4-Feb-19
© 2003 Herlihy and Shavit

91. Maybe the Functions Overwrite
A applies fA
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
4-Feb-19
© 2003 Herlihy and Shavit

92. Maybe the Functions Overwrite
These states look the same to C c
A applies fA
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
4-Feb-19
© 2003 Herlihy and Shavit

93. Impact Many early machines provided these “weak” RMW instructions
Test-and-set (IBM 360)
Fetch-and-add (NYU Ultracomputer)
Swap (Original SPARCs)
We now understand their limitations
But why do we want consensus anyway?
4-Feb-19
© 2003 Herlihy and Shavit

94. Example: compare&swap
public abstract class RMW {
private int value;
public void CAS(int old, int new) {
int prior = this.value;
if (this.value == old)
this.value = new;
return prior; }
4-Feb-19
(1)
© 2003 Herlihy and Shavit

95. CAS Has Unbounded Consensus Number
Initialized to -1
public class RMWConsensus
implements Consensus {
private RMW r;
public Object decide() {
int i = Thread.myIndex();
int j = r.CAS(-1,i);
if (j == -1)
return this.announce[i];
else
return this.announce[j]; }}
Am I first?
Yes, return my input
No, return other’s input
4-Feb-19
(4)
© 2003 Herlihy and Shavit

96. The Consensus Hierarchy
1 Read/Write Registers, Snapshots…
2 T&S, F&I, Swap… .
∞ CAS,…
4-Feb-19
© 2003 Herlihy and Shavit

97. Multiple Assignment Atomic k-assignment
Solves consensus for 2k-2 threads
Every even consensus number has an object (can be extended to odd numbers)
4-Feb-19
© 2003 Herlihy and Shavit

98. Lock-Free Implementations
Infinitely often some method call completes in a finite number of steps
Stalinistic real world systems approach
Implies no mutual exclusion
4-Feb-19
(2)
© 2003 Herlihy and Shavit

99. Lock-Free Implementations
Lock-free = Wait-free for finite executions
Lock-free consensus is just as impossible
All the results we presented hold for lock-free algorithms also.
4-Feb-19
(2)
© 2003 Herlihy and Shavit

100. There is More: Universality
Consensus is universal
From n-thread consensus
Wait-free/Lock-free
Linearizable
n-threaded
Implementation
Of any sequential object
4-Feb-19
© 2003 Herlihy and Shavit