1. When Is Agreement Possible
When Is Agreement Possible? CS 188 Distributed Systems February 24, 2015

2. Introduction Basics of agreement protocols
Impossibility of agreement in asynchronous system with failures
When is agreement possible?

3. Basics of Agreement Protocols
What is agreement?
What are the necessary conditions for agreement?

4. What Do We Mean By Agreement?
In simplest case, can n processors agree that a variable takes on value 0 or 1?
Only non-faulty processors need agree
More complex agreements can be built from this simple agreement

5. Conditions for Agreement Protocols
Consistency
All participants agree on same value and decisions are final
Validity
Participants agree on a value at least one of them wanted
Termination
All participants choose a value in a finite number of steps

6. Impossibility of Agreement in Async System With Failures
Assume a reliable, but asynchronous, message passing system
Any message may face arbitrary delays
Can a set of processors reach agreement if one of the processors fails?

7. Agreement Isn’t Always Possible
In the general case for arbitrary systems
Adding some special properties to the system may change that result
But without those properties, provably impossible
A result sometimes abbreviated FLP
For Fischer, Lynch, and Patterson, who proved it

8. Model of the System The system consists of n processors
The goal is for all non-faulty processors to agree on value 0 or 1
Rule out the trivial case of always agreeing on 0 (or 1)
Agreement depends on protocol, initial state, and inputs to each processor

9. Bivalent and Univalent States
A bivalent state is a system state that could lead to either value being decided
A univalent state can only lead to one of the values being decided
0-valent or 1-valent
Valency must take allowable failures into account!

10. System Configuration Processors have internal state
State of network is the set of messages sent, but not yet received
Event e is the receipt of message m by a processor
Which can lead to sending one or more new messages
Events are deterministic
A schedule is a sequence of events

11. Proving the Result Let’s assume the result is false
That we can reach agreement with one failure in these conditions
Use an adversarial model
Within rules of behavior, assume adversary can force any legal event
Look for contradictions

12. What Can the Adversary Do?
Force any processor to perform an event at any moment
Choose any message to be delivered to any processor when it requests a message
Delay any message arbitrarily long
Once, it can kill one processor permanently

13. The Necessity of Bivalency
There has to be an initial bivalent configuration for the system
Why?
If all processors started with value 1, the system would decide 1
If all processors started with value 0, the system would decide 0

14. Intermediate Initial States
If some processors start with value 0 and some with value 1
Some initial states lead to result 1
Some initial states lead to result 0
All initial states lead to one or the other
So there is a 1-valent initial state that differs from a 0-valent initial state by one processor’s initial value

15. A Graphical Representation
What’s in these states?
State x
State y
Node 1:0
Node 2:1
Node 3: 1 .
Node N: 0
Node 1:0
Node 2:1
Node 3: 1 .
Node N: 1
They differ in only one value
0-valent initial states
1-valent initial states

16. Why Does This Imply Bivalence?
What if that one differing processor is the processor that fails?
The system must still reach agreement from the remaining states
Which are identical, now
But on what value?

17. Is This Possible? State x Does the system decide on 1?
Looks like x and y must be bivalent
Does the system decide on 0?
State x
State y
Node 1:0
Node 2:1
Node 3: 1 .
Node N: 0
Node 1:0
Node 2:1
Node 3: 1 .
Node N: 1
Then State x wasn’t 0-valent, after all
Then State y wasn’t 1-valent, after all
0-valent initial states
1-valent initial states

18. So What? So there has to be at least one bivalent initial state
Why’s that so bad?
If the system never leaves a bivalent state, it never makes a decision
We must show our adversary can’t perpetually force bivalency

19. The Persistence of Bivalency
Let’s assume bivalency doesn’t persist
At some point, some bivalent state must transition to a univalent state
Implying at least two events
One to go to 0-valent
One to go to 1-valent
With no events leading to bivalent states

20. A Graphical RepresentationD e’ D’
Remember, these events are each delivery of a message
So m and m’ must have been in the message delivery system state simultaneously

21. Looking Closely at Events e and e’
What would happen if we executed e first, then e’?
What would happen if we executed them in the opposite order?
Well, why should I care?
Would executing them in either order lead to the same state?
If so, there’s a contradiction

22. Order of Events e and e’ C e D e’ D’ e’ e

23. Why Should They Lead to the Same State?
What if e and e’ occur on different processors?
Then they’re independent events
So they should produce the same result if executed in either order
So e and e’ could not have occurred on different processors

24. Could the Events Occur on the Same Processor P?
If e was first, the state became 0-valent
If e’ was first, the state became 1-valent
But what if P then fails?
Since the event happened only at P, only P sees the effects
So we’re still in a bivalent state

25. Recapitulating the Argument
It’s possible to start in a bivalent state
There must be some point at some processor P at which the bivalent state changes to univalent
If P fails before anyone knows the valency, the system becomes bivalent
And can never settle to univalency
Perpetual bivalency implies no agreement

26. When Is Agreement Possible?
Didn’t we show in the last class that we can reach agreement if less than 1/3 of our processors are faulty?
Yes, but only if the message passing system is synchronous
Whether agreement is possible in a system depends on certain parameters

27. Parameters for Agreement In Distributed Systems
Synchronous vs. asynchronous processors
Bounded vs. unbounded communications delay
Ordered vs. unordered messages
Point-to-point vs. broadcast communications

28. Synchronous vs. Asynchronous Processors
Synchronous processors imply that all processors make progress predictably
More precisely, there is a constant s such that
for every s+1 steps taken by Pi
all Pj will take at least one step

29. Bounded vs. Unbounded Communications Delay
Delay is bounded if and only if all messages arrive at their destination within t steps
Implies no lost messages
Doesn’t imply messages arrive in the order sent

30. Ordered vs. Unordered Messages
Messages are ordered if they are received in the same real time order as their sending
Using true real time
In some cases, merely receiving all messages in same order at all processors is enough

31. Point-to-Point vs. Broadcast Communications
Point-to-point communications means a given message sent by Pi is seen only by its destination Pj
Broadcast communications mean that Pi can send a message to all other processors in a single atomic step
Most typically by hardware broadcast

32. So, When Can We Reach Agreement?
Case 1: Processors are synchronous and communications is bounded
Case 2: Messages are ordered and the transmission medium is broadcast
Case 3: Processors are synchronous and messages are ordered
And that’s it
(Case 1 covers Byzantine agreement)

33. What Does This Result Mean?
For practical systems we really build
Not that we can never reach agreement
Good systems almost always do
But that we generally can’t guarantee it
Which implies that our systems should tolerate disagreements
At some times
Under some conditions

34. When Is Disagreement OK?
For preference, when it doesn’t matter
E.g., when reasonable results possible even without agreement
Or when it eventually works itself out
With possible inconsistencies in the meantime
Or, at worst, when it is visible to people who can fix it

35. When Is Disagreement Not OK?
When the consequences of disagreement are dire
When it results in unfixable problems
When its consequences are invisible, but relevant
Unfortunately, we don’t always get to choose when we can avoid it

36. Minimizing Chances of Disagreement
Understand when agreement is most critical
In those cases, use protocols that are less likely to fail on agreement
Which usually have heavy expenses
So don’t always use them

37. A Classification of Faults
More detailed than previously discussed
Produced by fault-tolerant computing community
Divides faults into classes
Stronger class is subset of weaker class

38. An Ordered Fault Classification
Byzantine
Authenticated Byzantine
Incorrect Computation
Timing
Omission
Crash
Fail Stop

39. Fail Stop Faults A processor ceases operation
But informs other processors in computation that it has stopped
Relatively easy to deal with

40. Crash Fault A processor crashes or loses internal state and halts
Without notification to anyone else
Hard to distinguish from a really slow processor

41. Omission Faults A processor fails to do something in time
Like respond to a message
But otherwise it may still be operating correctly
Or it may have crashed

42. Timing Fault
A processor completes a task before or after the window when it should
Or never
A late acknowledgement to a message, e.g.

43. Incorrect Computation Fault
A processor fails to produce the correct results for a given set of input
Which could be merely not producing the results soon enough
Or could be sending back trash

44. Authenticated Byzantine Fault
Processor performs an arbitrary or malicious fault
But authentication mechanisms note any alterations made to others’ messages

45. Byzantine Fault Any and every fault
Having arbitrarily bad consequences
Possibly working in combination with other faults to produce really bad results
In this classification, all other faults are subclasses of Byzantine faults