1 ZYZZYVA: SPECULATIVE BYZANTINE FAULT TOLERANCE R.Kotla, L. Alvisi, M. Dahlin, A. Clement and E. Wong U. T. Austin Best Paper Award at SOSP 2007

2 Motivation Why implement Byzantine Fault-Tolerant replication? –Increasing value of data and decreasing cost of hardware –More non-stop-fail behaviors than believed –BFT is becoming cheaper –Cost of 3-way non-BFT replication close to cost of BFT replication

3 Zyzzyva (I) Uses speculation to reduce the cost of BFT replication –Primary replica proposes order of client requests to all secondary replicas ( standard ) –Secondary replicas speculatively execute the request without going through an agreement protocol to validate that order ( new idea )

4 Zyzzyva (II) As a result –States of correct replicas may diverge –Replicas may send diverging replies to client Zyzzyva’s solution –Clients detect inconsistencies –Help convergence of correct replicas to a single total ordering of requests –Reject inconsistent replies

5 How? Clients observe a replicated state machine Replies contain enough information to let clients ascertain if the replies and the history are stable and guaranteed to be eventually committed Replicas have checkpoints

6 Byzantine agreement (I) No solution for less than four entities

7 Byzantine agreement (II) To achieve agreement in the presence of f failed nodes (“traitors”) we need –3 f + 1 entities

8 Practical BFT (I) Practical Byzantine Fault-Tolerant protocol (PBFT) [Castro and Liskov 1999]

9 Practical BFT (II) Replicas decide on correct ordering

10 Practical BFT (III) 1.Client sends signed request to primary replica 2.Primary assigns a sequence number to the request and sends to all other replicas a PRE-PREPARE message 3.Secondary replicas validate the message and send a PREPARE message to all replicas 4. Replicas that can collect 2 f PREPARE messages send a COMMIT message to all replicas 5. Replicas that can collect 2 f+ 1 COMMIT message send a REPLY to the client

11 A shortened version Faster agreement is achieved thanks to a more complex view change protocol

12 The explanation (I) " No replicated service that uses the traditional view change protocol can be live without an agreement protocol that includes both the prepare and commit full exchanges" "The traditional view change protocol lets correct replicas commit to a view change and become silent in a view without any guarantee that their action will lead to the view change."

13 The explanation (II) Zyzzyva –Adds an extra phase to its view change protocol –Guarantees that a correct replica will not abandon a view unless every other correct replica does it

14 Zyzzyva Agreement (I) Common case: no faulty replicas

15 Explanations Secondary replicas assume that –Primary replica gave the right ordering –All secondary replicas will participate in transaction Initiate speculative execution Client receives 3 f + 1 mutually consistent responses

16 Zyzzyva Agreement (II) With a faulty replica

17 Explanations (I) Client receives 3 f mutually consistent responses Gathers at least 2 f + 1 mutually consistent responses Distributes a commit certificate to the replicas Once at least 2 f + 1 replicas acknowledge receiving a commit certificate, the client considers the request completed

18 Explanations (II) If enough secondary replicas suspect that the primary replica is faulty, a view change is initiated and a new primary elected

19 Comparison with traditional solutions

20 State maintained at each replica

21 Explanations (I) Each replica maintains –A history of the requests it has executed –A copy of the max commit certificate it has received Let it distinguish between committed history and speculative history

22 Explanations (II) Each replica constructs a checkpoint every CP_INTERVAL requests It maintains one stable checkpoint with a corresponding stable application state snapshot It might also have up to one speculative checkpoint with its corresponding speculative application state snapshot

23 Explanations (III) Checkpoints and application state become committed through a process similar to that of earlier BFT agreement protocols –Replicas send signed checkpoint messages to all replicas when they generate a tentative checkpoint –Commit checkpoint after they collect f + 1 signed matching checkpoint messages

24 View change sub-protocol (I)

25 Explanations Two-phase protocol Elects a new primary Guarantees that it will not introduce any changes in a history that has already completed at a correct client

26 Performance: throughput

27 Comments Zyzzyva-5 is a special version of Zyzziva requiring more replicas but having a lower overhead

28 Performance: latency

29 Scalability: peak throughputs

30 CONCLUSIONS Systematically exploiting speculative execution results in a protocol much faster than conventional BFT agreement protocols. Observe that Zyzzyva is optimized for the most frequent case but provides the correct result in all cases A good rule to follow