1. EEC 693/793 Special Topics in Electrical Engineering Secure and Dependable Computing Lecture 16 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University wenbing@ieee.org

2. 2 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Outline Reminder: –Midterm #2, May 1, 4-6pm –May 3, no class –Project presentation: May 8 4-8pm, attendance mandatory –Project report due: May 8 midnight Review –Byzantine general problem Practical Byzantine fault tolerance –By Miguel Castro and Barbara Liskov, OSDI’99 –http://www.pmg.csail.mit.edu/papers/osdi99.pdf

3. 3 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Byzantine Generals Problem A commanding general must send an order to his n-1 lieutenants such that –IC1. All loyal lieutenants obey the same order –IC2. If the commanding general is loyal, then every loyal lieutenant obeys the order he sends

4. 4 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Byzantine Agreement Protocol Round 1: the commander sends a value to each of the lieutenants Round 2: each of the lieutenants sends the value it received to its peers At the end of round 2, each lieutenant check to see if there is a majority opinion (attack or retreat). We have a solution if there is Question is: Can you find a counter example to show that the above protocol does not work if f>=2?

5. 5 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Introduction to BFT Paper The growing reliance of industry and government on online information services Malicious attacks become more serious and successful More software errors due to increased size and complexity of software This paper presents “practical” algorithm for state machine replication that works in asynchronous systems like the Internet

6. 6 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Assumptions Asynchronous distributed system The network may fail to deliver, delay, duplicate or deliver them out of order Faulty nodes may behave arbitrarily Independent node failures The adversary cannot delay correct nodes indefinitely All messages are cryptographically signed by their sender and these signatures cannot be subverted by the adversary

7. 7 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Service Properties A (deterministic) service is replicated among ≥ 3f+1 processors. Resilient to ≤ f failures Safety: All replicas guaranteed to process the same requests in the same order Liveness: Clients eventually receive replies to their requests

8. 8 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Optimal Resiliency Imagine non-faulty processors trying to agree upon a piece of data by telling each other what they believe the data to be A non-faulty processor must be sure about a piece of data before it can proceed f replicas may refuse to send messages, so each processor must be ready to proceed after having received ( n-1)-f messages –Total of n-1 other replicas

9. 9 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Optimal Resiliency But what if f of the ( n-1)-f messages come from faulty replicas? To avoid confusion, the majority of messages must come from non-faulty nodes, i.e, ( n-f-1)/2 ≥ f => Need a total of ≥3f+1 replicas

10. 10 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao BFT Algorithm in a Nutshell Client Primary Backup f + 1 Match (OK)

11. 11 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Replicas and Views R0 R1 R2 R|R-1| Set of replicas (R): |R| ≥ 3f + 1 For view v: primary p is assigned such that p= v mod |R| R0 View 0 R0 R1 View 1 ………

12. 12 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Safeguards If the client does not receive replies soon enough, it broadcasts the request to all replicas If the request has already been processed, the replicas simply re-send the reply (replicas remember the last reply message they sent to each client) If the primary does not multicast the request to the group, it will eventually be suspected to be faulty by enough replicas to cause a view change

13. 13 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Normal Case Operation Client Primary {REQUEST, o, t, c} o – Operation t – Timestamp c - Client Timestamps are totally ordered such that later requests have higher timestamps than earlier ones

14. 14 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Normal Case Operation Primary p receives a client request m, it starts a three-phase protocol Three phases are: –pre-prepare –prepare –commit

15. 15 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Pre-Prepare Phase v – view number n – sequence number d – digest of the message D(m) m – message Primary Backup, m>, m>

16. 16 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Prepare Phase A backup accepts the PRE-PREPARE message only if: –The signatures are valid and the digest matches m –It is in view v –It has not accepted a PRE-PREPARE for the same v and n –Sequence number is within accepted bounds

17. 17 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Prepare Phase If backup i accepts the pre-prepare message it enters prepare phase by multicasting to all other replicas and adds both messages to its log Otherwise does nothing Replica (including primary) accepts prepare message and adds them to its log, provided that –Signatures are correct –View numbers match the current view –Sequence number is within accepted bounds

18. 18 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Prepare Phase At replica i, prepared (m, v, n, i) = true, iff 2f PREPARE from different backups (not including replica i) that match the pre-prepare When prepared = true, replica i multicasts to other replicas

19. 19 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Agreement Achieved If primary is non-faulty then all 2f+1 non-faulty replicas agree on the sequence number If primary is faulty –Either ≥f+1 non-faulty replicas (majority) agree on some other sequence and the rest realize that the primary is faulty –Or, all non-faulty replicas will suspect the primary is faulty When a faulty primary is replaced, the minority of confused non-faulty replicas are brought up to date up by the majority

20. 20 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Commit Phase Replicas accept commit messages and insert them in their log provided signatures are same Define committed and committed-local predicates as –Committed (m, v, n) = true, iff prepared (m, v, n, i) is true for all i in some set of f+1 non- faulty replicas –Committed-local (m, v, n, i) = true iff the replica has accepted 2f+1 commit message from different replicas that match the pre- prepare for m If Committed-local (m,v,n,i) is true for some non- faulty replica i, then committed (m,v,n) is true

21. 21 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Commit Phase Replica i executes the operation requested by m after committed-local (m, v, n, i) = true and i’s state reflects the sequential execution of all requests with lower sequence numbers The PRE-PREPARE and PREPARE phases of the protocol ensure agreement on the total order of requests within a view The PREPARE and COMMIT phases ensure total ordering across views

22. 22 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Normal Operation Reply All replicas sends the reply, directly to the client v = current view number t = timestamp of the corresponding request i = replica number r = result of executing the requested operation c = client id Client waits for f+1 replies with valid signatures from different replicas, and with same t and r, before accepting the result r

23. 23 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Normal Case Operation: Summery C Primary: 0 2 1 Faulty: 3 Request Pre-prepare Prepare Commit Reply X

24. 24 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Garbage Collection Used to discard messages from the log For the safety condition to hold, messages must be kept in a replica’s log until it knows that the requests have been executed by at least f+1 non-faulty replicas Achieved using a checkpoint, which occur when a request with sequence number (n) is divisible by some constant is executed

25. 25 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Garbage Collection When a replica i produces a checkpoint it multicasts a message to other replicas Each replica collects checkpoint messages in its log until it has 2f+1 of them for sequence number n with same digest d This creates a stable checkpoint and the replica discards all the pre-prepare, prepare and commit messages

26. 26 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao View Changes Triggered by timeouts that prevent backups from waiting indefinitely for request to execute If the timer of backup expires in view v, the backup starts a view change to move to view v+1 by, –Not accepting messages (other than check-point, view- change, and new-view messages) –Multicasting a VIEW-CHANGE message

27. 27 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao View Changes VIEW-CHANGE message is defined as where, C = 2f + 1 checkpoint messages P = set of sets Pm Pm = a PRE-PREPARE msg + all PREPARE messages for all messages with committed = false

28. 28 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao View Change - Primary Primary p of view v+1 receives 2f valid VIEW- CHANGE messages Multicasts a message to all other replicas where –V = set of 2f valid VIEW-CHANGE messages –O = set of reissued PRE-PREPARE messages Moves to view v+1

29. 29 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao View Changes - Backups Accepts NEW-VIEW by checking V and O Sends PREPARE messages for everything in O Moves to view v+1

30. 30 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Events Before the View Change Before the view change we have two groups of non-faulty replicas: the Confused minority and the Agreed majority A non-faulty replica becomes Confused when it is kept by the faulty's from agreeing on a sequence number for a request It can't process this request and so it will time out, causing the replica to vote for a new view

31. 31 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Events Before the View Change The minority Confused replicas send a VIEW- CHANGE message and drop off the network The majority Agreed replicas continue working as long as the faulty's help with agreement The two groups can go out of synch but the majority keeps working until the faulty's cease helping with agreement

32. 32 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao System State: Faulty Primary Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas P System State Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas ≤2f replicas: NOT enough to change views Is Erroneous View Change Possible? P Confused Minority ≤f non-faulty replicas f faulty replicas

33. 33 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Events Before the View Change Given ≥f+1 non-faulty replicas that are trying to agree, the faulty replicas can either help that or hinder that ➲ If they help, then agreement on request ordering is achieved and the clients get ≥f+1 matching replies for all requests with the faulty's help ➲ If they hinder, then the ≥f+1 non-faulty's will time out and demand for a new view When the new majority is in favor of a view change, we can proceed to the new view

34. 34 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao System State: Faulty Primary Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas P System State Is it possible to continue processing requests? Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas YES ≥2f+1 replicas: enough for agreement P f faulty replicas

35. 35 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao System State: Faulty Primary Adversary f non-faulty replicas Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas YES ≥2f+1 replicas: enough for agreement Faulty replicas cease helping with agreement P P Confused Majority 2f+1 non-faulty replicas Enough to agree to change views Majority now large enough to independently move to a new view f faulty replicas

36. 36 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Liveness Replicas must move to a new view if they are unable to execute a request To avoid starting a view change too soon, a replica that multicasts a view-change message for view v+1, waits for 2f+1 view-change messages and then starts the timer T If the timer T expires before receiving new-view message it starts the view change for view v+2 The timer will wait 2T before starting a view- change from v+2 to v+3

37. 37 Spring 2007EEC693: Secure & Dependable ComputingWenbing Zhao Liveness If a replica receives f+1 valid view-change messages from other replicas for views greater than its current view, it sends a view-change message for the smallest view in the set, even if T has not expired Faulty replicas cannot cause a view-change by sending a view-change message since a view- change will happen only if at least f+1 replicas send view-change message The above three techniques guarantee liveness, unless message delays grow faster than the timeout period indefinitely