1. Group Communication using Ensemble Part II

2. 2 Introduction From previous tutorial: Ensemble’s application interface: Concepts of Group Membership, View, Sending and receiving Today: Ensemble’s architecture Discussion of Ensemble’s layers

3. 3 Ensemble Infrastructure Layered protocol architecture All features are implemented as microprotocols/layers Layers are implemented as a set of callbacks that handle events passed to them. Each instance of a layer maintain a private local state. A new stack is created for a new configuration at each endpoint (node) Ability to change the group protocol on the fly

4. 4 Stacks Combinations of layers that work together to provide high-level protocols Stack creation: A new protocol stack is created at each endpoint of a group whenever the configuration (e.g. the view) of the group changes. All endpoint in the same partition receive the same ViewState record to create their stack: select appropriate layers according to the ViewState create a new local state for each layer compose the protocol layers

5. 5 Communication Ensemble embodies two forms of communication. The first is communication between protocol stacks in a group, using messages sent via some communication transport. The second is intra-stack communication between protocol layers sharing a protocol stack, using Ensemble events

6. 6 Intra-stack communication Layers communicate with their neighbors in the same stack by sending events. Each layer has a callback reference to the two layers immediately above and below itself. The Ensemble infrastructure guarantees FIFO ordering on events passed between two layers No concurrency guarantee: The system infrastructure that hands events to layers through the callbacks never invokes a layer’s callbacks concurrently. It guarantees that at most one invocation of any callback is executing at a time and that the current callback returns before another callback is made to the protocol layer.

7. 7 Intra-stack communication cont. The execution of a protocol stack can be visualized as a set of protocol layers executing with a pair of event queues between each pair: one queue for events going up and another for events going down. The protocol layers are then automata that repeatedly are scheduled to take pending events from one of the adjacent incoming queues, execute it, and then deposit zero or more events into two adjacent outgoing queues

8. 8 Intra-stack communication cont.

9. 9 Inter-stack communication A layer in the stack may send a message to its corresponding peer on a different stack in the following ways: Generating a new message Piggybacking information to a message received from the layer above Layers never read or modify other layers’ message headers

10. 10 Inter-stack communication cont.

11. 11 Block Chain

12. 12 Exploring the Ensemble layers - Reliable multicast Mnak layer: Implement a reliable fifo-ordered multicast protocol Messages from live members are delivered reliably Messages from faulty members are retransmitted by live members Protocol: Keep a record of all multicast messages to retransmit on demand Use Stable event from Stable layer: StblVct vector is used for garbage collection NumCast vector gives an indication to lost messages => recover them

13. 13 Stability Stable layer: Track the stability of multicast messages Protocol: Maintain Acks[N][N] by unreliable multicast: Acks[s][t]: #( s ’ messages) that t has acknowledged Stability vector StblVct = {(minimum of row s): (for each s)} NumCast vector NumCast = {(maximum of row s): (for each s)} Occasionally, recompute StblVct and NumCast, then send them down in a Stable event.

14. 14 Ordering Sequencer layer Provide total ordering Protocol: Members buffer all messages received from below in a local buffer The leader periodically multicasts an ordering message Members deliver the buffered messages according to the leader’s instructions

15. 15 Failure Detector Suspect layer: Regularly ping other members to check for suspected failures Protocol: If (#unacknowledged Ping messages for a member > threshold) send a Suspect event down Slander layer: Share suspicions between members of a partition The leader is informed so that faulty members are removed, even if the leader does not detect the failures. Protocol: The protocol multicasts slander messages to other members whenever receiving a new Suspect event