Foundations for Highly-Available Content-based Publish/Subscribe Overlays Young Yoon, Vinod Muthusamy and Hans-Arno Jacobsen

PADRES: an Overlay of Content-based Pub/Sub Brokers 1. Advertise Publisher 3. Publish Subscriber This is a project on PADRES, an enterprise service bus as an overlay of content-based pub/sub brokers. See how a notifcation path gets constructed with advertisement and subscription routing. 2. Subscribe 2/22/2019 http://msrg.org

Inside a PADRES Broker Matching Engine + Routing Table input queue subscription dest Matching Engine Routing Table + input queue output queue dest1 output queue dest2 service time < 3s dest2 Inside the broker, there are matching engine, routing table, input/output queues. service time < 2s dest3 2/22/2019 http://msrg.org

Content-based Publish/Subscribe as an Application Integration Fabric Stock-market monitoring Network management and monitoring Algorithmic trading Distributed business process execution Business activity monitoring Enterprise service bus Content-based Publish/Subscribe as an Application Integration Fabric, as pointed out in the paper. A number of examples can be found. These application can be in a large scale, thus a publish/subscribe overlay is needed, like PADRES. 2/22/2019 http://msrg.org

Obstacles for the Pub/Sub Overlay Publisher Congestion due to I/O overload or matching overhead Crash failure Periodic broker maintenance Subscriber Congestion can occur for various reasons, such as I/O overhead and matching overhead at a broker. This can significantly affect message delivery. Messages can get delayed or even get lost 2/22/2019 http://msrg.org

Approaches for High-Availability Overlay transformation by link re-wiring (planned) P2 Planned or on-demand approaches to transforming an overlay to achieve high-availability. The terms can be debatable, but the key difference whether you deploy a new broker from a pool of standby brokers or not. Planned transformation is about changing a topology, on-demand does not change increases a broker capacity Explain implication on the message delivery re-wiring is not enough Overlay transformation by adaptive broker replication (on-demand) 2/22/2019 http://msrg.org

Outline High-availability for pub/sub overlays Primitive operations for overlay transformations Fundamental message resumption protocols 2/22/2019 http://msrg.org

What Does High-Availability Mean in Pub/Sub Overlays? Avoiding the violation of functional and non-functional properties on pub/sub messaging 2/22/2019 http://msrg.org

Functional Properties of Pub/Sub Messaging Complete delivery of publications to all interested subscribers In-order delivery of publications to all interested subscribers More formally defined in the paper. A stronger functional properties require to be satisfied even with a broker failure. 2/22/2019 http://msrg.org

Non-Functional Properties of Pub/Sub Messaging Publication delivery resumption delay How long does it take to resume the message delivery? Last message processed time How long does it take to process all the pending messages due to failures in the overlay? 2/22/2019 http://msrg.org

High-availability through planned overlay transformation: Primitive operations and protocols First, let’s go through the primitive operations and protocols for a planned overlay transformation 2/22/2019 http://msrg.org

Primitive Operations for Planned Overlay Transformation SHIFT(Bi, Bj , Bk ) Bi Bi Bj Bj Bk Bk 2/22/2019 http://msrg.org

Theorem There is always a plan to transform any valid* overlay G to another valid overlay G’ using only the SHIFT primitives. * acyclic and connected graph 2/22/2019 http://msrg.org

Disruption during Planned Transformation (1) Routing table entry: subscription id | last hop B3 B1 S1 | B1 S1 | B2 S1 | B1 S1 | B2 B2 B3 B2 B3 B1 B1 SHIFT(B1, B2, B3) S1 S1 Routing state changes at the brokers 2/22/2019 http://msrg.org

Disruption during Planned Transformation (2) Publication drop or looping Not updated yet Updated S1 | B3 S1 | B2 P P1 B2 B3 B1 S1 2/22/2019 http://msrg.org

Disruption during Planned Transformation (3) Messages can be delivered out of order S1 | B1 S1 | B2 S1 | B3 S1 | B1 S1 | B3 S1 | B1 B2 B3 B2 B3 B2 B3 P2 P1 P1 P2 B1 B1 B1 S1 S1 S1 P1 sent @ 09:00 (propagates slowly) P2 sent @ 09:01 (propagates quickly to B1) P2 arrives at B1 @ 09:03 P1 arrives at B1 @ 09:04 2/22/2019 http://msrg.org

Synchronous Shift Operation Routing table update Make a new connection and Buffer user messages S1 | B1 S1 | B2 S1 | B3 S1 | B2 B2 B3 B2 B3 B2 B3 Shift request Shift request B1 B1 B1 Routing table update S1 | B3 S1 | B1 S1 | B3 S1 | B1 S1 | B3 S1 | B1 B2 B3 B2 B3 B2 B3 ACK FIN ACK B1 B1 B1 Routing table update 2/22/2019 http://msrg.org

Planned Transformation Example B1 B2 B3 B4 B5 B6 B7 B8 B1 B2 B3 B4 B5 B6 B7 B8 Suppose the following extreme case of a chained broker topology. Assume that replacing the link betwee B1 and B2 with the link between B1 and B8 turned is a better plan. 2/22/2019 http://msrg.org

Our Implementation of Planned Transformation: Global v. Incremental Involve all brokers at once B1 B2 B3 B4 B5 B6 B7 B8 Involve 3 brokers at a time Suppose the following extreme case of a chained broker topology. Assume that replacing the link betwee B1 and B2 with the link between B1 and B8 turned is a better plan. B1 B2 B3 B4 B5 B6 B7 B8 . . . 2/22/2019 http://msrg.org

Disruption due to Global and Incremental Transformation (1) Incremental approach shows less disruption 2/22/2019 http://msrg.org

Disruption due to Global and Incremental Transformation (2) Incremental approach exhibits constant publication delivery resumption delay 2/22/2019 http://msrg.org

High-availability through on-demand overlay transformation: Primitive operations and protocols 2/22/2019 http://msrg.org

Primitive Operations for On-demand Overlay Transformation REPLICATE(Bi, Bj) CONSOLIDATE() N N Neighbor Broker F Faulty Broker . . . F R R R R Replica N 2/22/2019 http://msrg.org

Purpose of the On-Demand Transformation Replace a permanently failed broker Exclusively re-route through the replica N N Faulty Broker Replica Faulty Broker Replica Offload temporarily failed broker Load-balance between the replicas N N S S 2/22/2019 http://msrg.org

P P S S Replication N N N N Adv_Request Sub_forward Adv_Request Faulty Broker Replica Faulty Broker Replica Adv_Request Sub_forward N N Established overlay S S Physical connection 2/22/2019 http://msrg.org

Disruptions During On-demand Transformation (1) Message can be dropped P1 N1 Not delivered to S2 P1 @ 09:04 F R S1 @ 09:00 N3 N2 S2 @ 11:30 S1 S2 2/22/2019 http://msrg.org

Disruptions During On-demand Transformation (2) Messages can be delivered out of order P P2 @ 09:03 P1 @ 09:01 N1 F R N2 P1 @ 09:10 P2 @ 09:05 S 2/22/2019 http://msrg.org

Two Basic Message Resumption Protocols Synchronous resumption Wait until replication is completed on the new broker Incremental resumption Resume publication as soon as subscription path is established through the replica 2/22/2019 http://msrg.org

Incremental Message Resumption Protocol Stop retrieving incoming user messages R Notify when replication is done N Copy over outbound messages Stop forwarding messages F 2/22/2019 http://msrg.org

Advanced Queuing Method for Incremental Resumption Priority Queue High-priority control messages Wait Ready S1 P1 P1 Round –robin de-queue User-generated messages S2 P1 P1 S3 P1 P1 .. .. 2/22/2019 http://msrg.org

Incremental Resumption at the Replica (1): Incomplete Replication of Routing States Sub Lasthop S1 B1 S’ = {S1} Matching subs. of P1: S = {S1,S2,S3} R B1 B0 B2 F B3 2/22/2019 http://msrg.org

Incremental Resumption at the Replica (2): Caching of Publications Sub Lasthop S1 B1 S2 B2 Pub sentTo matches P1 {B1} {S2, S3} S2 B1 R B0 B2 F B3 2/22/2019 http://msrg.org

Synchronous vs. Incremental Resumption Policies Incremental approach resumes message delivery quicker Synchronous approach takes less replication time 2/22/2019 http://msrg.org

Ordering and Performance Faulty Broker Replica N Not much delay caused by the re-ordering Ordered by source using sequence ID S 2/22/2019 http://msrg.org

Conclusions Defined primitives for content-based pub/sub overlay transformation Classified disruptions in terms of violation of functional and non-functional messaging properties Quantified the trade-offs of the overlay transformation protocols Demonstrated small overhead of our protocols 2/22/2019 http://msrg.org

Thank You! http://padres.msrg.utoronto.ca