1. Opportunistic Multipath Forwarding in Publish/Subscribe Systems Reza Sherafat Kazemzadeh AND Hans-Arno Jacobsen Middleware Systems Research Group University of Toronto

2. Agenda The content-based publish/subscribe model – Characteristics and challenges Our approach – Overlay neighborhoods – Adaptive multipath forwarding Experimental evaluation results 2Opportunistic Multipath ForwardingMiddleware 2012

3. Content-Based Pub/Sub Model 3 Pub/Sub S S S S S S S S S S S S S S P P Publish P P P P P P S S Subscribers P P P Publishers Opportunistic Multipath ForwardingMiddleware 2012

4. Content-Based Pub/Sub Model 4 Pub/Sub Publish S S Subscribers P P P Publishers Opportunistic Multipath ForwardingMiddleware 2012 Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching Unicast (one) Multicast (some) Broadcast (all) Variations in traffic patterns makes it difficult to design an optimal overlay network Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching Unicast (one) Multicast (some) Broadcast (all) Variations in traffic patterns makes it difficult to design an optimal overlay network

5. Forwarding paths in the overlay are constructed in fixed end-to-end manner (no/little path diversity) This results in a high number of “pure forwarding” brokers Current Pub/Sub Overlays D C E B A P S ✗ ✗ ✓ ✗ ✓ 5 P P #msgs delivered #msgs sent System Yield = = 1/5 = 20% Opportunistic Multipath ForwardingMiddleware 2012

6. Summary of Our Goals Construct a highly connected overlay mesh that provides high path diversity between publishers and subscribers Avoid pure forwarders by allowing brokers to make fine- grained forwarding decisions based on individual publications and their matching sets Improve system yield, efficiency, scalability and delivery delay Support dynamic adaptive routing based on live traffic patters while avoiding high costs of full overlay reconfiguration 6Opportunistic Multipath ForwardingMiddleware 2012

7. Fixed end-to-end (baseline) Total msgs: 6 Forwarding strategy 1 Total msgs: 5 Forwarding strategy 2 Total msgs: 3 ABC * D * * Forwarding Strategies ABC * D * * p * * * * * * p * * * * * * ABC * D * * p * * * * * * 7Opportunistic Multipath ForwardingMiddleware 2012

8. Routing Tables (Δ-neighborhoods knowledge) Links Management (best links via a gain function) Pub. Forwarding (Path Computations for strategies) Our Approach in a Nutshell A A Δ=1 Δ=2 Δ=3 S S 8Opportunistic Multipath ForwardingMiddleware 2012

9. AC ** * B D AC ** * B D Path Computation for Forwarding Strategies Strategy 1 Middleware 2012Opportunistic Multipath Forwarding9 ABC * D * * p * * * * * * ABC * D * * p * * * * * * Strategy 2

10. Experimental Evaluations We have implemented the algorithms and performed large-scale experimental evaluations with up to 500 brokers Datasets – Synthesized based on Zipf distribution – Social networking traces from Facebook We measured performance of the system in terms of: – Overlay mesh connectivity – Delivery delay – Maximum system throughput – System yield – Publication propagation path length – Memory and CPU overhead Middleware 2012Opportunistic Multipath Forwarding10

11. Number of Available paths Path Diversity in Overlay Mesh Network size:120 Delta=3 Network size:250 Delta=3 Path diversity: 20% of brokers w/ 100 paths Path diversity: 10% of brokers w/ 1000 paths 11Opportunistic Multipath ForwardingMiddleware 2012 Graph is based on a snapshot of the state of links in a running system

12. Publication Hop Count Experiment setup: 120 Brokers Publish rate is 1,800 msgs/sec and number of deliveries: 73,000 (in 5 min) 12Opportunistic Multipath ForwardingMiddleware 2012 Fixed-end-to-end Strategy 2 115% throughput enhancement

13. Conclusions Brokers build a highly connected overlay mesh and make fine-grained forwarding decisions for each publication in order to avoid pure forwarding neighbors We used the notion of overlay neighborhoods to enable local traffic profiling and avoid high costs of overlay reconfiguration Our approach enhances system’s efficiency and yield, and ultimately improves its scalability and performance 13Opportunistic Multipath ForwardingMiddleware 2012

14. Thanks Questions! 14Opportunistic Multipath ForwardingMiddleware 2012

15. Client Broker Pub/Sub Characteristics Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching – Unicast (one) – multicast (some) – broadcast (all) Traffic patterns depends on workload application and may change over time 15Opportunistic Multipath ForwardingMiddleware 2012

16. Related Work: Overlay Reconfiguration Broker overlay is “reconfigured” by addition and removal of links between brokers Advantages – Forwarding path may be improved bringing some publishers and subscribers closer together Disadvantages – Some forwarding paths between publishers and subscribers may indeed suffer – Resulting overlay still relies on fixed end-to-end paths – Reconfiguration is costly and requires full or partial re-propagation of subscriptions [1] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks, FTDCS ‘03 [2] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks. In: FTDCS. (2003)   Re- configure 16Opportunistic Multipath ForwardingMiddleware 2012

17. Links types – Primary links – Soft links – shortcut links – Candidate links – expeditionary links Traffic profiling – In intervals of T sec, brokers count the number of pubs sent over each links – Gain function gain(A,B,T) = pub_traffic_during_T * dist(A,B) Soft link selection – Profiling input – Link state measured – Broker load information exchanged Links ranking A B C D E F G E F … high → low Links Management Overlay Network AB C D B B Primary link Candidate link Soft link 17Opportunistic Multipath ForwardingMiddleware 2012

18. Master v. Working Routing Data Structures Overlay map captured by brokers’ Δ-neighborhoods are relatively static  Master overlay Map (MOM) Brokers link connectivity change dynamically, brokers need an efficient way to compute forwarding paths over the changing set of links  Working Overlay Map (WOM) MOM is a concise representation of the primary overlay that only contains brokers with a direct link Master Routing Tables Working Routing Tables construct 18Opportunistic Multipath ForwardingMiddleware 2012

19. Master v. Working Subscription Tables Similar to WOM, brokers adapt their subscription tables based on the current set of available links:  Working Subscription Table (WST) Between Brokers Sets Beyond Brokers Sets Behind Brokers Sets Master Subscription Table Working Subscription Table Working Overlay Map 19Opportunistic Multipath ForwardingMiddleware 2012

20. Delivered publications StrategyNumber of pure Pure Forwarders System Yield 284,000 (Dense Workload) Baseline 195,00059% Strategy 1 104,00073% Strategy 2 69,00080% Delivered publications Strategy Number of pure Pure Forwarders System Yield 73,000 (Sparse Workload) Fixed end-to-end 91,00044% Strategy 1 42,00063% Strategy 2 29,00071% System Yield (measure of efficiency) 20Opportunistic Multipath ForwardingMiddleware 2012

21. Publication Hop Count Experiment setup: 120 Brokers Sparse and dense workloads Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min 21Opportunistic Multipath ForwardingMiddleware 2012

22. Publication Hop Count Experiment setup: 120 Brokers Sparse publication/subscription workload Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min 22Opportunistic Multipath ForwardingMiddleware 2012

23. Publication Hop Count Sparse Matching WorkloadDense Matching Workload Multi-path forwarding is more effective in sparse workloads 23Opportunistic Multipath ForwardingMiddleware 2012

24. Impact of Broker Fanout on Subscription Covering Experiment setup: 500 brokers Fanout of 5-25 24Opportunistic Multipath ForwardingMiddleware 2012