Some Open Problems in Publish/Subscribe Networking David S. Rosenblum Chief Technology Officer PreCache Inc.

Acknowledgments Alexander L. Wolf  University of Colorado at Boulder Antonio Carzaniga  University of Colorado at Boulder PreCache Engineering Team

Background

Information-Centric Internet Applications Software and Antivirus Updates Consumer Alerts Location-Based Services for Mobile Wireless Multiplayer Online Games Web Search Engines e-Business (e.g., Supply Chain Mgmt) Distributed Sensor Networks Publish/subscribe is a natural fit!

Publish/Subscribe Networking Publish/subscribe is traditionally implemented by centralized servers But server-based realizations do not scale to Internet-wide applications So existing networks require “faking it”  Request/response interaction  Continual subscriber polling  Enormous server farms  Dumb caching And so we must realize publish/subscribe via a distributed network of routers

S IENA Content-Based Routing Subscription Forwarding a s1s1 s1:a s1:1 s1:2 s1:3 s1:2 s1:6 s1:3 s1:1 s1:5

S IENA Content-Based Routing Subscription Merging s1:1 s1:2 s1:6 s1:3 s1:1 s2s2 b s1:3 s1:2 s1:5 s1:1 s2:5 s1:1 s2:5 s1:2 s2:8 s1:2 s2:8 s1:5 s2:b s1:5 s2:b a s 1 covers s 2 1 s1:a s2:2 s1:a s2:2

S IENA Content-Based Routing Notification Delivery b s1:1 s2:5 s1:1 s2:5 s1:2 s1:6 s1:3 s1:1 s1:3 s1:2 s2:8 s1:2 s2:8 a s1:a s2:2 s1:a s2:2 s1:5 s2:b s1:5 s2:b n 1 matches s 1 n 1 matches s 2 n1n1

PreCache N ET I NJECTOR Architecture = Routing Engine= Channel Manager Internet Publisher Subscriber = Event Agent

PreCache N ET I NJECTOR Routing and forwarding based on S IENA  Generalize idea of subscription merging Compute single subscription covering all received subscriptions  Employ approximate matching Constant time and space complexity Log time and space with additional leakage reduction Channel services  Namespace management  Resource allocation  Load balancing, fault tolerance, authentication

Open Problems

Comments on the Problems Problems identified based on experience with N ET I NJECTOR and S IENA Many of the problems arise because of a desire for scalability Some problems are deeply technical Other problems are simply pragmatic

Problem Wireless Mobile Devices (WMDs?) a s1:a s1:1 s1:2 s1:3 s1:2 s1:6 s1:3 s1:1 s1:5 a

Problem Issues with Wireless Mobile Devices Caching notifications in the network Stream reconstruction and duplicate suppression Frequency of movement versus overhead of reconfiguration Gateways for , SMS, etc.

Problem Security Traditional security properties are address-based  Example: Authentication Bob wants to make sure Alice sent the message  Content-based analog Bob wants to make sure a message represents reality Pub/sub admits new kinds of vulnerabilities  Example: Denial of Service Highly generic subscription (“Price > 0”) causes flood of notifications to subscriber  How do you distinguish a malicious subscriber from a greedy subscriber? How do you do content-based routing when the content is encrypted???

Problem Client Connections and Firewalls Want constant connection between subscriber and edge router  Otherwise subscriber polls for notifications  Connections limits may require multiplexing Client must initiate connection to edge router in order to breach firewall And if port 80 is the only open port …  Need HTTP encapsulation of messages  May need HTML formatting of messages  Routers need to multiplex and/or demultiplex message traffic

Problem Approximate Matching Rationale: High-performance routing  Expect approximate matching to have better time/space complexity than exact matching Approximation must be conservative  False positives OK, false negatives not  Must still perform exact match at some point before delivery to subscriber Leakage may increase traffic  Tradeoff in computational resources We need simulation tools to explore this!

Problem Optimizing for Traffic Variations Can routers dynamically optimize for traffic variations? Example: The Brittany Spears Effect  All subscribers want certain notifications N 1  Few subscribers want other notifications N 2  N 1 notifications may flood network Example: The Google Effect  Certain subscribers S 1 want all notifications  Other subscribers S 2 want few notifications  S 1 subscriptions may dominate routing We need simulation tools to explore this!

Problem Service-Provider Deployment Difficult to convince network service providers to enhance their networks with publish/subscribe  Application demand not yet critical  Lack of standards Economic barriers govern router design  Example: 100M users, $10K/router 1000 users/router: 100K routers, $1G outlay 100 routers: $1M outlay, 1M users/router

Problem Peer-to-Peer Deployment Reasonable alternative to service-provider deployment  “Grass roots” generation of demand Challenges  Dynamically aligning peer topology to underlying network topology  Dynamically partitioning routing responsibilities across peers  Ensuring reliability, privacy and/or integrity of messages

Problem Unicast Fanout at Edge Routers Example: 100M users on 1K routers  100K users per router  10Kbyte notification >80 milliseconds over OC-192 >80 seconds over 10Mb Ethernet >4 hours over 56K modem Idea: Use publish/subscribe for “leveling”  Partition users into classes Example: Last digit of serial number  Publish once per class  Tune publication rate to available bandwidth and SLA

Conclusion

Many Internet applications naturally require publish/subscribe messaging Scalability can be achieved through publish/subscribe networking S IENA, PreCache, and others have established many fundamental results But many open problems remain to be solved