1. Transactional Mobility in Distributed Content-Based Publish/Subscribe Systems Songlin Hu*, Vinod Muthusamy +, Guoli Li +, Hans-Arno Jacobsen + * Chinese Academy of Sciences, Beijing + University of Toronto June 25, 2009 29th Int’l Conference on Distributed Computing Systems (ICDCS 2009) MIDDLEWARE SYSTEMS RESEARCH GROUP http://padres.msrg.toronto.edu

2. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 2 Many adaptive distributed applications require reprovisioning of software components Distributed content-based publish/subscribe Multiplayer online games Grid computing platforms Stream processing engines Mobile agent frameworks Process execution engines Mobile environments

3. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 3 Movement (reprovisioning) of software components should be well-behaved No lost or duplicated messages Other components not disrupted Movement is transparent to both moving and stationary components Isolated from operations by other components

4. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 4 Contributions Define properties for mobile clients in a distributed publish/subscribe system Develop protocols that achieve efficient movement Evaluate proposed and traditional movement protocols

5. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 5 Review of distributed publish/subscribe routing Publisher Subscriber 1. Advertise 2. Subscribe 3. Publish Advertisement: item = computer brand = ibm price < 2000 Subscription: item = computer brand = ibm price < 1500 Publication: item = computer brand = ibm price = 1400

6. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 6 A movement operation relocates a client (including its state) Movement may fail because target broker rejects it, etc. During movement, there may be multiple copies of a client, but only one should be “running” at any time Client container helps coordinate the movement B1B1 B8B8 B2B2 B7B7 ∙∙∙ Client Container 1 Client A Client Container 7 Client A MOVE (to Broker 7) Client B

7. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 7 Transactional Movement Properties

8. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 8 Modular transactional properties simplify reasoning and implementation Application Pub/Sub Stub Client Ads/Subs/Pubs Notifications Routing Messaging Broker (Considered in this work) (Out of scope)

9. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 9 ACID-like transactional properties for various layers IsolationConsistencyAtomicity A movement should only modify those routing table entries associated with the moving client. Each broker should have the minimal set of routing table entries for a set of advs and subs. Either all or none of the set of routing table updates required for an operation (adv, sub, etc.) should all be applied. Routing layer The set of notifications delivered to stationary clients from a moving publisher should be independent of whether the publisher successfully moves. A moving client (whether successful or not) should receive the same set of notifications as one that did not move. Notifications are delivered exactly once to either the source or target client. Notifications layer Only the initial or final states of a moving client should be observable. There must be at most one running instance of each client. A moving client must be exclusively either at its source or target broker. Client layer Durability is omitted but can be achieved by persisting all state to stable storage Refer to the paper for formal definitions

10. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 10 Movement Protocol

11. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 11 The source and target brokers negotiate the movement of a client The intermediate broker routing tables are reconfigured along with message (2)

12. MIDDLEWARE SYSTEMS RESEARCH GROUP The client and coordinator at the source and target are modelled with state diagrams Coordinators are based on the three-phase commit protocol Global reachable state graph is used to prove some properties  E.g., in the commit state, the source client is clean, and the target client is started

13. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 13 The broker routing tables must be reconfigured when the client moves Routing tables should be correct whether movement succeeds or fails Reconfiguration should be efficient  Network traffic  Broker computation  Isolated from operations

14. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 14 BjBj BiBi BlBl Subscriber Publisher Unadv Traditional end-to-end movement can be expensive

15. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 15 BjBj BiBi BlBl Subscriber Publisher Adv Traditional end-to-end movement can be expensive

16. MIDDLEWARE SYSTEMS RESEARCH GROUP BjBj BiBi BlBl The presence of a covering advertisement can avoid flooding

17. MIDDLEWARE SYSTEMS RESEARCH GROUP B2B2 B1B1 BjBj BiBi BlBl A A A Movement can still trigger burst of covered messages C B CB A A

18. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 18 1 11 1 11 BjBj BiBi (2) Approve message 2 2 2 Subscriber Publisher Proposed protocol reconfigures routing tables hop-by-hop 1 Routing entry Routing shadow copy 2

19. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 19 Proposed protocol reconfigures routing tables hop-by-hop 1 11 1 11 BjBj BiBi (3) Ack + State message 2 2 21 1 1 1 Routing entry Routing shadow copy 2 Subscriber Publisher Only brokers between B i and B j are updated Predictable number of messages Advertisement entries are simply reversed Subscription entries are more complicated

20. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 20 Evaluations

21. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 21 Experimental setup Two movement algorithms: proposed reconfiguration, traditional covering-based Deployments in dedicated LAN and shared WAN (PlanetLab) environments 400 clients repeatedly move between brokers 1,13 and 2,14  Pause for 10 seconds at each broker Five subscription workloads: covered, chained, tree, distinct, random Metrics: network message traffic, movement duration, and throughput Default topologySubscriptions

22. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 22 The reconfiguration protocol is much faster than the covering protocol Movement of “root” subscriptions is more expensive in the covering protocol

23. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 23 The reconfiguration protocol scales with the number of moving clients The reconfiguration protocol achieves better movement latency despite more total messages because it is less bursty

24. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 24 The reconfiguration protocol is more stable and efficient with respect to the workload Covered workload experiences high latency despite relatively few messages due to bimodal behaviour of covering protocol

25. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 25 The covering protocol is very sensitive to the movement of “root” subscriptions Only one client is moving, confirming the effect of the pathological case for the covering protocol

26. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 26 The presence of stationary clients has little effect on moving clients Each additional set of moving clients: 10 roots from covered, tree, chained workloads, 10 leaves from previous workloads, 10 from distinct workload

27. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 27 Keeping the length between source and target brokers constant, the topology size has little effect The covered workload is used here to exaggerate the effects

28. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 28 Experiments on PlanetLab WAN support the earlier conclusions Longer latencies are due to more limited network and compute resources Same relative performance and trends are seen

29. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 29 Conclusions Distributed publish/subscribe systems lack well-defined guarantees for the movement of clients ACID-like transactional properties were defined for this problem  Properties are modularized to simplify reasoning and implementation Client layer movement and routing layer hop-by-hop reconfiguration protocols were developed Evaluations show proposed protocol is more efficient and stable with respect to various parameters  End-to-end movement using covering negatively affects performance Future work includes more efficient failure resilience in the publish/subscribe routing layer Use cases for supporting transactions that span multiple operations would be interesting

30. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 30 Transactional Mobility in Distributed Content-Based Publish/Subscribe Systems http:// padres.msrg.toronto.edu Q&A

31. MIDDLEWARE SYSTEMS RESEARCH GROUP ICDCS ’09 Transactional Mobility in Pub/Sub 31 Extra slides

32. MIDDLEWARE SYSTEMS RESEARCH GROUP Client and coordinator states at the source and target Coordinators are based on the three-phase commit protocol The failure and timeout transitions are omitted for brevity

33. MIDDLEWARE SYSTEMS RESEARCH GROUP The global reachable state graph can be used to prove some of the transactional properties E.g., in the commit state, the source client is clean, and the target client is started Refer to the paper for proofs

34. MIDDLEWARE SYSTEMS RESEARCH GROUP Proposed protocol reconfigures routing tables hop-by-hop BjBj BiBi BlBl Only brokers between B i and B j are updated Advertisement entries are simply reversed Subscription entries are more complicated