Composite Subscriptions in Content-based Pub/Sub Systems Guoli Li

Middleware 2005 Grenoble France Agenda Motivation PADRES System Composite Subscriptions Case Study: Workflow Management Evaluations Conclusions and Future Work Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Motivation A large number of atomic events Lack of event correlation Potential Applications: Workflow processing and business process execution Network management … Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France The PADRES System PADRES: Publish/Subscribe Applied to Distributed REsource Scheduling Features: Predicate-based subscription language (attribute, op, value) pair Extended to supporting composite subscriptions Rule-based publication/subscription matching Publications are mapped to facts Subscriptions are mapped to rules Advertisement-based routing Content-based, it uses predicate-based subscription language Middleware 2005 Grenoble France 2019/1/15

Content-based Routing Advertising Advertisement Distributed Overlay Broker Network Broker network Clients are connected as publisher/subscribers … … PADRES Middleware 2005 Grenoble France 2019/1/15

Content-based Routing Subscribing Subscription Distributed Overlay Broker Network … … PADRES Middleware 2005 Grenoble France 2019/1/15

Content-based Routing Publishing Publication Distributed Overlay Broker Network … … PADRES Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Several Concepts Message Advertisement Subscription Publication Event Atomic event: publication in pub/sub literature Composite event: a pattern of events Subscriptions Atomic subscription: atomic event Composite subscription: composite event Middleware 2005 Grenoble France 2019/1/15

Composite Subscriptions Composite subscription consists of primitive subscriptions Provide a higher level view for subscribers Save network bandwidth Subscription language features Operators: AND (&), OR (||) Variables: ($x) Advanced event patterns & || S1 S2 S3 S4 S5 {} is used to specify the priority of operators CS={{S1 OR S2} AND{S3 OR S4} AND S5} Middleware 2005 Grenoble France 2019/1/15

Composite Subscriptions Parallelization pattern: s1 & s2 Alternation pattern: s1 || s2 Sequence pattern: (s1;[timespan:ts]s2)[within:wi] Repetition pattern: Repetition(S, n, attr, v) e.g. {rule ((job (appl = $X) (name = A) (state = succ)) & (job (appl = $X) (name = B) (state = succ))) => (forward a notification to proper destination)} Middleware 2005 Grenoble France 2019/1/15

Composite Subscriptions Parallelization pattern: s1 & s2 Alternation pattern: s1 || s2 Sequence pattern: (s1;[timespan:ts]s2)[within:wi] Repetition pattern: Repetition(S, n, attr, v) e.g. {rule ((job …. (name = A) (time = $Y)) & (job …. (name = B) (time > $Y+ts) (time < $Y+wi))) => (forward a notification to proper destination)} e.g. {rule ((job …. (name = A) (attr = $Z)) & (job …. (name = A) (attr = $Z+v)) & … & (job …. (name = A) (attr = $Z+(n-1)v)) ) => (forward a notification to proper destination)} Middleware 2005 Grenoble France 2019/1/15

Composite Subscription Routing CS={{S1 AND S2} ANDS3} P Publishers & || S1 S2 S3 S Subscribers CS’={S1 AND S2} P1 Distributed Overlay Broker Network B1 B5 CS S S1 B3 CS’ B4 S2 S3 B2 B6 P2 P3 Middleware 2005 Grenoble France 2019/1/15

Composite Event Detection CS={{S1 AND S2} ANDS3} P Publishers AND S1 S2 S3 S Subscribers CS’={S1 AND S2} P1 P1 Distributed Overlay Broker Network S1 CS B1 B5 S CS’ CS B3 P12 B4 P123 B2 P2 S2 B6 P2 S3 P3 P3 Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Advantages Rule-based approach naturally enables the composite subscription Event detection results can be shared Network traffic is reduced Composite subscription is forwarded into the network as far as possible before it is split Composite events are detected close to the data source and not widely disseminated Middleware 2005 Grenoble France 2019/1/15

Case Study: Workflow Management Decentralized Workflow Management based on Pub/Sub Functions Workflow transform Workflow deployment Workflow execution Workflow control and monitoring Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Architecture Middleware 2005 Grenoble France 2019/1/15

Workflow Descriptions Workflows are described using XML-based languages, such as BPEL Activities defined in a workflow have dependencies among each other A workflow instance is generated by a trigger Failures can be handled in the workflow description trigger A E F B C D Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Workflow Transform The first job(s) subscribe to trigger messages e.g. Job A subscribes to trigger publication: {[class,eq,trigger],[workflow,eq,payroll], [instanceID,=,$x]} Job dependencies are implied by subscriptions e.g. Job E subscribes to job A failed: {[class,eq,job_info],[workflow,eq,payroll], [nstanceID,=,$x], [job,eq,A], [status,eq,fail]} e.g. Job D subscribes to B and C finished, which is a composite subscription Advertisements are generated from a workflow specification file Middleware 2005 Grenoble France 2019/1/15

Distributed Overlay Broker Network Workflow Deployment A B C D BPEL sub/advs Job D sub/advs Job C sub/advs Job B sub/advs Job A Deployer sub/advs Job D Distributed Overlay Broker Network Job D Agent B1 B5 B3 B4 sub/advs Job A B2 Job A Agent B6 sub/advs Job C sub/advs Job B Job C Agent Job B Agent Middleware 2005 Grenoble France 2019/1/15

Workflow Execution (Success) A B C D BPEL trigger Workflow Manager END Distributed Overlay Broker Network Job D Agent B1 B5 BC B3 A B4 A sub/advs Job D B2 Job A Agent B6 A sub/advs Job A Job C Agent Job B Agent sub/advs Job C B C sub/advs Job B Middleware 2005 Grenoble France 2019/1/15

Workflow Execution (Failure) B C D BPEL E F trigger Workflow Manager sub/advs Job D Distributed Overlay Broker Network Job D Agent B1 B5 Workflow deployment and execution are performed in the Pub/Sub layer Workflow execution is event-driven B3 B4 END B2 Job A Agent B6 Job F Agent A sub/advs Job A Job C Agent sub/advs Job F Job B Agent Job E Agent E sub/advs Job E sub/advs Job C sub/advs Job B Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Workflow Monitoring Monitor is a Pub/Sub client Monitor/control workflow execution status Workflow level Activity level Support dynamic workflow modification Visualize topology of broker overlay network Visualize message routing Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Evaluations Environment Intel Xeon 3GHz, 1GB RAM JDK 1.4.2 Workload description Predicates based on 20 attributes with random operators Values are uniformly distributed in the value range Two workflows with different number of jobs Metrics Matching time Routing delay per composite subscription Network traffic zipf Middleware 2005 Grenoble France 2019/1/15

Publication Matching Time Evaluate the performance of the rule-based matching engine in PADRES. X-axis is the number of subscriptions. Y-axis is the matching time of publications in logarithmic scale. Each data point is obtained by averaging the time taken to route 5000 publications. We compared our PADRES matching engine with Naive Matching algorithm and Counting algorithm Middleware 2005 Grenoble France 2019/1/15

Composite Event Detection We measure the composite event detection time. X-axis is the number of atomic subscriptions per CS. Y-axis is the detection time. We show the detection time of injecting 100, 5000, and 10000 publications. The detection time includes: mapping the cs to a rule, and matching the rule against publication facts. Each data point is an average of 200 composite subscriptions. Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Routing Delay We measure the routing delay of composite subscriptions. X-axis is the number of atomic subscriptions per CS. Y-axis is the routing delay per broker. At each broker we need to split the CS according to the destinations. The complexity is O(N), where N is the number of atomic subscriptions per CS. Middleware 2005 Grenoble France 2019/1/15

Number of Notifications Composite subscriptions can reduce the number of notifications delivered to clients. X-axis is the number of issued publications. Y-axis is the number of notifications that the client received. We issue 200 composite subscriptions (each has 5 atomic subscriptions) in the first case (blue line). And issue 1000 atomic subscriptions which consist the 200 CSs in the second case (red line) Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Workflow Traffic The network traffic in the execution phase Workflow A has 4 jobs Workflow B has 8 jobs The network traffic is generated by 10 generations of the workflow. Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Related Work P.R. Pietzuch, B. Shand, and J. Bacon. Composite event detection as a generic middleware extension. IEEE Network Magazine, Special Issue on Middleware Technologies for Future Communication Networks, Jan 2004 SIENA REBECA Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Conclusions An expressive subscription language A rule-based approach of message matching and composite event detection Case study: a decentralized workflow management system based on PADRES Future Works Support BPEL4WS QoS based workflow execution Middleware 2005 Grenoble France 2019/1/15

Middleware 2005 Grenoble France Questions? Middleware 2005 Grenoble France 2019/1/15