1. Composite Subscriptions in Content-based Pub/Sub Systems
Guoli Li

2. Middleware 2005 Grenoble France
Agenda
Motivation
PADRES System
Composite Subscriptions
Case Study: Workflow Management
Evaluations
Conclusions and Future Work
Middleware 2005 Grenoble France
2019/1/15

3. Middleware 2005 Grenoble France
Motivation
A large number of atomic events
Lack of event correlation
Potential Applications:
Workflow processing and business process execution
Network management …
Middleware 2005 Grenoble France
2019/1/15

4. Middleware 2005 Grenoble France
The PADRES System
PADRES: Publish/Subscribe Applied to Distributed REsource Scheduling
Features:
Predicate-based subscription language
(attribute, op, value) pair
Extended to supporting composite subscriptions
Rule-based publication/subscription matching
Publications are mapped to facts
Subscriptions are mapped to rules
Advertisement-based routing
Content-based, it uses predicate-based subscription language
Middleware 2005 Grenoble France
2019/1/15

5. Content-based Routing
Advertising
Advertisement
Distributed Overlay
Broker Network
Broker network
Clients are connected as publisher/subscribers … …
PADRES
Middleware 2005 Grenoble France
2019/1/15

6. Content-based Routing
Subscribing
Subscription
Distributed Overlay
Broker Network … …
PADRES
Middleware 2005 Grenoble France
2019/1/15

7. Content-based Routing
Publishing
Publication
Distributed Overlay
Broker Network … …
PADRES
Middleware 2005 Grenoble France
2019/1/15

8. Middleware 2005 Grenoble France
Several Concepts
Message
Advertisement
Subscription
Publication
Event
Atomic event: publication in pub/sub literature
Composite event: a pattern of events
Subscriptions
Atomic subscription: atomic event
Composite subscription: composite event
Middleware 2005 Grenoble France
2019/1/15

9. Composite Subscriptions
Composite subscription consists of primitive subscriptions
Provide a higher level view for subscribers
Save network bandwidth
Subscription language features
Operators: AND (&), OR (||)
Variables: ($x)
Advanced event patterns
& || S1 S2 S3 S4 S5
{} is used to specify the priority of operators
CS={{S1 OR S2} AND{S3 OR S4} AND S5}
Middleware 2005 Grenoble France
2019/1/15

10. Composite Subscriptions
Parallelization pattern: s1 & s2
Alternation pattern: s1 || s2
Sequence pattern: (s1;[timespan:ts]s2)[within:wi]
Repetition pattern: Repetition(S, n, attr, v)
e.g. {rule ((job (appl = $X) (name = A) (state = succ)) &
(job (appl = $X) (name = B) (state = succ)))
=> (forward a notification to proper destination)}
Middleware 2005 Grenoble France
2019/1/15

11. Composite Subscriptions
Parallelization pattern: s1 & s2
Alternation pattern: s1 || s2
Sequence pattern: (s1;[timespan:ts]s2)[within:wi]
Repetition pattern: Repetition(S, n, attr, v)
e.g. {rule ((job …. (name = A) (time = $Y)) &
(job …. (name = B) (time > $Y+ts) (time < $Y+wi)))
=> (forward a notification to proper destination)}
e.g. {rule ((job …. (name = A) (attr = $Z)) &
(job …. (name = A) (attr = $Z+v)) &
… &
(job …. (name = A) (attr = $Z+(n-1)v)) )
=> (forward a notification to proper destination)}
Middleware 2005 Grenoble France
2019/1/15

12. Composite Subscription Routing
CS={{S1 AND S2} ANDS3} P
Publishers
& || S1 S2 S3 S
Subscribers
CS’={S1 AND S2} P1
Distributed Overlay Broker Network B1 B5 CS S S1 B3
CS’ B4 S2 S3 B2 B6 P2 P3
Middleware 2005 Grenoble France
2019/1/15

13. Composite Event Detection
CS={{S1 AND S2} ANDS3} P
Publishers
AND S1 S2 S3 S
Subscribers
CS’={S1 AND S2} P1 P1
Distributed Overlay Broker Network S1 CS B1 B5 S
CS’ CS B3
P12 B4
P123 B2 P2 S2 B6 P2 S3 P3 P3
Middleware 2005 Grenoble France
2019/1/15

14. Middleware 2005 Grenoble France
Advantages
Rule-based approach naturally enables the composite subscription
Event detection results can be shared
Network traffic is reduced
Composite subscription is forwarded into the network as far as possible before it is split
Composite events are detected close to the data source and not widely disseminated
Middleware 2005 Grenoble France
2019/1/15

15. Case Study: Workflow Management
Decentralized Workflow Management based on Pub/Sub
Functions
Workflow transform
Workflow deployment
Workflow execution
Workflow control and monitoring
Middleware 2005 Grenoble France
2019/1/15

16. Middleware 2005 Grenoble France
Architecture
Middleware 2005 Grenoble France
2019/1/15

17. Workflow Descriptions
Workflows are described using XML-based languages, such as BPEL
Activities defined in a workflow have dependencies among each other
A workflow instance is generated by a trigger
Failures can be handled in the workflow description
trigger A E F B C D
Middleware 2005 Grenoble France
2019/1/15

18. Middleware 2005 Grenoble France
Workflow Transform
The first job(s) subscribe to trigger messages
e.g. Job A subscribes to trigger publication: {[class,eq,trigger],[workflow,eq,payroll], [instanceID,=,$x]}
Job dependencies are implied by subscriptions
e.g. Job E subscribes to job A failed:
{[class,eq,job_info],[workflow,eq,payroll], [nstanceID,=,$x], [job,eq,A], [status,eq,fail]}
e.g. Job D subscribes to B and C finished, which is a composite subscription
Advertisements are generated from a workflow specification file
Middleware 2005 Grenoble France
2019/1/15

19. Distributed Overlay Broker Network
Workflow Deployment A B C D
BPEL
sub/advs
Job D
sub/advs
Job C
sub/advs
Job B
sub/advs
Job A
Deployer
sub/advs
Job D
Distributed Overlay Broker Network
Job D
Agent B1 B5 B3 B4
sub/advs
Job A B2
Job A
Agent B6
sub/advs
Job C
sub/advs
Job B
Job C
Agent
Job B
Agent
Middleware 2005 Grenoble France
2019/1/15

20. Workflow Execution (Success)A B C D
BPEL
trigger
Workflow
Manager
END
Distributed Overlay Broker Network
Job D
Agent B1 B5 BC B3 A B4 A
sub/advs
Job D B2
Job A
Agent B6 A
sub/advs
Job A
Job C
Agent
Job B
Agent
sub/advs
Job C B C
sub/advs
Job B
Middleware 2005 Grenoble France
2019/1/15

21. Workflow Execution (Failure)B C D
BPEL E F
trigger
Workflow
Manager
sub/advs
Job D
Distributed Overlay Broker Network
Job D
Agent B1 B5
Workflow deployment and execution are performed in the Pub/Sub layer
Workflow execution is event-driven B3 B4
END B2
Job A
Agent B6
Job F
Agent A
sub/advs
Job A
Job C
Agent
sub/advs
Job F
Job B
Agent
Job E
Agent E
sub/advs
Job E
sub/advs
Job C
sub/advs
Job B
Middleware 2005 Grenoble France
2019/1/15

22. Middleware 2005 Grenoble France
Workflow Monitoring
Monitor is a Pub/Sub client
Monitor/control workflow execution status
Workflow level
Activity level
Support dynamic workflow modification
Visualize topology of broker overlay network
Visualize message routing
Middleware 2005 Grenoble France
2019/1/15

23. Middleware 2005 Grenoble France
Evaluations
Environment
Intel Xeon 3GHz, 1GB RAM
JDK 1.4.2
Workload description
Predicates based on 20 attributes with random operators
Values are uniformly distributed in the value range
Two workflows with different number of jobs
Metrics
Matching time
Routing delay per composite subscription
Network traffic
zipf
Middleware 2005 Grenoble France
2019/1/15

24. Publication Matching Time
Evaluate the performance of the rule-based matching engine in PADRES.
X-axis is the number of subscriptions. Y-axis is the matching time of publications in logarithmic scale. Each data point is obtained by averaging the time taken to route 5000 publications.
We compared our PADRES matching engine with Naive Matching algorithm and Counting algorithm
Middleware 2005 Grenoble France
2019/1/15

25. Composite Event Detection
We measure the composite event detection time.
X-axis is the number of atomic subscriptions per CS. Y-axis is the detection time. We show the detection time of injecting 100, 5000, and publications.
The detection time includes: mapping the cs to a rule, and matching the rule against publication facts. Each data point is an average of 200 composite subscriptions.
Middleware 2005 Grenoble France
2019/1/15

26. Middleware 2005 Grenoble France
Routing Delay
We measure the routing delay of composite subscriptions.
X-axis is the number of atomic subscriptions per CS. Y-axis is the routing delay per broker. At each broker we need to split the CS according to the destinations. The complexity is O(N), where N is the number of atomic subscriptions per CS.
Middleware 2005 Grenoble France
2019/1/15

27. Number of Notifications
Composite subscriptions can reduce the number of notifications delivered to clients.
X-axis is the number of issued publications. Y-axis is the number of notifications that the client received.
We issue 200 composite subscriptions (each has 5 atomic subscriptions) in the first case (blue line).
And issue 1000 atomic subscriptions which consist the 200 CSs in the second case (red line)
Middleware 2005 Grenoble France
2019/1/15

28. Middleware 2005 Grenoble France
Workflow Traffic
The network traffic in the execution phase
Workflow A has 4 jobs
Workflow B has 8 jobs
The network traffic is generated by 10 generations of the workflow.
Middleware 2005 Grenoble France
2019/1/15

29. Middleware 2005 Grenoble France
Related Work
P.R. Pietzuch, B. Shand, and J. Bacon. Composite event detection as a generic middleware extension. IEEE Network Magazine, Special Issue on Middleware Technologies for Future Communication Networks, Jan 2004
SIENA
REBECA
Middleware 2005 Grenoble France
2019/1/15

30. Middleware 2005 Grenoble France
Conclusions
An expressive subscription language
A rule-based approach of message matching and composite event detection
Case study: a decentralized workflow management system based on PADRES
Future Works
Support BPEL4WS
QoS based workflow execution
Middleware 2005 Grenoble France
2019/1/15

31. Middleware 2005 Grenoble France
Questions?
Middleware 2005 Grenoble France
2019/1/15