Classification and Analysis of Distributed Event Filtering Algorithms Sven Bittner Dr. Annika Hinze University of Waikato New Zealand Presentation at CoopIS Cyprus

2/25 Motivation Event Notification System e 2 : t=30°C e 3 : r=0,2 liter e 4 : r=2 liter e 1 : t=15°C Events Filtering Efficient scalable filtering  Messages (e2)(e2)(e2)(e2) (e1)(e1)  (e 3 ),  (e 4 )    Profiles Subscribers p 1 =(t>22°C) p 2 =(t<18°C) p 3 =(r>0,1 lit.) Providers(Sensors)  Facilty Mangement (one building) >10 4 Profiles >10 4 Profiles >10 3 Events/Second >10 3 Events/Second

3/25 ClassificationTheoretical analysisExperimentsOpen Research Distributed Filtering ENS ENS Distributed 6 2 Central Filter- components 6 2 Acyclic overlay network for distribution of profiles and events - manage set of exclusive local clients - forward matching event messages to clients Clients communicate with Brokers S P

4/25 State of the Art: Distributed algorithms  Rendezvous nodes: certain brokers specialise in filtering of certain event types, meeting meeting points for profiles and event messages [11,12]  Event/profile forwarding: –brokers know all profiles, propagates events down the tree [1] –Brokers only know profiles registered at it‘s children, event forwarded up to root and then down to leaves [3,14] –Point2point: broker knows about profiles of neighbours, forwarding of events or profiles [3]  Optimizations: –Covering and merging of overlapping profiles [9,10] –Immediate covering computation [3] –Covering computation of request [7]

5/25 Problem definition: Efficient algorithms  Evaluations so far –Mostly based on simulated network topologies –Restricted analysis of influencial factors –Independently for single algorithms (ie under different conditons) Open issues: 1. classification scheme for distributed filter algorithms 2. uniform performance analysis of filter algorithms that allows for a comparison of the algorithms’ efficiency

6/25 Roadmap  Motivation  Classification –Schema for distributed filter algorithms –Classification of filter algorithms  Theoretical performance analysis  Experimental performance analysis  Conclusion and Outlook ClassificationTheoretical analysisExperimentsOpen Research

7/25 Classification: Dimensions  Location of filtering –Subscribers, Publishers, Arbitrary  Spreading of Filter complexity –Exclusive –Distributed Storage strategies: preventive vs optimitsticStorage strategies: preventive vs optimitstic  Communication with subscribers –Direct, forwarding, transparent ClassificationTheoretical analysisExperimentsOpen Research

8/25 Classification: Dimensions  Filter Location –Close to subscribers: Profile submitted only to closest broker, flooding of event messages over the network (event flooding) –Close to publishers: Events published only to closest broker, flooding of profiles over the network (profile flooding) ENS Distributed PS –Arbitrary dedicated brokers responsible for filtering of specific message types, forwarding of events and profiles to these brokers (rendezvous nodes) ClassificationTheoretical analysisExperimentsOpen Research

9/25 Classification: Dimensions  Spreading of Filter complexity –Exlusive filtering at certain brokers –Distributed: broker filters so that the neighbours with matching profiles are found ENS Distributed PS –Storage Strategies: Preventive – store all available profiles, also duplicates and covered ones (more memory, faster unsubscriptions)Preventive – store all available profiles, also duplicates and covered ones (more memory, faster unsubscriptions) Optimistic – minimize number of stored profiles by discarding covered ones (less memory)Optimistic – minimize number of stored profiles by discarding covered ones (less memory) ClassificationTheoretical analysisExperimentsOpen Research

10/25 Classification: Dimensions  Communication with subscribers –Direct: only filtering broker and subscriber are involved (costly estabishing communication) ENS Distributed PS –Forwarding: only neighbor brokers and local clients (higher memory consumption) –Transparent: communication via broker proxiescommunication via broker proxies Brokers act as subscribers to their neighbors (limits number of subscribers to each broker)Brokers act as subscribers to their neighbors (limits number of subscribers to each broker) ClassificationTheoretical analysisExperimentsOpen Research

11/25 ClassificationTheoretical analysisExperimentsOpen Research Classification: Schema 17 possible variations Existing algorithms & systems:  Precise classification difficult (lack of detail)  Often several options possible  Not all variations implemented

12/25 Roadmap  Motivation  Classification  Theoretical performance analysis  Experimental performance analysis  Conclusion and Outlook ClassificationTheoretical analysisExperimentsOpen Research

13/25 Theoretical Evaluation: Dimensions –Network Traffic (worst case: blocking) –Memory Usage (worst case: swapping) –Efficiency / Performance –Scalability

14/25 Theoretical Evaluation: Example  Event flooding –Filtering close to subscribers –Exclusive filtering –Direct communication P1P1 e S p p3p3   (e) – High traffic due to flooding with high frequency – Low memory usage (profiles stored just once) – Medium performance – Low scalability due to traffic increase ClassificationTheoretical analysisExperimentsOpen Research

15/25 Roadmap  Motivation  Classification  Theoretical performance analysis  Experimental performance analysis  Conclusion and Outlook ClassificationTheoretical analysisExperimentsOpen Research

16/25 Practical evaluation  Selection of 3 algorithms 1.Event Flooding: Close to subscribers, exclusive filtering, direct communication 2.Profile flooding: Close to publishers, distributed filtering, optimistic storage, transparent communication 3.Rendezvous nodes: Arbitrary, distributed filtering, optimistic storage, transparent communication ClassificationTheoretical analysisExperimentsOpen Research

17/25 Practical evaluation: Experiments  Parameters –Number of profiles –Proportion of matching events –Proportion of matching profiles –Number of brokers –Covering between profiles –Number of different event types –Locality bewteen events and profiles  Implementation of algorithms in system DAS (distributed version of A-mediAS) Number of profiles Number of profiles Proportion of matching events Proportion of matching events Proportion of matching profiles Proportion of matching profiles Number of brokers Number of brokers ClassificationTheoretical analysisExperimentsOpen Research

18/25 Practical evaluation: Experiments (1)  Influence of number of profiles ClassificationTheoretical analysisExperimentsOpen Research PFEFRN

19/25 Practical evaluation: Experiments (2)  Influence of proportion of matching events ClassificationTheoretical analysisExperimentsOpen Research PFEFRN

20/25 Practical evaluation: Experiments (3)  Influence of number of brokers ClassificationTheoretical analysisExperimentsOpen Research PFEFRN

21/25 Roadmap  Motivation  Classification  Theoretical performance analysis  Experimental performance analysis  Conclusion and Outlook ClassificationTheoretical analysisExperimentsOpen Research

22/25 Conclusion & Outlook: Selected results  Proposed classification schema  First classification of algorithms  Theoretical evaluation of 17 variations –limited information available –various options  Practical evaluation of 3 variations –Evaluated 7 parameters –Restriction in practicality of number of brokers –Limitation of topology variations ClassificationTheoretical analysisExperimentsOpen Research

23/25 Conclusion & Outlook: Selected results  Profile forwarding –Mostly best efficiency and lowest network load –Largest memory consumption  Event forwarding –Very high network load –Optimal memory usage –High proportion of matching profiles  best efficiency –Large number of profiles  best scalability  Rendezvous nodes –In none of the configurations better than the other algorithms (influence of topology?) ClassificationTheoretical analysisExperimentsOpen Research

24/25 Conclusion & Outlook  Optimal algorithm depends on –system load, –system usage, and –application --> System should support various filter algorithms and adapt dynamically to changing situations --> System should support various filter algorithms and adapt dynamically to changing situationsOutlook: –Integrate in adaptive system A-mediAS –Composite filtering algorithms in grids and mobile environments ClassificationTheoretical analysisExperimentsOpen Research

Thank you for your attention! And now…questions. Contact: Annika Hinze