Presentation is loading. Please wait.

Presentation is loading. Please wait.

Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey,

Similar presentations


Presentation on theme: "Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey,"— Presentation transcript:

1 Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, Hans-Arno Jacobsen Middleware Systems Research Group, University of Toronto DEBS Grand Challenge 2012

2 Agenda Complex Event Processing Scenarios System Architecture Evaluation Demo 18/07/20122msrg.org

3 Motivation Course Large-Scale Data Management Big data storage Large scale event processing Complex event processing scenarios Application performance management Smart traffic monitoring Energy monitoring Resulting team project DEBS Grand Challenge 18/07/20123msrg.org

4 Scenario I: Application Performance Management Monitoring of enterprise systems Find bottlenecks, problems Trace transactions, measure utilization 18/07/20124msrg.org

5 Scenario II: Smart Traffic Monitoring Traffic data from cars, mobile devices, road sensors Event aggregation, filtering, correlation Traffic status, accident detection, etc. 18/07/20125msrg.org

6 Scenario III: Energy Monitoring Green computing Application-level energy monitoring API-based energy consumption estimation Operating System & Hardware Applications CPU Network I/O Formulae Sensors … Store API 18/07/20126msrg.org

7 Common Denominators High data rates 1000 – events / sec Small data points < 1 KB Complex queries Filtering, aggregation, correlation Persistent storage Distributed setup DEBS Grand Challenge 18/07/20127msrg.org

8 Continuous Query Evaluation High-Level Architecture Monitoring Service Input data stream, marshalling Event Dissemination Substrate (Optional) pub/sub layer, queues Continuous Query Evaluation Consumes input, computes results Storage Manager Stores data, enables querying Client Visualizes query results Java-based implementation Monitoring Service Monitoring Service Storage Manager Storage Manager Client Event Dissemination Substrate Stable Storage 18/07/20128msrg.org

9 Storage Architecture Data is stored in tables Key-value pairs Table Index Fast lookup Compressor Efficient data storage Run length encoding Grand Challenge data Run-length: Compression: 99.99% 18/07/20129msrg.org

10 Client Google Web Toolkit-based Client Java-code compiled to JavaScript Displays all Grand Challenge results Plots, results, alarms 18/07/201210msrg.org

11 Evaluation Distributed setup Separated servers for data generator and monitoring tool Configuration 2 servers 2 x dual core Xeon processor 4 GB RAM Gigabit Ethernet Data set: 5 min + 18 days + synthetic Synthetic data Many errors (every min) Maximum throughput (no network) Metrics: latency & throughput 18/07/201211msrg.org

12 Processing Overhead Real Workload 0 – 200 queries (Q1,Q2 repeated) Linear in the number of queries Stable with increasing generator speedup 18/07/201212msrg.org

13 Throughput Real Workload Throughput controlled by data generator Data generator does not saturate system Peak 9000 events/sec ~ 0.11 ms arrival rate Well below maximum throughput 18/07/201213msrg.org

14 Latency Synthetic Workload Maximum throughput Minimum latency ms (10 queries) Maximum latency 0.63 ms (1400 queries) 18/07/201214msrg.org

15 Throughput Synthetic Workload Maximum throughput events / sec (20 queries) Minimum throughput 1500 events / sec (1400 queries) 18/07/201215msrg.org

16 Demo Live! 18/07/201216msrg.org

17 Conclusions Complex event processing scenarios Application performance management Smart traffic monitoring Energy monitoring DEBS Grand Challenge Efficient implementation of the Grand Challenge Java-based Google Web Toolkit GUI Synthetic data generator Up to events per second Up to 1400 queries 18/07/201217msrg.org

18 Thank You! Questions? Contact: Tilmann Rabl msrg.org 18/07/201218msrg.org

19 Back-Up: DEBS Grand Challenge Manufacturing equipment monitoring Large, real data set 18 days, 100 Hz 2 queries State of sensors and valves Difference Threshold Power consumption Range Average Threshold 18/07/201219msrg.org


Download ppt "Solving Manufacturing Equipment Monitoring Through Efficient Complex Event Processing Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey,"

Similar presentations


Ads by Google