Communication-Efficient Distributed Monitoring of Thresholded Counts Ram Keralapura, UC-Davis Graham Cormode, Bell Labs Jai Ramamirtham, Bell Labs

June 28, 2006 Ram Keralapura, UCDavis 2 Introduction Monitoring is critical to managing distributed networked systems Main challenges:  Continuous  Distributed  Resource-constrained environments

June 28, 2006 Ram Keralapura, UCDavis 3 Thresholded Counts New fundamental class of problems “Tracking counts for an event beyond a given threshold value with user-specified accuracy” Motivating scenarios:  Total # of connections to a server when it exceeds the normal operational condition (ex, DDoS attacks)  Total traffic to a particular destination prefix when it exceeds the pre-defined limit  Tracking the total number of cars on a highway

June 28, 2006 Ram Keralapura, UCDavis 4 Thresholded Counts (cont’d) Two key properties  Threshold value  User specified tracking accuracy

June 28, 2006 Ram Keralapura, UCDavis 5 System Architecture Remote Site 1 Remote Site m Remote Site 2 Remote Site i Coordinator Site (Central Node) remote sites (or monitors) and a coordinator site (or central node) m Non-continuous updates Local thresholds at remote sites Counts can be positive, negative, or fractional Ignore network delays and losses

June 28, 2006 Ram Keralapura, UCDavis 6 Every remote monitor, maintains a set of local thresholds: Local count at monitor, should always lie between two neighboring thresholds Global estimate at the central node: Approach ii

June 28, 2006 Ram Keralapura, UCDavis 7 Approach (cont’d) Maximum error in the global estimate should satisfy: Two methods to set local thresholds  Static thresholding  Adaptive thresholding

June 28, 2006 Ram Keralapura, UCDavis 8 Static Thresholding Problem: For given values and, we have to determine such that, T  ),0[,,  jt ji

Uniform Proportional Monitor-1 Monitor-2 Monitor-3 Central Node Blended threshold assignment T  Max error = N 

June 28, 2006 Ram Keralapura, UCDavis 10 Static Thresholding (cont’d) Blended threshold assignment   uniform threshold assignment   proportional threshold assignment Complexity: 0  1 

June 28, 2006 Ram Keralapura, UCDavis 11 Adaptive Thresholding Every monitor maintains only two threshold values: and Problem: For given values and, and a threshold violation from monitor, determine for all the monitors such that, T  k iH t

Slack Monitor-1Monitor-2Monitor-3 Central Node Monitor-1Monitor-2Monitor-3 Central Node TN)1( ˆ  TN)1( ˆ  Basic Adaptive Algorithm

June 28, 2006 Ram Keralapura, UCDavis 13 Experimental Setup Built a simulator with monitoring nodes and a central node Implemented all the static and adaptive algorithms Data set: Public traces from NLANR m

June 28, 2006 Ram Keralapura, UCDavis 14 Count Accuracy

June 28, 2006 Ram Keralapura, UCDavis 15 Validating the Theoretical Model

June 28, 2006 Ram Keralapura, UCDavis 16 Comparing Costs – Static and Adaptive Cases

June 28, 2006 Ram Keralapura, UCDavis 17 Related Work Top-k monitoring [Babcock et al] Heavy-hitter definition Adaptive filters for continuous queries [Olston et al]  Distributed continuous queries but does not address the thresholded counts problem Distributed triggers [Jain et al]  Simplified version of the thresholded counts problem  Randomized algorithms with statistical guarantees Geometric approach for threshold functions [Sharfman et al]  Focus is mainly on non-linear functions

June 28, 2006 Ram Keralapura, UCDavis 18 Summary We defined a fundamental class of problems called “Thresholded Counts” We proposed algorithms to address the problem – static and adaptive Analyzed the complexities of these algorithms and provided proofs Using experiments, we showed the effectiveness of our algorithms

June 28, 2006 Ram Keralapura, UCDavis 19 Future Work Building the monitoring system for real networks to explore the practical aspects of our framework  Sensor networks  IP network monitoring Address scalability issues  For example, hierarchical monitoring architecture Extend for different query types with thresholded nature  For example, arithmetic combinations

June 28, 2006 Ram Keralapura, UCDavis 20 Thank you!! Questions?? Contact: