1. Recent Results in Resource Signal Measurement, Dissemination, and Prediction App Transport Network Data Link Physical App Transport Network Data Link Physical Sensor Header Editing Consumer Data Extraction Sensor data piggybacked on existing application packets Number and size of packets unchanged Data TCPIPEthernetPadding Overwrite unused or redundant fields with sensor data Shannon entropy of packet headers: 4.8 bits per byte In practice: 17..32 bits at IP and TCP, padding varies Implementations: Minet, Linux Kernel Modules [LCR 2002, NWU-CS-02-12] http://www.cs.northwestern.edu/~plab/Diffusion Sensor Data Diffusion: Zero Cost Information Dissemination (with Brian Cornell, Jack Lange, NSF REU) Peter A. Dinda Prescience Lab Department of Computer Science Northwestern University.cs.northwestern.edu Wavelet Transform Level 0 Sensor Inverse Wavelet Transform Application Level M-1 Level M Level 0 Level L Multicast Sensor sends all levels appropriate to sampling rate Each application receives levels based on its needs Applications and sensors decoupled Level rates decrease logarithmically. StreamInterval Sensor Video App Course-grain measurement Fine-grain measurement Grid App … Resource Signal (periodic sampling) Example: host load Resource- appropriate measurement Tension between different application needs Application and sensor needs coupled Inefficient bandwidth usage, especially in unicast Limited proof of concept implemented in RPS: Works Wavelet toolbox in next RPS release Biggest issue: The Delay Problem Transforms introduce sample delay Exponential in the number of levels Affects both streaming and block transforms Current efforts to overcome the delay problem Exploit prediction (limited success so far) Exploit “wavelet-like” decompositions that can trade-off between reconstruction accuracy and delay [HPDC 2001] http://www.cs.northwestern.edu/~plab/Tsunami Tsunami: Wavelet-based Approaches (with Jason Skicewicz) Multiscale Prediction of Network Bandwidth (with Yi Qiao, Jason Skicewicz) Large study of predictability of binned packet traces Offline RPS predictors (linear models) Different resolutions Both power-of-two binning and low-pass via D8 wavelets Over 200 NLANR and other traces Mostly WANs All long period traces available at time of trace Random selection of short-term traces Hierarchical classification of traces Short period, Long Period, Bellcore Predictability using linear models highly variable Many traces unpredictabile white noise Predictability varies with resolutiion Sweet Spot: Predictability often maximized at particular resolution [NWU-CS-02-12, NWU-CS-02-13] http://www.cs.northwestern.edu/~plab/Clairvoyance RPS: The Resource Prediction System Toolkit RPS: Measure, Predict, and Disseminate information about dynamic resource supply Ultimate goal: provide advice to adaptive applications Publicly available, Extensible, Portable, Easy buy-in Resource signals: Discrete-time signals strongly correlated with resource supply Host load Windows performance counters (using WatchTower) Network flow bandwidth and latency (using Remos) Any text-based source Online predictive modeling Simple models (MEAN, BESTMEAN, BESTMEDIAN, LAST…) Box/Jenkins Models (AR, MA, ARMA, ARIMA,…) Fractional ARIMAs Nonlinear modeling (TARs, Wavelet-decompositions) [HPDC 99, Cluster 00, Cluster 02, SIGMETRICS 01, IPDPS 02, SC 01, SHAMAN 02] http://www.cs.northwestern.edu/~RPS