D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 1/36 Towards a Meaningful MRA for Traffic Matrices D. Rincón, M. Roughan, W. Willinger IMC 2008

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 2/36 Outline Seeking a sparse model for TMs Multi-Resolution Analysis on graphs with Diffusion Wavelets MRA of TMs: preliminary results Open issues

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 3/36 Context: Abilene 12 nodes (2004) Abilene topology (2004) STTL SNVA DNVR LOSA KSCY HSTN IPLS ATLA CHIN NYCM WASH ATLA-M5

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 4/36 Example: Abilene traffic matrix

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 5/36 Traffic matrices Open problems Good TM models Synthesis of TMs for planning / design of networks Traffic prediction – anomaly detection Traffic engineering algorithms Traffic and topology are intertwined Hierarchical scales in the global Internet apply also to traffic Time evolution of TMs How to reduce the dimensionality catch of the inference problem? Our goals Can we find a general model for TMs? Can we develop Multi-Resolution machinery for jointly analyzing topology and traffic, in spatial and time scales?

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 6/36 Can we find a general model for TMs? Our criterion: the TM model should be sparse Sparsity: energy concentrates in few coefficients (M << N 2 ) Tradeoff between predictive power and model fidelity Easier to attach physical meaning Could help with the underconstrained inference problem Multiresolution analysis (MRA) “Classical MRA”: wavelet transforms observe the data at different time / space resolutions Wavelets (approximately) decorrelate input signals Energy concentrates in few coefficients Threshold the transform coefficients  sparse representation (denoising, compression) Successfully applied in time series (1D) and images (2D)

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 7/36 How to perform MRA on TMs? Traffic matrices are 2D functions defined on a graph 2D Discrete Wavelet Transform of TM as images Uniform sampling in R 2 TMs are NOT images! – the intrinsic geometry is lost Graph wavelets (Crovella & Kolaczyk, 2003) Spatial analysis of differences between link loads -anomaly detection Drawbacks of the graph wavelets approach –Non-orthogonal transform - overcomplete representation –Lack of fast computation algorithm

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 8/36 How to perform MRA on TMs? Diffusion Wavelets (Coifman & Maggioni. 2004) MRA on manifolds and graphs Diffusion operator “learns” the underlying geometry as powers increase – random walk steps Amount of “important” eigenvalues - vectors decreases with powers of T Those under certain precision  are related to high-frequency details, while those over  are related to low-frequency approximations Operator T W1W1 V1V1 W2W2 V2V2 W3W3 V3V3

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 9/36 How to perform MRA on TMs? Diffusion Wavelets (Coifman & Maggioni. 2004) Operator T W1W1 V1V1 W2W2 V2V2  Eigenvalues (low to high frequency) C v2 5C W2 3C W1 2

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 10/36 Diffusion Wavelets and our goals Unidimensional functions of the vertices F(v 1 ) can be projected onto the multi-resolution spaces defined by the DW. Network topology can be studied by defining the right operator and representing the coarsened versions of the graph. But Traffic Matrices are 2D functions of the origin and destination vertices, and can also be functions of time: TM(V 1,V 2,t)

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 11/36 2D Diffusion wavelets Extension of DW to 2D functions defined on a graph F(v 1,v 2 ) Construction of separable 2D bases by “projecting twice” into both “directions” Tensor product Similar to 2D DWT Orthonormal, invertible, energy conserving transform Operator T VW 1 WW 1 WV 1 VV 1 VW 2 WW 2 WV 2 VV 2 VW 3 WW 3 WV 3 VV 3

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 12/36 2D Diffusion wavelets Extension of DW to 2D functions defined on a graph Operator T VW 1 WW 1 WV 1 VV 1 VW 2 WW 2 WV 2 VV 2

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 13/36 MRA of Traffic Matrices More than TMs from operational networks Abilene (2004), granularity 5 mins GÉANT (2005), granularity 15 mins Acknowledgments: Yin Zhang (UTexas), S. Uhlig (Delft), Diffusion operator: A: unweighted adjacency matrix “Symmetrised” version of the random walk – same eigenvalues Double stochastic (!) Precision ε = 10 -7

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 14/36 2D Diffusion wavelets – Abilene example W1W1 V1V1 W2W2 V2V2 W3W3 V3V3 W4W4 V4V4 V0V0 W5W5 V5V5 W6W6 V6V6 W7W7 V7V7 W8W8 V8V8 V4V4 # eigenvalues at each subspaceW j = WV j + VW j + WW j

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 15/36 2D Diffusion wavelets – Abilene example STTL SNVA DNVR LOSA KSCY HSTN IPLS ATLA CHIN NYCM WASH ATLA-M5

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 16/36 2D Diffusion wavelets – Abilene example

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 17/36 2D Diffusion wavelets – Abilene example DW coefficients Abilene 14th July 2004 (24 hours) Coefficient index (high to low freq) Time (5 min intervals)

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 18/36 2D Diffusion wavelets – Abilene example How concentrated is the energy of the TM? Wavelet coefficients for the Abilene TM 12 x 12 = 144 coefficients, low- to high-frequency Coefficients – high to low frequency

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 19/36 Compressibility of TMs

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 20/36 Stability of DW coefficients

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 21/36 Rank signature Coefficient index Time (5 min intervals) Coefficient rank – Abilene March 2004

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 22/36 Rank signature – anomaly detection?

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 23/36 Conclusions and open issues Representation of TMs in the DW domain TMs seem to be sparse in the DW domain Consistency across time and different networks Ongoing work Develop a sparse model for TMs How the sparse representation relates to previous models (e.g. Gravity) ? Exploit DW’s dimensionality reduction in the inference problem Exploring weighted / routing-related diffusion operators Exploring bandwidth-related diffusion operators Introducing time correlations in the diffusion operator Diffusion wavelet packets – best basis algorithms for compression DW analysis of network topologies

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 24/36 Thank you ! Questions?

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 25/36 Extra slides

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 26/36 Géant 23 nodes (2005)

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 27/36 Context: topology Spatial hierarchy AS1 AS2 AS3 PoPs Access Networks Network/AS

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 28/36 Multi-Resolution Analysis Intuition: “to observe at different scales”

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 29/36 Multi-Resolution Analysis Approximations: coarse representations of the original data

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 30/36 Multi-Resolution Analysis Mathematical formalism Set of nested scaling subspaces (low- frequency approximations) generated by the scaling functions The orthogonal complement of V i inside V i+1 are called detail (high-frequency) or wavelet subspaces W i, generated by wavelet functions W1W1 V1V1 W2W2 V2V2 W3W3 V3V3 V0V0

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 31/36 Multi-Resolution Analysis Scaling functions: averaging, low- frequency functions Wavelet functions: differencing, high- frequency functions

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 32/36 Multi-Resolution Analysis (2D) Separable bases: horizontal x vertical Example: 2D scaling function

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 33/36 Wavelet transform example 2D wavelet decomposition of the image for j=2 levels Vertical/horizontal high/low frequency subbands

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 34/36 Our approach Can we develop Multi-Resolution machinery for analyzing topology and traffic, in spatial and time scales? Classical 1D or 2D wavelet transforms are not an option We need a new graph-based wavelet transform! Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Diffusion wavelets (M. Maggioni et al, 06)

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 35/36 The tools: Graph wavelets Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Exploit spatial correlation of traffic data Sampled 2D wavelets

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 36/36 The tools: Graph wavelets Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Link analysis Definition of scale j: j-hop neighbours

D. Rincón, M. Roughan, W. Willinger – Towards a Meaningful MRA of Traffic Matrices 37/36 The tools: Graph wavelets j=1 j=3 j=5 traffic Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) Anomaly detection in Abilene