Multiscale Traffic Processing Techniques for Network Inference and Control Richard Baraniuk Edward Knightly Robert Nowak Rolf Riedi Rice University INCITE Project April 2001

Rice University | INCITE.rice.edu | April 2001 InterNet Control and Inference Tools at the Edge Overall Objective: Scalable, edge-based tools for on-line network analysis, modeling, and measurement Theme for DARPA NMS Research: Multiscale traffic analysis, modeling, and processing via multifractals Expertise: Statistical signal processing, mathematics, network QoS Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Technical Challenges Poor understanding of origins of complex network dynamics Lack of adequate modeling techniques for network dynamics Internal network inaccessible Need: Manageable, reduced-complexity models with characterizable accuracy Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Multiscale modeling Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Multiscale Analysis Time Multiscale statistics Var1 Scale Var2 Var3 Analysis: flow up the tree by adding Varj Start at bottom with trace itself Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Multiscale Synthesis Time Start at top with total arrival Multiscale parameters Var1 Scale Var2 Var3 Synthesis: flow down via innovations Varj Signal: bottom nodes Rice University | INCITE.rice.edu | April 2001

Multifractal Wavelet Model (MWM) Random multiplicative innovations Aj,k on [0,1] eg: beta Parsimonious modeling (one parameter per scale) Strong ties with rich theory of multifractals Rice University | INCITE.rice.edu | April 2001

Multiscale Traffic Trace Matching Auckland 2000 MWM match 4ms 16ms 64ms Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Multiscale Queuing Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Probing the Network Rice University | INCITE.rice.edu | April 2001

Probing Ideally: delay spread of packet pair spaced by T sec correlates with cross-traffic volume at time-scale T Rice University | INCITE.rice.edu | April 2001

Probing Uncertainty Principle Should not allow queue to empty between probe packets Small T for accurate measurements but probe traffic would disturb cross-traffic (and overflow bottleneck buffer!) Larger T leads to measurement uncertainties queue could empty between probes To the rescue: model-based inference Rice University | INCITE.rice.edu | April 2001

Multifractal Cross-Traffic Inference Model bursty cross-traffic using MWM Rice University | INCITE.rice.edu | April 2001

Efficient Probing: Packet Chirps MWM tree inspires geometric chirp probe MLE estimates of cross-traffic at multiple scales Rice University | INCITE.rice.edu | April 2001

Chirp Probe Cross-Traffic Inference Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 ns-2 Simulation Inference improves with increased utilization Low utilization (39%) High utilization (65%) Rice University | INCITE.rice.edu | April 2001

ns-2 Simulation (Adaptivity) Inference improves as MWM parameters adapt MWM parameters Inferred x-traffic Rice University | INCITE.rice.edu | April 2001

Adaptivity (MWM Cross-Traffic) Eg: Route changes Rice University | INCITE.rice.edu | April 2001

Comparing Probing schemes Rice University | INCITE.rice.edu | April 2001

Comparing probing schemes `Classical’: Bandwidth estimation by packet pairs and trains Novel: Traffic estimation, probing best by Uniform? Poisson? Chirp? Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Model based Probing Chirp: model based, superior Uniform: Uncertainty increases error Rice University | INCITE.rice.edu | April 2001

Impact of Probing on Performance Heavy Heavy probing - reduces bandwidth - increases loss - inflicts time-outs NS-simulation: Same `web-traffic’ with variable probing rates Light Rice University | INCITE.rice.edu | April 2001

Influence of probing rate on error Chirp probing performing uniformly good Uniform requires higher rates to perform Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Synergies SAIC (Warren): MWM code for real time simulator SLAC (Cottrell, Feng): Modify PingER for chirp-probing High performance networks Demo: C-code for real world chirp-probing using NetDyn (TCP) + simple Daemon at receiver (INRIA France, UFMG Brazil, Michigan State) Rice University | INCITE.rice.edu | April 2001

INCITE: Near-term / Ongoing Verification with real Internet experiments Rice testbed (practical issues) SAIC (real time algorithms) SLAC / ESNet (real world verification) Enhancements: rigorous statistical error analysis deal with random losses multiple bottleneck queues (see demo) passive monitoring (novel models) closed loop paths/feedback (ns-simulation) Rice University | INCITE.rice.edu | April 2001

INCITE: Longer-Term Goals New traffic models, inference algorithms theory, simulation, real implementation Applications to Control, QoS, Network Meltdown early warning Leverage from our other projects ATR program (DARPA, ONR, ARO) RENE (Rice Everywhere Network:NSF) NSF ITR DoE Rice University | INCITE.rice.edu | April 2001

Stationary multifractals 40 Minutes 35 Minutes: instead of 3 slides on multifractal bursts just one summary (see end) Rice University | INCITE.rice.edu | April 2001

Stationary multiplicative models j(s): stationary, indep., E[j(s)]=1 A(t) = lim 0t 1(s) 2(s)… n(s) ds May degenerate (compare: MWM is conservative) stationary increments Assume j(2j s) are i.i.d.; Renewal reward Compare MWM: j(2j s) constant over [k,k+1] If Var()<1: Convergence in L2 ; E[A(t)]=t Multifractal function: T(q)=q-log2E[q] Compare: MWM has Lambda that is constant in integer intervals: deterministic geometry Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Simulation L2 criterion for convergence translates to T(2)>0 Conjecture: For q>1 converge in Lq if T(q)>0 Thus non-degenerate iff T’(1)>0, ie E[ L log (L /2) ] >0 Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001 Parameter estimation No conservation: can’t isolate multipliers Possible correlation within multipliers IID values: Z(s) = log [ 1(s) 2(s)… n(s) ] Cov(Z(t)Z(t+s))= Si=1..n exp(-lis)Var i(s) `LRD-scaling’ at medium scales, but SRD. Multifractal subordination -> true LRD. One can thus fit the auto-correlation and induce the parameters li and Var i(s) For MWM in log domain: Cov(Z(t)Z(t+s))= S Var log[i(s)] Where the sum goes now over all I such that t and t+s are in the same Dyadic interval of order I. Recursive computation of var possible. Advantage of working in real time (not log time): can compute the actual i(s) for all i and s (step functions) because of conservation of mass, thus Obtain distr, not only var. Rice University | INCITE.rice.edu | April 2001

Rice University | INCITE.rice.edu | April 2001