1. AT&T Labs - Research Internet Measurement Conference 2003 27-29 Of October, 2003 Miami, Florida, USA http://www.icir.org/vern/imc-2003/ Date for student travel grant applications: Sept 5th

2. AT&T Labs - Research An Information-Theoretic Approach to Traffic Matrix Estimation Yin Zhang, Matthew Roughan, Carsten Lund – AT&T Research David Donoho – Stanford Shannon Lab

3. AT&T Labs - Research Want to know demands from source to destination Problem Have link traffic measurements A B C

4. AT&T Labs - Research Example App: reliability analysis Under a link failure, routes change want to find an traffic invariant A B C

5. AT&T Labs - Research Approach Principle * “Don’t try to estimate something if you don’t have any information about it” zMaximum Entropy yEntropy is a measure of uncertainty xMore information = less entropy yTo include measurements, maximize entropy subject to the constraints imposed by the data yImpose the fewest assumptions on the results zInstantiation: Maximize “relative entropy” yMinimum Mutual Information

6. AT&T Labs - Research Mathematical Formalism Only measure traffic at links 1 3 2 router link 1 link 2 link 3 Traffic y 1

7. AT&T Labs - Research Mathematical Formalism 1 3 2 router route 2 route 1 route 3 Problem: Estimate traffic matrix (x’s) from the link measurements (y’s) Traffic y 1 Traffic matrix element x 1

8. AT&T Labs - Research Mathematical Formalism 1 3 2 router route 2 route 1 route 3 Problem: Estimate traffic matrix (x’s) from the link measurements (y’s)

9. AT&T Labs - Research Mathematical Formalism 1 3 2 router route 2 route 1 route 3 Problem: Estimate traffic matrix (x’s) from the link measurements (y’s)

10. AT&T Labs - Research Mathematical Formalism 1 3 2 router route 2 route 1 route 3 For non-trivial network UNDERCONSTRAINED y = Ax Routing matrix

11. AT&T Labs - Research Regularization zWant a solution that satisfies constraints: y = Ax yMany more unknowns than measurement: O(N 2 ) vs O(N) yUnderconstrained system yMany solutions satisfy the equations yMust somehow choose the “best” solution zSuch (ill-posed linear inverse) problems occur in yMedical imaging: e.g CAT scans ySeismology yAstronomy zStatistical intuition => Regularization yPenalty function J(x) ysolution:

12. AT&T Labs - Research How does this relate to other methods? zPrevious methods are just particular cases of J(x) zTomogravity (Zhang, Roughan, Greenberg and Duffield) yJ(x) is a weighted quadratic distance from a gravity model zA very natural alternative yStart from a penalty function that satisfies the “maximum entropy” principle yMinimum Mutual Information

13. AT&T Labs - Research Minimum Mutual Information (MMI) zMutual Information I(S,D) yInformation gained about Source from Destination yI(S,D) = -relative entropy with respect to independent S and D I(S,D) = 0 S and D are independent p(D|S) = p(D) gravity model zNatural application of principle * yAssume independence in the absence of other information yAggregates have similar behavior to network overall zWhen we get additional information (e.g. y = Ax) yMaximize entropy  Minimize I(S,D) (subject to constraints) yJ(x) = I(S,D) equivalent

14. AT&T Labs - Research MMI in practice zIn general there aren’t enough constraints zConstraints give a subspace of possible solutions y = Ax

15. AT&T Labs - Research MMI in practice zIndependence gives us a starting point y = Ax independent solution

16. AT&T Labs - Research MMI in practice zFind a solution which ySatisfies the constraint yIs closest to the independent solution solution Distance measure is the Kullback-Lieber divergence

17. AT&T Labs - Research Is that it? zNot quite that simple yNeed to do some networking specific things ye.g. conditional independence to model hot-potato routing zCan be solved using standard optimization toolkits yTaking advantage of sparseness of routing matrix A zBack to tomogravity yConditional independence = generalized gravity model yQuadratic distance function is a first order approximation to the Kullback-Leibler divergence yTomogravity is a first-order approximation to MMI

18. AT&T Labs - Research Results – Single example z±20% bounds for larger flows zAverage error ~11% zFast (< 5 seconds) zScales: yO(100) nodes

19. AT&T Labs - Research More results tomogravity method simple approximation >80% of demands have <20% error Large errors are in small flows

20. AT&T Labs - Research Other experiments zSensitivity yVery insensitive to lambda ySimple approximations work well zRobustness yMissing data yErroneous link data yErroneous routing data zDependence on network topology yVia Rocketfuel network topologies zAdditional information yNetflow yLocal traffic matrices

21. AT&T Labs - Research Dependence on Topology clique star (20 nodes)

22. AT&T Labs - Research Additional information – Netflow

23. AT&T Labs - Research Local traffic matrix (George Varghese) for reference previous case 0% 1% 5% 10%

24. AT&T Labs - Research Conclusion zWe have a good estimation method yRobust, fast, and scales to required size yAccuracy depends on ratio of unknowns to measurements yDerived from principle zApproach gives some insight into other methods yWhy they work – regularization yShould provide better idea of the way forward zAdditional insights about the network and traffic yTraffic and network are connected zImplemented yUsed in AT&T’s NA backbone yAccurate enough in practice

25. AT&T Labs - Research Additional Slides

26. AT&T Labs - Research Results zMethodology yUse netflow based partial (~80%) traffic matrix ySimulate SNMP measurements using routing sim, and y = Ax yCompare estimates, and true traffic matrix zAdvantage yRealistic network, routing, and traffic yComparison is direct, we know errors are due to algorithm not errors in the data yCan do controlled experiments (e.g. introduce known errors) zData yOne hour traffic matrices (don’t need fine grained data) y506 data sets, comprising the majority of June 2002 yIncludes all times of day, and days of week

27. AT&T Labs - Research Robustness (input errors)

28. AT&T Labs - Research Robustness (missing data)

29. AT&T Labs - Research Point-to-multipoint We don’t see whole Internet – What if an edge link fails? Point-to-point traffic matrix isn’t invariant

30. AT&T Labs - Research Point-to-multipoint zIncluded in this approach zImplicit in results above zExplicit results worse yAmbiguity in demands in increased yMore demands use exactly the same sets of routes zuse in applications is better Point-to-point Point-to-multipoint Link failure analysis

31. AT&T Labs - Research Independent model

32. AT&T Labs - Research Conditional independence zInternet routing is asymmetric zA provider can control exit points for traffic going to peer networks peer links access links

33. AT&T Labs - Research Conditional independence peer links access links zInternet routing is asymmetric zA provider can control exit points for traffic going to peer networks zHave much less control of where traffic enters

34. AT&T Labs - Research Conditional independence

35. AT&T Labs - Research Minimum Mutual Information (MMI) zMutual Information I(S,D)=0 yInformation gained about S from D yI(S,D) = relative entropy with respect to independence yCan also be given by Kullback-Leibler information divergence zWhy this model yIn the absence of information, let’s assume no information yMinimal assumption about the traffic yLarge aggregates tend to behave like overall network?

36. AT&T Labs - Research Dependence on Topology Unknowns per RelativeErrors (%) NetworkPoPsLinks measurement GeographicRandom Exodus*17584.6912.620.0 Sprint*191003.428.018.9 Abovenet*11482.293.811.7 StarN2(N-1)N/2=1024.0 CliqueNN(N-1)10.2 AT&T--3.54-3.9710.6 * These are not the actual networks, but only estimates made by Rocketfuel

37. AT&T Labs - Research yBayesian (e.g. Tebaldi and West) xJ(x) = -log  (x), where  (x) is the prior model yMLE (e.g. Vardi, Cao et al, …) xIn their thinking the prior model generates extra constraints xEqually, can be modeled as a (complicated) penalty function Uses deviations from higher order moments predicted by model

38. AT&T Labs - Research Acknowledgements zLocal traffic matrix measurements yGeorge Varghese zPDSCO optimization toolkit for Matlab yMichael Saunders zData collection yFred True, Joel Gottlieb zTomogravity yAlbert Greenberg and Nick Duffield