1. NetQuest: A Flexible Framework for Large-Scale Network Measurement Lili Qiu University of Texas at Austin lili@cs.utexas.edu Joint work with Han Hee Song and Yin Zhang ACM SIGMETRICS 2006 June 27, 2006

2. 2 C&W UUNet AOL Earthlink Sprint Qwest AT&T Motivating Scenario I Network Diagnosis Why is it so slow?

3. 3 Motivating Scenario II Performance Monitoring in ISP Networks ISP

4. 4 Key Requirements Scalable: work for large networks (e.g., thousands of nodes) Flexible: accommodate different applications –Differentiated design Different quantities have different importance, e.g., a subset of paths belong to a major customer –Augmented design Conduct additional experiments given existing observations, e.g., after measurement failures –Multi-user design Multiple users interested in different parts of network or have different objective functions Q: Which measurement to conduct to estimate the quantities of interest?

5. 5 What We Want A function f(x) of link performance x –We use a linear function f(x)=F*x in this talk 2 5 1 4 7 6 x1x1 x 11 x4x4 x5x5 x 10 x9x9 x7x7 x6x6 x8x8 3 x2x2 x3x3 Ex. 1: average link delay f(x) = (x1+…+x11)/11 Ex. 2: end-to-end delay Apply to any additive metric, eg. Log (1 – loss rate)

6. 6 Problem Formulation What we can measure: e2e performance Network performance estimation –Goal: e2e performance on some paths  f(x) Input: y S (end-to-end performance on a subset of paths S), A S, and y S =A S x Output: f(x) –Design of measurement experiments Select which subset of paths S for active probing –Network inference Infer x based on partial indirect observations

7. 7 Design of Experiments State of the art –Probe every path (e.g., RON) Not scalable since # paths grow quadratically with #nodes –Rank-based approach [sigcomm04] Let A denote routing matrix Monitor rank(A) paths that are linearly independent to exactly reconstruct end-to-end path properties Still very expensive Our work –If we can tolerate some error, we can significantly reduce measurement cost. –How to select a given # paths to probe to estimate f(x) as accurately as possible? Need a metric to quantify goodness of a given set of paths

8. 8 Bayesian Experimental Design Many practical applications –Car crash test, medicine design, software testing A good design should maximize the expected utility under the optimal inference algorithm Different utility functions yield different design criteria –We use Bayesian A-optimal and Bayesian D-optimal design criteria

9. 9 Bayesian A-Optimal Design Goal –Minimize the squared error –Maximize the following expected utility Let, where is covariance matrix of x Assuming a normal linear system, the Bayesian procedure yields This is equivalent to minimize

10. 10 Bayesian D-Optimal Design Goal –Maximize the expected gain in Shannon information Let, where is covariance matrix of x Assuming a normal linear system, the Bayesian procedure yields This is equivalent to minimize

11. 11 Search Algorithm Given a design criterion, next step is to find s rows of A to minimize –This problem is NP-hard –We use a sequential search algorithm Start with an empty initial design Sequentially add rows to the design that results in the largest reduction in

12. 12 Flexibility Differentiated design –Give higher weights to the important rows of matrix F Augmented design –Identify the additional paths to probe such that in conjunction with previously monitored paths maximize the utility Multi-user design –New design criteria: a linear combination of different users’ design criteria

13. 13 Network Inference Goal: infer x s.t. y S =A S x Main challenge: under-constrained problem L2-norm minimization L1-norm minimization Maximum entropy estimation

14. 14 Evaluation Methodology Data sets Accuracy metric Only show the RTT results from PlanetLab –Refer to our paper for more extensive results, e.g., loss estimation and comparison of inference algorithms # nodes# overlay nodes # paths# linksRank PlanetLab-RTT25146136575467769 PlanetLab-loss17956032704628690 Brite-n1000-o20010002003980028832051 Brite-n5000-o6005000600359400146989729

15. 15 Comparison of DOE Algorithms: Estimating Network-Wide Mean RTT A-optimal yields the lowest error.

16. 16 Comparison of DOE Algorithms: Estimating Per-Path RTT A-optimal yields the lowest error.

17. 17 Differentiated Design: Inference Error on Preferred Paths Lower error on the paths with higher weights.

18. 18 Differentiated Design: Inference Error on the Remaining Paths Error on the remaining paths increases slightly.

19. 19 Augmented Design A-optimal is most effective in augmenting an existing design.

20. 20 Multi-user Design A-optimal yields the lowest error.

21. 21 Summary Our contributions –Apply Bayesian experimental design to large-scale network performance monitoring –Develop a flexible framework to accommodate different design requirements –Develop a toolkit on PlanetLab to measure and estimate network performance –Our results Higher or comparable accuracy Flexible: differentiated design, augmented design, multi-user design Scalable: can handle 1,000,000 paths and 50,000 links Future work –Making measurement design fault tolerant –Extend our framework to incorporate additional design constraints –Apply our technique to traffic matrix estimation

22. 22 Thank you!