Internet Measurement (and some inference & modeling) Shivkumar (“Shiv”) Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu http://www.ecse.rpi.edu/Homepages/shivkuma/ GOOGLE: “Shiv RPI”

Topics Measurement philosophy: why, what, when, where, how? Some measurement projects & results Techniques: passive & active Packet tracing SNMP Probing Inference and Modeling Tomography & Traffic Matrix Estimation for network engineering Traffic modeling Rocketfuel: inferring topologies from outside ISP networks

Why Measurement? We built it, we depend on it, so we must try to understand it … as it works in reality... Measurement gives us the data and basis for this understanding. Modeling, Inference etc to get new understanding & learning from data Complex interactions between protocols not well modeled during their design. Need support for troubleshooting and network management Wide area behavior unpredictable Change is normal

Characteristics of the Internet The Internet is Decentralized (loose confederation of peers) Self-configuring (no global registry of topology) Stateless (limited information in the routers) Connectionless (no fixed connection between hosts) These attributes contribute To the success of Internet To the rapid growth of the Internet … and the difficulty of controlling the Internet! ISP sender receiver

Internet Measurement Challenges Size of the Internet O(100M) hosts, O(1M) routers, O(10K) networks Complexity of the Internet Components, protocols, applications, users Constant change is the norm Web, e-commerce, peer-to-peer, wireless, next? The Internet was not developed with measurement as a fundamental feature Nearly every network operator would like to keep most data on their network private Floyd and Paxson, “Difficulties in Simulating the Internet”, IEEE/ACM Transactions on Networking, 2000.

Themes Measurement has been the basis for critical improvements Without measurement, what do you know? Measurement capability in the Internet is limited The systems not designed to support measurement Measurement tools and infrastructures are few and limited Size, diversity, complexity and change Measurement data presents many challenges Networking researchers need better connections with experts in other domains

Operator Philosophy: Tension With IP Accountability of network resources But, routers don’t maintain state about transfers But, measurement isn’t part of the infrastructure Reliability/predictability of services But, IP doesn’t provide performance guarantees But, equipment is not especially reliable (no “five-9s”) Fine-grain control over the network But, routers don’t do fine-grain resource allocation But, network automatically re-routes after failures End-to-end control over communication But, end hosts and applications adapt to congestion But, traffic may traverse multiple domains of control

Network Operations: Measure, Model, and Control Network-wide “what-if” model Topology/ Configuration Offered traffic Changes to the network measure control Operational network

“Operations” Research: Detect, Diagnose, and Fix Detect: note the symptoms of a problem Periodic polling of link load statistics Active probes measuring performance Customer complaining (via the phone network?) Diagnose: identify the illness Change in user behavior? Router/link failure or policy change? Denial of service attack? Fix: select and dispense the medicine Routing protocol reconfiguration Installation of packet filters Network measurement plays a key role in each step!

Traffic Measurement: Control vs. Discovery Discovery: characterizing the network End-to-end characteristics of delay, throughput, and loss Verification of models of TCP congestion control Workload models capturing the behavior of Web users Understanding self-similarity/multi-fractal traffic Control: managing the network Generating reports for customers and internal groups Diagnosing performance and reliability problems Tuning the configuration of the network to the traffic Planning outlay of equipment (routers, proxies, links)

Measurement Techniques

Time Scales for Network Operations Minutes to hours Denial-of-service attacks Router and link failures Serious congestion Hours to weeks Time-of-day or day-of-week engineering Outlay of new routers and links Addition/deletion of customers or peers Weeks to years Planning of new capacity and topology changes Evaluation of network designs and routing protocols

Traffic Measurement: SNMP Data Simple Network Management Protocol (SNMP) Router CPU utilization, link utilization, link loss, … Collected from every router/link every few minutes Applications Detecting overloaded links and sudden traffic shifts Inferring the domain-wide traffic matrix Advantage Open standard, available for every router and link Disadvantage Coarse granularity, both spatially and temporally

Traffic Measurement: Packet-Level Traces Packet monitoring IP, TCP/UDP, and application-level headers Collected by tapping individual links in the network Applications Fine-grain timing of the packets on the link Fine-grain view of packet header fields Advantages Most detailed view possible at the IP level Disadvantages Expensive to have in more than a few locations Challenging to collect on very high-speed links Extremely high volume of measurement data

Extracting Data from IP Packets TCP TCP TCP Application message (e.g., HTTP response) Many layers of information IP: source/dest IP addresses, protocol (TCP/UDP), … TCP/UDP: src/dest port numbers, seq/ack, flags, … Application: URL, user keystrokes, BGP updates,…

Aggregating Packets into Flows Set of packets that “belong together” Source/destination IP addresses and port numbers Same protocol, ToS bits, … Same input/output interfaces at a router (if known) Packets that are “close” together in time Maximum inter-packet spacing (e.g., 15 sec, 30 sec) Example: flows 2 and 4 are different flows due to time

Summary: Traffic Measurement: Flow-Level Traces Flow monitoring (e.g., Cisco Netflow) Measurements at the level of sets of related packets Single list of shared attributes (addresses, port #s, …) Number of bytes and packets, start and finish times Applications Computing application mix and detecting DoS attacks Measuring the traffic matrix for the network Advantages Medium-grain traffic view, supported on some routers Disadvantages Not uniformly supported across router products Large data volume, and may slow down some routers Memory overhead (size of flow cache) grows with link speed

Summary: Reducing Packet/Flow Measurement Overhead Filtering: select a subset of the traffic E.g., destination prefix for a customer E.g., port number for an application (e.g., 80 for Web) Aggregation: grouping related traffic E.g., packets/flows with same next-hop AS E.g., packets/flows destined to a particular service Sampling: subselecting the traffic Random, deterministic, or hash-based sampling 1-out-of-n or stratified based on packet/flow size Combining filtering, aggregation, and sampling

Summary: Comparison of Techniques Filtering Aggregation Sampling Precision exact exact approximate constrained a-priori constrained a-priori Generality general Local Processing filter criterion for every object table update for every object only sampling decision Local memory one bin per value of interest none none depends on data depends on data Compression controlled

Inference and Modeling…

DATA-DRIVEN…

Eg: The Network Design Problem 200 65 258 134 30 42 Düsseldorf Frankfurt Berlin Hamburg München Communication Demands Düsseldorf Frankfurt Berlin Hamburg München Potential topology & Capacities

Traffic Modeling …

Mandelbrot’s Construction Renewal reward processes and their aggregates Aggregate is made up of many constituents Each constituent is of the on/off type On/off periods have a “duration” Constituents make contributions (“rewards”) when “on” Constituents make no contributions when “off” What can be said about the aggregate? In terms of assumed type of “randomness” for durations and rewards In terms of implied type of “burstiness”

Mandelbrot’s Types of “Randomness” Distribution functions/random variables “Mild” → finite variance (Gaussian) “Wild” → infinite variance Correlation function of stochastic process None => “IID” (independent, identically distributed) “Mild” → short-range dependence (SRD, Markovian) “Wild” → long-range dependence (LRD)

Mandelbrot’s Types of “Burstiness” Bursty BURSTY smooth bursty Mild Wild Distribution function Mild Wild Correlation structure Tail-driven burstiness (“Noah effect”) Dependence-driven burstiness (“Joseph effect”)

Type of Burstiness: “Smooth” CCDF Function 1-F(x) 1-F(x) on log scale x on linear scale Correlation Function r(n) Log-linear scales r(n) on log scale lag n on linear scale

Type of Burstiness: “bursty” CCDF Function 1-F(x) 1-F(x) on log scale x on linear scale Correlation Function r(n) Log-linear scale Log-log scale r(n) on log scale lag n on log scale

Type of Burstiness: “Bursty” CCDF Function 1-F(x) 1-F(x) on log scale x on log scale Correlation Function r(n) Log-log scale Log-linear scale ? r(n) on log scale lag n on linear scale

Type of Burstiness: “BURSTY” CCDF Function 1-F(x) ? 1-F(x) on log scale x on log scale Correlation Function r(n) Log-log scales ? r(n) on log scale lag n on log scale

Mandelbrot’s Types of “Burstiness” Bursty BURSTY smooth bursty Mild Wild Distribution function Mild Wild Correlation structure Tail-driven burstiness (“Noah effect”) Dependence-driven burstiness (“Joseph effect”)

Inference For Network Engineering: Traffic Matrix Estimation…

Network Engineering: Inference Reliability analysis Predicting traffic under planned or unexpected router/link failures Traffic engineering Optimizing OSPF weights to minimize congestion Capacity planning Forecasting future capacity requirements Routes change under failures Network engineering? Many people think the network is just a collection of dumb pipes. So do we really need to engineer a collection of dumb pipes? The answer is of course yes. And it takes quite a bit of intelligence in order to run a large IP network. Here are some common network engineering tasks …

Traffic Matrix Problem

i.e. # Unknowns > # Equations

Naïve Approach Naïve approach: singular value decomposition … In real networks the problem is highly under-constrained

Simple Gravity Model y1  x1 x2 y2  x2 x3 y3  x1 x3 Motivated by Newton’s Law of Gravitation Assume traffic between sites is proportional to traffic at each site y1  x1 x2 y2  x2 x3 y3  x1 x3 Assume there is no systematic difference between traffic in different locations Only the total volume matters Could include a distance term, but locality of information is not so important in the Internet as in other networks An alternative approach is gravity modeling. It is motivated by Newton’s law of gravitation and has been commonly used in fields social engineering …

Simple Gravity Model Better than naïve, but still not very accurate

Generalized Gravity Model Internet routing is asymmetric Hot potato routing: use the closest exit point Generalized gravity model For outbound traffic, assumes proportionality on per-peer basis (as opposed to per-router) peer links access links A major reason that the simple gravity model works not so well is the asymmetric nature of Internet routing …

Generalized Gravity Model Fairly accurate given that no link constraint is used

Tomographic Approach x = AT y Apply the link constraints 1 route 1 router 2 route 3 route 2 3 In gravity modeling, the only information we use is the total load at each node. The tomographic approach instead tries to estimate traffic matrices by applying the link constraints … x = AT y

Tomographic Approach Under-constrained linear inverse problem Find additional constraints based on models Typical approach: use higher order statistics Disadvantages Complex algorithm – doesn’t scale Large networks have 1000+ nodes, 10000+ routes Reliance on higher order statistics is not robust given the problems in SNMP data Artifacts, Missing data Violations of model assumptions (e.g. non-stationarity) Relatively low sampling frequency: 1 sample every 5 min Unevenly spaced sample points Not very accurate at least on simulated TM As I mentioned before, the problem is a highly under-constrained linear inverse problem. The natural solution to this is of course to add some constraints …

Inference: Network Tomography From link counts to the traffic matrix Sources 5Mbps 3Mbps 4Mbps 4Mbps Destinations

Tomography: Formalizing the Problem Source-destination pairs p is a source-destination pair of nodes xp is the (unknown) traffic volume for this pair Routing Rlp = 1 if link l is on the path for src-dest pair p Or, Rlp is the proportion of p’s traffic that traverses l Links in the network l is a unidirectional edge yl is the observed traffic volume on this link Relationship: y = Rx (now work back to get x)

Tomography: Single Observation is Insufficient Linear system is underdetermined Number of nodes n Number of links e is around O(n) Number of src-dest pairs c is O(n2) Dimension of solution sub-space at least c - e Multiple observations are needed k independent observations (over time) Stochastic model with src-dest counts Poisson & i.i.d Maximum likelihood estimation to infer traffic matrix Vardi, “Network Tomography,” JASA, March 1996

Tomography: Challenges Limitations Cannot handle packet loss or multicast traffic Statistical assumptions don’t match IP traffic Significant error even with large # of samples High computation overhead for large networks Directions for future work More realistic assumptions about the IP traffic Partial queries over subgraphs in the network Incorporating additional measurement data

Tomo-gravity “Tomo-gravity” = tomography + gravity modeling Exploit topological equivalence to reduce problem size Use least-squares method to get the solution, which Satisfies the constraints Is closest to the gravity model solution Can use weighted least-squares to make more robust least square solution Now I can explain how we come up with the name tomo-gravity. Tomo-gravity = tomography + gravity modeling. It tries to take the best of both approaches … gravity model solution constraint subspace

Tomo-gravity: Accuracy Accurate within 10-20% (esp. for large elements)

Tomo-gravity Solution Tomo-gravity infers traffic matrices from widely available measurements of link loads Accurate: especially accurate for large elements Robust: copes easily with data glitches, loss Flexible: extends easily to incorporate more detailed measurements, where available Fast: for example, solves AT&T’s IP backbone network in a few seconds In daily use for AT&T IP network engineering Reliability analysis, capacity planning, and traffic engineering

Summary: Tomo-gravity Tomo-gravity takes the best of both tomography and gravity modeling Simple, and quick A few seconds for whole AT&T backbone Satisfies link constraints Gravity model solutions don’t Uses widely available SNMP data Can work within the limitations of SNMP data Only uses first order statistics  interpolation very effective Limited scope for improvement Incorporate additional constraints from other data sources: e.g., Netflow where available Operational experience very positive In daily use for AT&T IP network engineering Successfully prevented service disruption during simultaneous link failures To summarize, tomo-gravity really works! It takes the best of both tomography and gravity modeling … (end with the story on how we successfully prevented service disruption during disastrous simultaneous link failures)