1 Traffic Engineering for ISP Networks Jennifer Rexford IP Network Management and Performance AT&T Labs - Research; Florham Park, NJ

2 Outline  Internet routing protocols  Traffic engineering using traditional protocols –Optimizing network configuration to prevailing traffic –Requirements for topology, routing, and traffic info  Traffic demands –Volume of load between edges of the network –Technique for measuring the traffic demands  Route optimization –Tuning the link weights to the offered traffic –Incorporating various operational constraints  Other ways of measuring the traffic matrix  Conclusions

3 Autonomous Systems (ASes)  Internet divided into ASes –Distinct regions of administrative control (~14,000) –Routers and links managed by a single institution –Service provider, company, university, …  Hierarchy of ASes –Large, tier-1 provider with a nationwide backbone –Medium-sized regional provider with smaller backbone –Small network run by a single company or university  Interaction between ASes –Internal topology is not shared between ASes –… but, neighbors interact to coordinate routing

4 Path Traversing Multiple ASes Client Web server Path: 6, 5, 4, 3, 2, 1

5 Interdomain Routing: Border Gateway Protocol  ASes exchange info about who they can reach –IP prefix: block of destination IP addresses –AS path: sequence of ASes along the path  Policies configured by the AS’s network operator –Path selection: which of the paths to use? –Path export: which neighbors to tell? Client ( ) “I can reach /24” “I can reach /24 via AS 1”

6 Intradomain Routing: OSPF or IS-IS  Shortest path routing based on link weights –Routers flood the link-state information to each other –Routers compute the “next hop” to reach other routers  Weights configured by the AS’s network operator –Simple heuristics: link capacity or physical distance –Traffic engineering: tuning the link weights to the traffic

7 Motivating Problem: Congested Link  Detecting that a link is congested –Utilization statistics reported every five minutes –Sample probe traffic suffers degraded performance –Customers complain (via the telephone network?)  Reasons why the link might be congested –Increase in demand between some set of src-dest pairs –Failed router/link within the AS causes routing change –Failure/reconfiguration in another AS changes routes  Challenges –Know the cause, not just the manifestations –Predict the effects of possible changes to link weights

8 Traffic Engineering in an ISP Backbone  Topology of the ISP backbone –Connectivity and capacity of routers and links  Traffic demands –Offered load between points in the network  Routing configuration –Link weights for selecting paths  Performance objective –Balanced load, low latency, …  Question: Given the topology and traffic demands in an IP network, what link weights should be used?

9 Modeling Traffic Demands  Volume of traffic V(s,d,t) –From a particular source s –To a particular destination d –Over a particular time period t  Time period –Performance debugging -- minutes or tens of minutes –Time-of-day traffic engineering -- hours or days –Network design -- days to weeks  Sources and destinations –Individual hosts -- interesting, but huge! –Individual prefixes -- still big; not seen by any one AS! –Individual edge links in an ISP backbone -- hmmm….

10 Traffic Matrix in out Traffic matrix: V(in,out,t) for all pairs (in,out)

11 Problem: Hot Potato Routing  ISP is in the middle of the Internet –Multiple connections to multiple other ASes –Egress point depends on intradomain routing  Problem with point-to-point models –Want to predict impact of changing intradomain routing –But, a change in weights may change the egress point!

12 Demand Matrix: Motivating Example Big Internet Web Site User Site

13 Coupling of Inter and Intradomain Routing Web Site User Site AS 1 AS 3 AS 4 U AS 3, U AS 4, AS 3, U AS 2

14 Intradomain Routing: Hot Potato Zoom in on AS IN OUT Hot-potato routing: change in internal routing (link weights) configuration changes flow exit point! 50 OUT 3 OUT 1

15 Traffic Demand: Multiple Egress Points  Definition: V(in, {out}, t) –Entry link (in) –Set of possible egress links ({out}) –Time period (t) –Volume of traffic (V(in,{out},t))  Computing the traffic demands –Measure the traffic where it enters the ISP backbone –Identify the set of egress links where traffic could leave –Sum over all traffic with same in, {out}, and t

16 Traffic Mapping: Ingress Measurement  Packet measurement (e.g., Netflow, sampling) –Ingress point i –Destination prefix d –Traffic volume V id i d ingress destination

17 Traffic Mapping: Egress Point(s)  Routing data (e.g., forwarding tables) –Destination prefix d –Set of egress points e d d destination

18 Traffic Mapping: Combining the Data  Combining multiple types of data –Traffic: V id (ingress i, destination prefix d) –Routing: e d (set e d of egress links toward d) –Combining: sum over V id with same e d i ingress egress set

19 Application on the AT&T Backbone  Measurement data –Netflow data (ingress traffic) –Forwarding tables (sets of egress points) –Configuration files (topology and link weights)  Effectiveness –Ingress traffic could be matched with egress sets –Simulated flow of traffic consistent with link loads  Challenges –Loss of Netflow records during delivery (can correct for it!) –Egress set changes between table dumps (not very many) –Topology changes between configuration dumps (just one!)

20 Traffic Analysis Results  Small number of demands contribute most traffic –Optimize routes for just the heavy hitters –Measure a small fraction of the traffic –Must watch out for changes in load and set of exit links  Time-of-day fluctuations –Reoptimize routes a few times a day (three?)  Traffic (in)stability –Select routes that are good for different demand sets –Reoptimize routes after sudden changes in load

21 Three Traffic Demands in San Francisco

22 Underpinnings of the Optimization  Route prediction engine (“what-if” tool) –Model the influence of link weights on traffic flow »Select a closest exit point based on link weights »Compute shortest path(s) based on link weights »Capture splitting over multiple shortest paths –Sum the traffic volume traversing each link  Objective function –Rate the “goodness” of a setting of the link weights –E.g., “max link utilization” or “sum of exp(utilization)”

23 Weight Optimization  Local search –Generate a candidate setting of the weights –Predict the resulting load on the network links –Compute the value of the objective function –Repeat, and select solution with lowest objective function  Efficient computation –Explore the “neighborhood” around good solutions –Exploit efficient incremental graph algorithms  Performance results on AT&T’s network –Much better than simple heuristics (link capacity, distance) –Quite competitive with multi-commodity flow solution

24 Incorporating Operational Realities  Minimize changes to the network –Changing just one or two link weights is often enough  Tolerate failure of network equipment –Weights settings usually remain good after failure –… or can be fixed by changing one or two weights  Limit the number of distinct weight values –Small number of integer values is sufficient  Limit dependence on accuracy of traffic demands –Good weights remain good after introducing random noise  Limit frequency of changes to the weights –Joint optimization for day and night traffic matrices

25 Ways to Populate the Domain-Wide Models  Mapping: assumptions about routing –Traffic data: packet/flow statistics at network edge –Routing data: egress point(s) per destination prefix  Inference: assumptions about traffic and routing –Traffic data: byte counts per link (over time) –Routing data: path(s) between each pair of nodes  Direct observation: no assumptions –Traffic data: packet samples at every link –Routing data: none

26 4Mbps 3Mbps5Mbps Inference: Network Tomography Sources Destinations From link counts to the traffic matrix

27 Tomography: Formalizing the Problem  Source-destination pairs –p is a source-destination pair of nodes –x p is the (unknown) traffic volume for this pair  Routing –R lp = 1 if link l is on the path for src-dest pair p –Or, R lp is the proportion of p’s traffic that traverses l  Links in the network –l is a unidirectional edge –y l is the observed traffic volume on this link  Relationship: y = Rx (now work back to get x)

28 Tomography: Single Observation is Insufficient  Linear system is underdetermined –Number of nodes n –Number of links e is around O(n) –Number of src-dest pairs c is O(n 2 ) –Dimension of solution sub-space at least c - e  Multiple observations are needed –k independent observations (over time) –Stochastic model with src-dest counts Poisson & i.i.d –Maximum likelihood estimation to infer traffic matrix –Vardi, “Network Tomography,” JASA, March 1996

29 Tomography: Limitations  Cannot handle packet loss or multicast traffic –Assumes traffic flows from an entry to an egress  Statistical assumptions don’t match IP traffic\ –Poisson, Gaussian  Significant error even with large # of samples –Faulty assumptions and traffic changes over time  High computation overhead for large networks –Large matrix inversion problem

30 Promising Extension: Gravity Models [Zhang, Roughan, Duffield, Greenberg]  Gravitational assumption –Ingress point a has traffic v i a –Egress point b has traffic v e b –Pair (a,b) has traffic proportional to v i a * v e b  Incorporating hot-potato routing –Combine traffic across egress points to the same peer –Gravity divides a’s traffic proportional to peer loads –“Hot potato” identifies single egress point for a’s traffic  Experimental results on AT&T network –Reasonable accuracy, especially for large (a,b) pairs –Sufficient accuracy for traffic engineering applications

31 Conclusions  Our approach –Measure: network-wide view of traffic and routing –Model: data representations and “what-if” tools –Control: intelligent changes to operational network  Application in AT&T’s network –Capacity planning –Customer acquisition –Preparing for maintenance activities –Comparing different routing protocols  Ongoing work: interdomain traffic engineering

32 To Learn More…  Overview papers –“Traffic engineering for IP networks” ( –“Traffic engineering with traditional IP routing protocols” (  Traffic measurement –"Measurement and analysis of IP network usage and behavior” ( –“Deriving traffic demands for operational IP networks” (  Topology and configuration –“IP network configuration for intradomain traffic engineering” (  Intradomain route optimization –“Internet traffic engineering by optimizing OSPF weights” ( –“Optimizing OSPF/IS-IS weights in a changing world” (