1. Traffic Engineering for ISP Networks
Jennifer Rexford
Computer Science Department
Princeton University

2. Internet Service Provider (ISP) backbones
Outline
Internet Service Provider (ISP) backbones
Wide-area network with multiple Points of Presence
Shortest-path link-state routing between edge routers
Optimization: Tuning routing to the traffic
Optimizing routing given a topology and traffic matrix
Local search to select the integer link weights
Tomography: Inferring the traffic matrix
Estimating traffic matrix from routing and link loads
Conclusion and ongoing work

3. Example Backbone: Abilene Internet2 Newtork

4. Points-of-Presence (PoPs)
Inter-PoP links
Long distances
High bandwidth
Intra-PoP links
Short cables between racks or floors
Aggregated bandwidth
Links to other networks
Wide range of media and bandwidth
Inter-PoP
Intra-PoP
Other networks

5. Routing Inside an Internet Service Provider
Routers flood information to learn the topology
Routers determine “next hop” to reach other routers…
By computing shortest paths based on the link weights
Routers forward packets via the “next hop” link(s) 2 1 3 1 3 2 1 5 4 3

6. Optimization: Tuning Routing to the Traffic

7. Link Weights Control the Flow of Traffic
Routers compute paths
Shortest paths as sum of link weights
Operators set the link weights
To control where the traffic goes 2 1 3 1 3 2 1 3 5 4 3

8. Heuristics for Setting the Link Weights
Proportional to physical distance
Cross-country links have higher weights than local ones
Minimizes end-to-end propagation delay
Inversely proportional to link capacity
Smaller weights for higher-bandwidth links
Attracts more traffic to links with more capacity
Tuned based on the offered traffic
Network-wide optimization of weights based on traffic
Directly minimizes key metrics like max link utilization

9. Why Are the Link Weights Static?
Strawman alternative: load-sensitive routing
Link metrics based on traffic load
Flood dynamic metrics as they change
Adapt automatically to changes in offered load
Reasons why this is typically not done
Delay-based routing unsuccessful in the early days
Oscillation as routers adapt to out-of-date information
Most Internet transfers are very short-lived
Research and standards work continues…
… but operators have to do what they can today

10. Big Picture: Measure, Model, and Control
Network-wide
“what if” model
Topology/
Configuration
Offered
traffic
Changes to
the network
measure
control
Operational network

11. Traffic Engineering in an ISP Backbone
Topology
Connectivity and capacity of routers and links
Traffic matrix
Offered load between points in the network
Link weights
Configurable parameters for Interior Gateway Protocol
Performance objective
Balanced load, low latency, service level agreements …
Question: Given the topology and traffic matrix in an IP network, which link weights should be used?

12. Key Ingredients of Our Approach
Measurement
Topology: monitoring of the routing protocols
Traffic matrix: widely deployed traffic measurement
Network-wide models
Representations of topology and traffic
“What-if” models of shortest-path routing
Network optimization
Efficient algorithms to find good configurations
Operational experience to identify key constraints

13. Formalizing the Optimization Problem
Input: graph G(R,L)
R is the set of routers
L is the set of unidirectional links
cl is the capacity of link l
Input: traffic matrix
Mi,j is traffic load from router i to j
Output: setting of the link weights
wl is weight on unidirectional link l
Pi,j,l is fraction of traffic from i to j traversing link l

14. Multiple Shortest Paths With Even Splitting
0.5
0.25
1.0
Values of Pi,j,l

15. Defining the Objective Function
Computing the link utilization
Link load: ul = Si,j Mi,j Pi,j,l
Utilization: ul/cl
Objective functions
min(maxl(ul/cl))
min(Sl f(ul/cl))
f(x) 1 x

16. Complexity of the Optimization Problem
NP-hard optimization problem
No efficient algorithm to find the link weights
Even for the simple convex objective functions
Why can’t we just do multi-commodity flow?
E.g., solve the multi-commodity flow problem…
… and the link weights pop out as the dual
Because IP routers cannot split arbitrarily over ties
What are the implications?
Have to resort to searching through weight settings

17. Optimization Based on Local Search
Start with an initial setting of the link weights
E.g., same integer weight on every link
E.g., weights inversely proportional to link capacity
E.g., existing weights in the operational network
Compute the objective function
Compute the all-pairs shortest paths to get Pi,j,l
Apply the traffic matrix Mi,j to get link loads ul
Evaluate the objective function from the ul/cl
Generate a new setting of the link weights
repeat

18. Making the Search Efficient
Avoid repeating the same weight setting
Keep track of past values of the weight setting
… or keep a small signature (e.g., a hash) of past values
Do not evaluate a weight setting if signatures match
Avoid computing the shortest paths from scratch
Explore weight settings that changes just one weight
Apply fast incremental shortest-path algorithms
Limit the number of unique values of link weights
Do not explore all 216 possible values for each weight
Stop early, before exploring the whole search space

19. Incorporating Operational Realities
Minimize number of changes to the network
Changing just 1 or 2 link weights is often enough
Tolerate failure of network equipment
Weights settings usually remain good after failure
… or can be fixed by changing one or two weights
Limit dependence on measurement accuracy
Good weights remain good, despite random noise
Limit frequency of changes to the weights
Joint optimization for day and night traffic matrices

20. Application to AT&T’s Backbone Network
Performance of the optimized weights
Search finds a good solution within a few minutes
Much better than link capacity or physical distance
Competitive with multi-commodity flow solution
How AT&T changes the link weights
Maintenance done every night from midnight to 6am
Predict effects of removing link(s) from the network
Reoptimize the link weights to avoid congestion
Configure new weights before disabling equipment

21. Example from My Visit to AT&T’s Operations Center
Amtrak repairing/moving part of the train track
Need to move some of the fiber optic cables
Or, heightened risk of the cables being cut
Amtrak notifies us of the time the work will be done
AT&T engineers model the effects
Determine which IP links go over the affected fiber
Pretend the network no longer has these links
Evaluate the new shortest paths and traffic flow
Identify whether link loads will be too high

22. Same process applied to other cases
Example Continued
If load will be too high
Reoptimize the weights on the remaining links
Schedule the time for the new weights to be configured
Roll back to the old weight setting after Amtrak is done
Same process applied to other cases
Assessing the network’s risk to possible failures
Planning for maintenance of existing equipment
Adapting the link weights to installation of new links
Adapting the link weights in response to traffic shifts

23. Conclusions on Traffic Engineering
IP networks do not adapt on their own
Routers compute shortest paths based on static weights
Service providers need to adapt the weights
Due to failures, congestion, or planned maintenance
Leads to an interesting optimization problems
Optimize link weights based on topology and traffic
Optimization problem is computationally difficult
Forces the use of efficient local-search techniques
Results of the local search are pretty good
Near-optimal solutions that minimize disruptions

24. Robust link-weight assignments
Ongoing Work
Robust link-weight assignments
Link/node failures
Range of traffic matrices
More complex routing models
Hot-potato routing
BGP routing policies
Interaction between ASes
Inter-AS negotiation for joint optimization
Grappling with scalability and trust issues

25. Tomography: Inferring the Traffic Matrix

26. Computing the Traffic Matrix Mi,j
Hard to measure the traffic matrix
IP networks transmit data as individual packets
Routers do not keep traffic statistics, except link utilization on (say) a five-minute time scale
Need to infer the traffic matrix Mi,j from
Current topology G(R,L)
Current routing Pi,j,l
Current link load ul
Link capacity cl

27. Inference: Network Tomography
From link counts to the traffic matrix
Sources
5Mbps
3Mbps
4Mbps
4Mbps
Destinations

28. Tomography: Formalizing the Problem
Ingress-egress pairs
p is a ingress-egress pair of nodes (i,j)
xp is the (unknown) traffic volume for this pair Mi,j
Routing
Plp is proportion of p’s traffic that traverses l
Links in the network
l is a unidirectional edge
ul is the observed traffic volume on this link
Relationship: u = Px (work backwards to get x)

29. Tomography: One Observation Not Enough
Linear system of n nodes is underdetermined
Number of links e is around O(n)
Number of ingress-egress pairs c is O(n2)
Dimension of solution sub-space at least c - e
Multiple observations are needed
k independent observations (over time)
Stochastic model with Poisson iid counts
Maximum likelihood estimation to infer matrix
Doesn’t work all that well in practice…

30. Approach Used at AT&T: Tomo-gravity
Gravitational assumption
Ingress point a has traffic via
Egress point b has traffic veb
Pair (a,b) has traffic proportional to via * veb 9 6 3 20 14 7 21 10

31. Approach Used at AT&T: Tomo-gravity
Problem with gravity model
Gravity model ignores the load on the inside links
Gravity assumption isn’t always 100% correct
Resulting traffic matrix might not satisfy the link loads
Combining the two techniques
Gravity: find a traffic matrix using the gravity model
Tomography: find the family of traffic matrices consistent with all link load statistics
Tomo-gravity: find the tomography solution that is closest to the output of the gravity model
Works extremely well (and fast) in practice

32. Managing IP networks is challenging
Conclusions
Managing IP networks is challenging
Routers don’t adapt on their own to congestion
Routers don’t reveal much information about traffic
Measurement provides a network-wide view
Topology
Traffic matrix
Optimization enables the network to adapt
Inferring the traffic matrix from the link loads
Optimizing the link weights based on the traffic matrix

33. New Research Direction: Design for Manage-ability
Two main parts of network management
Control: optimization
Measurement: tomography
Two research approaches
Bottom up: do the best with what you have
Top down: design systems that are easier to manage
Design for manage-ability
“If you are both the professor and the student, you create exam questions that are easy to answer.” – Mung Chiang