1. 1 Can coarse circuit switching work & What to do when it doesn't? Jerry Chou Advisor: Bill Lin University of California, San Diego CNS Review, Jan. 14, 2009

2. 2 Outline Motivation Overview of new optical networking paradigm How to provision optical circuits? What to do when provision circuits not enough? Conclusions

3. 3 Internet Traffic Ever Increasing

4. 4 Current Packet Routing Scenario Packets electronically routed hop-by-hop –IP routers interconnected over switched optical backbone –OEO conversion and queuing delays at each hop OXC 

5. 5 Optical Circuit Switching If optical circuit switching would work, then no intermediate per-hop queuing delays and OEO conversions = much faster OXC 

6. 6 Optical Switching Options Extremely difficult to implement packet buffers and logic in optics No viable dynamically reconfigurable active optical switches at this time scale Packet Switching 10 ns

7. 7 Optical Switching Options New signaling protocol and electronic control plane required to implement dynamic reservations Although active optical switches available at this time scale, coordination of such frequent network-wide reconfigurations not easy Packet Switching 10 ns Optical Burst Switching 1 ms

8. 8 Optical Switching Options Can we reasonably predict the traffic so that we can provision optical circuits to carry them? Can we provide a “fall-back” mechanism when circuit capacity is enough? Packet Switching 10 ns Optical Burst Switching 1 ms Quasi-Static Optical Circuits 1 hr Over 3 Million X

9. 9 Outline Motivation Overview of new optical networking paradigm How to provision optical circuits? What to do when provision circuits not enough? Conclusions

10. 10 Observation Aggregate traffic at the core is relatively smooth and variations are predictable Source: Roughan’03 on a Tier-1 US Backbone

11. 11 Case Study On high-performance public backbone networks –Abilene (US):11 nodes, 23 links –GEANT (Europe): 23 nodes, 74 links –Public traffic matrices are available Optical circuits only change on hourly basis Use historical traffic to “predict” how much traffic will occur in the future –Abilene: 03/01/04-04/21/04, GEANT: 01/01/05–04/10/05 Provision circuits to maximize likelihood that circuits have enough capacity Simulated actual traffic (over a week) –Abilene: 04/22/04-04/26/04, GEANT: 04/11/05–04/15/05

12. 12 Circuits Setup circuits possibly across multiple paths in physical layer Seattle Sunnyvale Indianapolis Denver Los Angeles Kansas City Chicago New York Washington Atlanta Houston

13. 13 Circuits Logically one (optical) circuit for each OD-pair (origin-destination pair) Seattle New York

14. 14 Abilene Network Drop rates is the percentage of offering traffic exceeding its circuit capacity To consider a highly utilized network, traffic is scaled, such that at least one link is saturated under OSPF Worst-case 6.41%, 0.33% on average, mostly at or near 0% Circuit switching works “most of the time” if carefully provisioned

15. 15 New Paradigm Provision optical circuits that maximize the probability of sufficient capacity to carry traffic Use optical circuit switching by default When actual traffic exceeds circuit capacities, route (electronically) over other “pre-configured circuits” with spare capacity OXC Optical transit traffic Traffic arriving to intermediate node Smaller (simpler) routers

16. 16 Analogy Direct “non-stop” flights (optical circuits) by default If overbooked, re-route (electronically) excess demand through alternative multi-hop flights Seattle NY Houston To:NYTo:HSTo:NY

17. 17 Abilene Network No packet drops with re-routing (adaptive load- balancing method to be discussed)

18. 18 Advantages of New Paradigm Minimize queuing delay and latency for packets Reduce workload on electronic routers Optical circuits change infrequently, and mechanisms exist to provision circuits Key idea is to re-route electronically excess traffic rather than “on-the-fly” dynamic optical circuit reconfigurations Avoid new signaling protocol and frequent coordination of network-wide reconfigurations

19. 19 Outline Motivation Overview of new optical networking paradigm How to provision optical circuits? What to do when provision circuits not enough? Conclusions

20. 20 Basic Idea Use historical traffic data sets to decide on bandwidth allocation –Major ISPs have data collection infrastructure already

21. 21 Ideally, Traffic is Stable Abilene –11 nodes connected by 10Gb/s links Seattle Sunnyvale Indianapolis Denver Los Angeles Kansas City Chicago New York Washington Atlanta Houston Seattle/NY: Always 5Gb/s Allocate: 5Gb/s Sunnyvale/Houston: Always 5Gb/s Allocate: 5Gb/s Both flows can be carried by provisioned circuits

22. 22 But, Flows Fluctuate Differently Abilene –11 nodes connected by 10Gb/s links Seattle Sunnyvale Indianapolis Denver Los Angeles Kansas City Chicago New York Washington Atlanta Houston Seattle/NY: High traffic mean Low traffic variance Sunnyvale/Houston: Low traffic mean High traffic variance Give more bandwidth to flows with “high mean” or “high variance”?

23. 23 Circuit Provisioning Approach Use Cumulative Distribution Function (CDF) as “utility function” (predictor of “acceptance probability”) Acceptance probability –The probability of a provisioned circuit with enough capacity to carry its offering traffic

24. 24 Example Abilene –11 nodes connected by 10Gb/s links Seattle Sunnyvale Indianapolis Denver Los Angeles Kansas City Chicago New York Washington Atlanta Houston Seattle/NY: 90% time ≤ 6Gb/s 50% time ≤ 4Gb/s Allocate: 6Gb/s Sunnyvale/Houston: 90% time ≤ 6Gb/s 80% time ≤ 4Gb/s Allocate: 4Gb/s Seattle/NY has 90% acceptance probability Sunnyvale/Houston has 80% acceptance probability

25. 25 Circuit Provisioning Approach Formulate bandwidth allocation (circuit provisioning) as multi-path utility max-min fair allocation problem –Utility functions represent traffic statistics (generally utility functions can be non-linear) –Max-min fairness reach balance between throughput and fairness –Multi-path circuits provide more freedom and better performance We provide the first solution to the multi-path utility max-min fair allocation

26. 26 Multi-path Utility Max-min Algorithm Allocation based on “water-filling algorithm” and maximum concurrent flow Steps: 1.Identify maximum common utility increment 2.Solve maximum concurrent flow problem to find multi- path routing 3.Identify saturated flow Max utility Fill-up by with a routing Saturated flow

27. 27 Multi-Path vs. Single-Path Significantly lower drop probability –Mean drop rate: 3.56% vs. 20.34% –Max drop rate: 18.25 vs. 34.72%

28. 28 Outline Motivation Overview of new optical networking paradigm How to provision optical circuits? What to do when provision circuits not enough? Conclusions

29. 29 r(C) = 20 Localized approach: –load-balance on outbound circuits, weighted by spare capacity r(B) = 30 r(D) = 25 B C D A 1. r(B) < B[A, B] ? YES NO 2. k = random (w k ) Optical Circuit 35 Problem1: greedy solution based only one-hop info. Problem2: oscillation of weight changes can happen Problem1: greedy solution based only one-hop info. Problem2: oscillation of weight changes can happen Adaptive Load-Balanced Routing

30. 30 Adaptive Load-balance Re-routing Distributed approach: Step1: Compute path cost by Distance-Vector-like protocol Step2: Update weights to reach Wardrop Equilibrium state –Every interval only shift weight by a small fraction δ –Achieve fast converge and prevent oscillation –Based on selfish routing no coordination among nodes s t 1 1 4 32 5 1 1 21 Current weights: w 1, w 2 δ = f (C 1, C 1, w 1, w 2 ) w 1 = w 1 + δ, w 2 = w 2 - δ path1 cost(C 1 ): (1+4)=5 path2 cost(C 2 ): (1+8)=9

31. 31 Abilene Network 90 percentile drop rate comparison –OSPF has 0% drop at scale factor of 1

32. 32 Abilene Network 90 percentile drop rate comparison –Cisco’s “ecmp” load-balances across equal cost shortest paths and achieve lower drop rate

33. 33 Abilene Network 90 percentile drop rate comparison –Without rerouting, we suffer small drop rates even at the scale factor of 1 –But show lower drop rates at larger scale factors b.c of greater path diversity and better load-balance

34. 34 Abilene Network 90 percentile drop rate comparison –Achieve lowest drop rates among all –With rerouting, we don’t have drop until at a factor of 1.75.

35. 35 Abilene Network Circuit provisioning achieve lower drop rates under high traffic load b.c of load-balanced routing path Rerouting effectively reduce drop rates under low traffic load by utilizing residual network capacity

36. 36 Outline Motivation Overview of new optical networking paradigm How to provision optical circuits? What to do when provision circuits not enough? Conclusions

37. 37 Conclusion A new paradigm of optical circuit switching by default, packet routing when necessary Formulate circuit provisioning as an utility max-min fair allocation problem and provide the first solution under multiple paths scenario Apply a adaptive load-balance protocol on re-routing Conduct empirical study on two backbone networks, Abilene and GEANT Show more than 95% of traffic can be carried by the network with carefully static circuit provisioning & all traffic can be routed after re-routing

38. 38 Publication Jerry Chou, Bill Lin, "Coarse Optical Circuit Switching by Default, Rerouting over Circuits for Adaptation,“ Journal of Optical Networking, vol. 8, no. 1, pp. 33-50 (2009).

39. 39 Thank You

40. 40 Backup Slides

41. 41 Work-In-Progress Capacity planning Fault-tolerance Better adaptive routing algorithms Joint circuit-provisioning and routability optimization

42. 42 Motivation Traffic growing nearly twice rate of Moore’s Law –Difficult for electronic packet routers to keep up On the other hand, optical switching provides abundance of transmission capacity (e.g. WDM) –Rate of increase in optical transport capacity keeping pace with traffic growth (with 100 Gbps per wavelength in next generation), well above Moore’s Law –Rate of decrease in cost per unit of optical transport capacity well below Moore’s Law

43. 43 Networks Traffic used for prediction (over months) –Abilene: 03/01/04 - 04/21/04, GEANT: 01/01/05 – 04/10/05 Optical circuits only change on hourly basis (method to be discussed) Simulated actual traffic (over a week) –Abilene: 04/22/04 - 04/26/04, GEANT: 04/11/05 – 04/15/05 To consider a highly utilized network, we scaled traffic by a factor, such that at least one link is saturated under OSPF. –Abilene: 4, GEANT: 2

44. 44 Questions How to decide on circuit provisioning to maximize probability that the circuits provide sufficient capacity to carry traffic? –Formulated as a multi-path utility max-min fair bandwidth allocation problem What to do when circuit capacity is not enough? –Adaptive load-balancing over circuits that have spare capacity

45. 45 Multi-Path Utility Max-Min Algorithm Based on water-filling algorithm and maximum concurrent flow (MCF) solver 1.Determine bandwidth allocation that achieves the maximum common utility for all flows 2.Determine path distribution by MCF routing 3.Identify saturated flows and fix their utility Max utility Fill-up by with a routing Saturated flow

46. 46 Binary Search Find maximum utility by binary search over [0, 1] –Determine flow traffic by utility functions –Find feasible route by querying a MCF solver If <1, decrease utility, otherwise increase utility 20 10304050 40 60 80 100 20 10304050 40 60 80 100 20 103040 50 40 60 80 100 20 10304050 40 60 80 100 BW Utility(%) C = 100 Max utilityTraffic 1(50,50,50,50)0.5. 0.6(10,40,10,40)1 0.5(10,30,10,40)1.25

47. 47 Piece-Wise Linear Search Approximate utility functions as piecewise linear functions Replace binary search by searching through each piecewise linear segment –Query MCF by the inverse of slope as traffic – is proportional to maximum utility Seg I Seg II Seg III Seg IV 20 10304050 40 60 80 100 BW Utility(%) 20 U[1] - U[0] BW[1]-BW[0] 10203040 0 10

48. 48 Identifying Saturated Flows By residual capacity is not enough –Miss-identified saturated flow in earlier iteration would produce smaller bandwidth allocation A B C D E F Let link capacity = 10 Bandwidth requirement: A  E = 5, A  F = 5 If select path A  C  D  F, A  E is saturated If select path A  B  D  F, A  E is not saturated

49. 49 Identifying Saturated Flows A flow is saturated if its utility cannot be increased by any feasible routing To guarantee optimality, flows have to be re-routed

50. 50 Multi-Path vs. Single-Path Significantly higher utility –Minimum utility 92.90% vs. 74.74%

51. 51 Avoiding Cycles Problem: packets may go in circles –Never reach destination –Waste circuit capacity One solution is to limit “time-to-live” (TTL) Alternatively, ensure “loop-free” routing by routing table construction

52. 52 Loop-Free Routing Tables For OD-pair, solve maxflow to derive largest “ayclic” graph on “circuit” Build routing tables using both “source” and “destination” prefixes s t

53. 53 Current Contributions New paradigm of optical circuit switching by default, packet routing when necessary First solution to the multi-path utility max-min fair bandwidth allocation problem Though not presented, utility max-min fair solver has been applied to a Denial-of-Service network security problem

54. 54 r(C) = 15r(C) = 20r(C) = 25 Localized Adaptive Re-routing Basic idea: load-balance on outbound circuits, weighted by spare capacity r(B) = 30 r(D) = 25 B C D A 1. r(B) < B[A, B] ? YES NO 2. k = random (w k ) Optical Circuit 35 r(B) = 35 Problem1: greedy solution based only one-hop info. Problem2: oscillation of weights could occur Problem1: greedy solution based only one-hop info. Problem2: oscillation of weights could occur

55. 55 Distributed Adaptive Re-routing Basic idea: 1.Collect path info. by a Distance-Vector-like protocol 2.Load-balance outgoing weights based on path cost s t 1 1 4 32 4 1 1 21

56. 56 Distributed Adaptive Re-routing Step1: Compute path cost –Every router measure downstream link cost –Exchange info. by a Distance-Vector-like protocol s t cost: 1 1 1 4 32 4 1 1 21

57. 57 Distributed Adaptive Re-routing Step1: Compute path cost –Every router measure downstream link cost –Exchange info. by a Distance-Vector-like protocol s t cost: 1+1=2 1 1 4 32 4 1 1 21 cost: 4+1=5 cost: 3+1=4

58. 58 Distributed Adaptive Re-routing Step1: Compute path cost –Every router measure downstream link cost –Exchange info. by a Distance-Vector-like protocol s t cost: 2+2=4 1 1 4 32 5 1 1 21 If weights are equal Cost: (2+4)*0.5 +(5+5)*0.5 = 8