1. 1 Resource Optimization in Hybrid Core Networks with 100G Links Malathi Veeraraghavan University of Virginia Date: Feb. 4, 2010 (Collaborator: Admela Jukan) Outline –What is a hybrid network? –Scheduled Dynamic Circuit Service (SDCS) –Single or separate hybrid networks? –SDCS usage –ASCR project software testing on DOE ANI network testbed –Summary Sponsored by DOE ASCR grant DE-SC0002350

2. What is a hybrid network? A network with –packet switches and circuit switches? –Connection-oriented (CO) and connectionless (CL) modes of operation? –All-optical switches and electronic switches? –“Optical and IP”? [dissimilar items] 2

3. A classification of switches/gateways A circuit switch is necessarily connection-oriented as positions have to be allocated to a communication session before data transfer can start 3 Line cards (multiplexing) Controller (admission control or not) Circuit mode (position-based mux: time, frequency, port ) Packet mode (header-based mux) Connectionless (CL) (no admission control) e.g., IP routers, Ethernet switches Connection-oriented (CO) (admission control) e.g., SONET, WDM switches Virtual-circuit switches (VLAN, MPLS)

4. ESnet: IP-routed network (2006 network) 4 Is this a hybrid network? –Leased lines between routers pass through SONET circuit switches –Thus, packet and circuit switches –Leased-lines: CO mode –CO and CL modes –Some switches could be all-optical (metro rings) –Thus all-optical and electronic Not quite! From ESnet document dated Oct. 08, 2006

5. Our understanding of “hybrid network” Supports two types of services: –IP-routed service –Dynamic Circuit Service To do so, what is required of the network? –Support for connection-oriented (CO) and connectionless modes –CO mode should support dynamic bandwidth sharing with control-plane software such as OSCARS schedulers, not just leased-line service –Circuit or virtual-circuit switches to support the dynamic circuit service –IP routers –As link speeds and switching capacities increase, all- optical switches may be required 5

6. Scheduled Dynamic Circuit Service (SDCS) Instead of advance-reservation dynamic circuit service, we propose the name SDCS because –circuit requests can be for immediate usage –so cannot limit it to “advance-reservation” or book-ahead –scheduler (e.g., OSCARS) returns a start time for the circuit hence “Scheduled Dynamic Circuit Service” Two types of SDCS requests: –User-specified start times (USST): co-scheduling apps with instrument or HPC or for a teleconference –Earliest start time (EST): immediate usage requested, but scheduler returns an assigned start time based on resource availability: suitable for large file transfers 6

7. How does SDCS fit in with other connectivity services? 7 RDP: Rate Duration Product RDP = 1500B RDP = 1.4MB (DS0*3 min) RDP = 5.9TB (T1*1 year) Gap to fill

8. How does SDCS differ from leased line service? 8 Leased line Circuit/VC switch Customer device e.g., IP router Customer device e.g., cluster computer SDCS access link Dynamic circuit Customer device e.g., LCD display

9. Differences ServiceLeased line serviceScheduled dynamic circuit service Components of contract One contract: duration rate specific endpoints Two contracts: SDCS access link of certain rate (like IP access link) as needed, requests for circuits of certain duration, rate, between any two SDCS endpoints Duration limit? NoMaximum limit required for acceptable blocking rate/start-time delay with moderate utilization Call blocking?NoYes; to achieve moderate utilization, some call blocking should be tolerated 9 Difficult to articulate differences between a “leased line” and a “dynamic circuit” BUT difference between leased-line service and dynamic-circuit service is clear

10. 10 Outline check Outline –What is a hybrid network? –Scheduled Dynamic Circuit Service (SDCS)  Single or separate hybrid networks? –SDCS usage –ASCR project software testing on DOE ANI network testbed –Summary

11. Single or separate networks? The term “hybrid networking” is more general than “hybrid network” –Hybrid networking: Deploy two separate networks to support the two services IP-routed service Scheduled dynamic circuit service (SDCS) –Hybrid network: seems to imply a single network 11

12. ESnet4 (from 2006 talk by Johnston & Burrescia) Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits), 500-600 Gbps by 2011-2012 (100 Gb/s circuits) Cleveland Europe (GEANT) Asia-Pacific New York Chicago Washington DC Atlanta CERN (30+ Gbps) Seattle Albuquerque Australia San Diego LA Denver South America (AMPATH) South America (AMPATH) Canada (CANARIE) CERN (30+ Gbps) Canada (CANARIE) Europe (GEANT) Asia- Pacific Asia Pacific GLORIAD (Russia and China) Boise Houston Jacksonville Tulsa Boston Science Data Network Core IP Core Kansas City Australia Core network fiber path is ~ 14,000 miles / 24,000 km 1625 miles / 2545 km 2700 miles / 4300 km Sunnyvale Production IP core (10Gbps) SDN core (20-30-40-50 Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections IP core hubs Primary DOE Labs SDN hubs High speed cross-connects with Ineternet2/Abilene Possible hubs USLHCNet

13. “Single” hybrid network What does a “single” network mean: –One node at each PoP –One set of inter-PoP links –One set of customer (e.g., site) access links Today’s deployment: –One node at each PoP for ESnet (IP-routed service) and one node for SDN (dynamic circuit service) –Separate inter-PoP links –Separate site access links –Reasons: Costs of MX vs. M-series interfaces, funding, etc. Expenditures saved: –Inter-PoP and site access link costs –Maintenance costs for nodes What are the negatives? 13

14. What is required for a “single” hybrid network deployment? Node equipment should support –IP routing capability –Circuit/VC capability (e.g. VLAN, MPLS, GMPLS) To support IP-routed service and SDCS on a single interface –Circuit/VC based rate enforcement –e.g., VLAN rate policing/limiting OSCARS IDC for circuit scheduling 14

15. Single node supports both IP-forwarding and virtual-circuit capabilities 15 IP (Layer 3) Ethernet/VLAN (L2) Access links to sites and peers Inter-PoP ESnet links Leased-line for IP-routed service Dynamic circuit Existing equipment already available –Juniper MX series switches –Cisco 6500 series –Carrier-grade Ethernet switches Separate SDN was proof-of-concept for dynamic circuit service? –Since same type of node (MX) used in both IP and SDN networks, both services can be offered on one substrate

16. Part of the ESnet topology (potential single network offering both services) 16 ALBU DENV PNWG SUNN ELPA 100 GbE PNNL IP Ethernet/VLAN IP Ethernet/VLAN IP Ethernet/VLAN IP Ethernet/VLAN PE router Leased-line for IP-routed service Dynamic circuit OSCARS IDC Spectrum NMS LANL IP Ethernet/VLAN

17. 17 Outline check Outline –What is a hybrid network? –Scheduled Dynamic Circuit Service (SDCS) –Single or separate hybrid networks?  SDCS usage –ASCR project software testing on DOE ANI network testbed –Summary

18. SDCS deployment Successful deployment in the core –important: inter-domain demonstrations –[Scheduled Circuit Routing Protocol: like BGP for SDCS] Status on deployment in: –site networks? With site deployments, development of new applications (remote instrument control, WAFS) can be encouraged Without data source-to-data sink (end-to-end) reach of SDCS: –usage scenarios for core SDCS service 18

19. Core network SDCS usage scenarios Usage I: Lambdastation –Applications on user hosts (dCache/SRM) communicate with Lamba Station Server, which runs at the “edge” of sites –Lambda Station Server communicates with core network IDC to reserve/provision dynamic circuit, and sets PBR in CE router to forward packets corresponding to that particular application flow to the circuit Usage II: Automatic long flow detection at PE routers and rerouting to dynamic circuits Usage III: Use SDCS instead of leased-line service for creating IP-routed topology, and resize/reroute these router-to-route circuits as aggregate IP loads on these circuits change –Spectrum monitors SNMP data and initiates changes by requesting these of the IDC 19

20. Two issues with Usage III Current ESnet weathermap shows light loading (5%-20%) on links between IP routers –If SNMP loads indicate a change from 5% to 25%, should the Spectrum NMS react and increase the rate of the circuit between the two routers? Commercial providers note: –Operations divisions typically have strong resistance to change the network topology because of the potential for: route flaps, and drastic changes in the end-end packet latency (e.g., > 10ms) –For these reasons, while theoretically management-plane traffic and network engineering is a potential user of core SDCS, we expect its use to be limited mainly to failure recovery. 20

21. Our ASCR project Focus on Usage II Long-flow identification methods: –5-tuple rule based flow classification –payload based classification –machine learning algorithms 21

22. Software implementation and testing on DOE ANI network testbed MFDB: Monitored Flow Data Base Flows in MFDB mirrored by router to HYNES server When one of these flows is detected, circuit setup is initiated and PBR set in router to route flow to circuit 22

23. Summary Proposed name “Scheduled Dynamic Circuit Service (SDCS)” to cover both –advance-reservation –immediate-request Hybrid networks support IP-routed service and SDCS SDCS: comparison with leased-line service Single hybrid network deployment Three usage scenarios for SDCS HYNES software testing on DOE ANI network testbed 23