TeraPaths TeraPaths: Flow-Based End-to-End QoS Paths through Modern Hybrid WANs Presented by Presented by Dimitrios Katramatos, BNL Dimitrios Katramatos, BNL

2 Outline  Background: the TeraPaths project  Objective  View of the world (network)  System architecture  Establishing flow-based end-to-end QoS paths  Domain interoperation  Experience and encountered issues  Project status and future work

3 Objective  Provide QoS guarantees at the individual data flow level, all the way to the end hosts, transparently  Data flows have varying priority/importance  Video streams  Critical data  Long duration transfers  Default “best effort” network behavior treats all data flows as equal  Capacity is not unlimited  Congestion causes bandwidth and latency variations  Performance and service disruption problems, unpredictability  Dynamic flow-based SLAs = schedule network utilization  Regulate and classify (prioritize) traffic  Select routing (if possible)

4 View of the Network WAN ctrl WAN 1 WAN 2 WAN 3 TeraPaths Domain ctrl TeraPaths RN TeraPaths WAN ctrl Site ASite BSite CSite D MPLS tunnel Dynamic circuit Domain control

5 TeraPaths TeraPaths Web Services Architecture Domain Controller DSM Web Interface NDC Database protected network API local WAN controllers Domain controllers (non-TeraPaths) WAN service clients (proxies) CLI s/w client Web browser NDC database Domain service clients (proxies) Site controller Site service hardware “virtual network engineer” remote

6 Establishing E2E QoS Paths  Multiple administrative domains  Cooperation, trust, but each maintains full control  Heterogeneous environment  Domain controller coordination through web services  Coordination models  Star  Requires extensive information for all domains  Daisy chain  Requires common flexible protocol across all domains  Hybrid (end-sites first)  Independent protocols  Direct end site negotiation … … …

7 Path Setup WAN WAN web services TeraPaths 1 2 3

8 Path Setup (ii)  End site subnets are configured by TeraPaths software instances (TeraPaths Domain Controllers or TDCs)  TDCs configure end site LANs to prioritize and regulate authorized flows via the DiffServ framework at the network device level  Source site polices/marks authorized flow packets  Destination site admits/re-polices/re-marks packets  End site LANs tx/rx marked packets to/from the WAN  WAN provides MPLS tunnels or dynamic circuits  Initiating TDC requests MPLS tunnel or dynamic circuit with matching bandwidth and lifetime, or…  TDC groups flows with common src/dst into MPLS tunnel or dynamic circuit with aggregate bandwidth and lifetime  WAN preserves packet markings

9 Path Setup (iii)  WAN domains interoperate  Each end site’s TDC has a single point of contact for WAN services  TDCs have no knowledge of WAN internals other than what is exposed by the WAN services  End sites have no direct control over the WAN  Either tunnel or circuit through WAN  Cannot mix and match

10 Interoperating with WAN Services  TeraPaths “proxy” servers  Implement interface required by TeraPaths core  Hide WAN service differences  Clients to WAN web services (currently OSCARS / DRAGON)  Close cooperation with ESnet and I2 development teams  Submit reservations for MPLS tunnels or dynamic circuits  Handle security requirements  Handle errors  MPLS tunnels vs. dynamic circuits  Utilization requires different approach

11 L2 vs. L3 (i)  MPLS tunnel starts and ends within WAN domain  Packets are admitted into the tunnel based on flow ID information (IP src, port src, IP dst, port dst )  WAN admission performed at the first router of the tunnel (ingress) WAN border router MPLS tunnel ingress/egress router MPLS tunnel ingress/egress router

12 L2 vs. L3 (ii)  Dynamic circuit appears as VLAN connecting end site border routers with single hop  Cannot use flow ID data directly  Flow must be directed to the proper VLAN  WAN admission performed within end site LAN  Select VLAN with Policy Based Routing (PBR) at both ends  Route can be selected on a per-flow basis WAN switch border router

13 Site LAN Setup (DiffServ, PBR) PBR

14 3 rd Party Network Segments  Some network segments may not be automatically configurable  Regional providers  Campus segments  Border routers  Static (once only) configuration required  Allow DSCP bits to go through  Only allow specific interfaces  ACLs and aggregate policers  Configure VLANs to be used for dynamic circuits  Trunked VLAN pass-thru  Virtual border router

15 Alternative Site LAN Setup (DSCP, VLAN pass-thru) PBR VLAN pass-thru

16 VLAN Setup for L2 TeraPaths-controlled “virtual border” router (directs flows w/PBR) e.g.,1 to X, 2 to Y WAN Site’s Border Router trunked VLAN pass-thru 50 VLAN ids ( ) 3550  X  Y  3599 interfaces trust DSCP TeraPaths-controlled host router #X #Y DSCP-friendly LAN host 1host nhost to X 2 to Y can be the same device Regional Provider’s Router

17 L2-Specific Issues  Limitations with VLANs  Tag range (tentatively selected 50 VLANs – 3550 to 3599)  Each site may have its own range  Tag conflicts  Rely on WAN service  Eliminate by synchronizing site databases  VLAN renaming (if/when possible)  Scalability issues  Flow grouping  Forward flows through same virtual WAN circuit  Create circuit with new parameters / switch current flows / cancel old circuit  Modify WAN reservations (if/when possible)  PBR overhead  Virtual border router  Sensitive/3 rd party network segments  VLAN pass-thru

18 Status  Currently: basic software ready, infrastructure tested  API and web interface, simple negotiation  Multiple service classes per site with statically allocated bandwidth  Utilization of L3 paths (MPLS tunnels) through ESnet (since 2006)  Utilization of L2 paths (dynamic circuits) through ESnet and Internet2 (demonstrated at SC’07)  “Circle of trust” security model, X.509 certificates  Simple user AAA  BNL, UMich, BU, SLAC  Multiple successful pass-thru configurations (BNL, UMich, NoX, Merit, MiLR)

19 TeraPaths Testbed during SC’07 US ATLAS T2 sites BNL OU UC/IUUMichBU SLAC ESnet UTA I2 NLR NoX StarLight UltraLight MiLR/Merit L2 (dynamic circuit) L3 (MPLS tunnel) L2 and L3

20 Weather Map

21 Traffic Regulation (demo) 1 2 2

22 In Progress / Future  Testbed  Expansion to more US ATLAS Tier 2 sites and beyond  BNL testbed router upgrade to 10Gbps  Support for different hardware  Dynamic bandwidth allocation within service classes  Flow grouping through WAN circuits  CLI, extended API, configurable negotiation  Grid-style AAA (GUMS/VOMS)  Plug-ins: SRM (dCache), others  Expand collaboration/interoperation