TeraPaths The TeraPaths Collaboration Presented by Presented by Dimitrios Katramatos, BNL Dimitrios Katramatos, BNL

2 Outline  Background: the TeraPaths project  Objective  View of the network  System architecture  Collaborative effort  The problem of being distributed…  Testing and deployment strategy  Milestones  What is the next step?

3 Objective  Provide QoS guarantees at the individual data flow level, between actual end hosts, transparently  Data flows have varying priority/importance  Video streams  Critical data  Long duration transfers  Default “best effort” network behavior treats all data flows as equal  Capacity is not unlimited  Congestion causes bandwidth and latency variations  Performance and service disruption problems, unpredictability  Dynamic flow-based SLAs = schedule network utilization  Regulate and classify (prioritize) traffic  Select routing (if possible)

4 View of the Network WAN ctrl WAN 1 WAN 2 WAN 3 TeraPaths Domain ctrl TeraPaths RN TeraPaths WAN ctrl Site ASite BSite CSite D MPLS tunnel Dynamic circuit Domain control

5 TeraPaths TeraPaths Web Services Architecture Domain Controller DSM Web Interface NDC Database protected network API local WAN controllers Domain controllers (non-TeraPaths) WAN service clients (proxies) CLI s/w client Web browser NDC database Domain service clients (proxies) Site controller Site service hardware “virtual network engineer” remote

6 Collaborative Effort  TeraPaths is a set of domain controllers tied together by distributed services modules (DSMs)  DSMs interoperate with WAN domain controllers  End site controllers directly configure network devices  This can be a problem…  Modifying router configs on the fly using experimental software requires trust and assurances  Thanks to our collaborators we were able to move from proof-of-concept to prototype running currently at 3 sites: BNL, UMich, and BU - SLAC also participates

7 “Do no harm”  BNL testbed (2 X Cisco 6509) not part of production network, i.e. breaking it is ok  Try software w/o risk to production network  Use only proven software to configure other site routers  Bring hosts under TeraPaths control selectively  Static part of configuration  Reduce adverse effects of bugs  Security  “Circle of trust” for servers running TeraPaths modules  X.509 certificates, accounts, grid membership, etc.  “Virtual network engineer” web service to configure routers  There is always some residual risk

8 Milestones  Proof of concept code demonstrated at SC’05  Automatic invocation of OSCARS for MPLS tunnels in early 2006  Prototype code deployed at BNL and UMich, true end-to-end path setup using MPLS tunnels through ESnet in summer 2006  Started looking into dynamic circuits in early 2007  Modified code utilizes MPLS tunnels and dynamic circuits  Testbed expansion to BU through NoX and 2nd connection to UMich through Merit/MiLR  Demo at SC’07: basic circuit utilization  Demo at JTW’08: traffic regulation within circuits

9 MPLS Tunnels vs. Dynamic Circuits WAN border router MPLS tunnel ingress/egress router MPLS tunnel ingress/egress router WAN switch border router vs.

10 VLAN Setup for L2 TeraPaths-controlled “virtual border” router (directs flows w/PBR) e.g.,1 to X, 2 to Y WAN Site’s Border Router trunked VLAN pass-thru 50 VLAN ids ( ) 3550  X  Y  3599 interfaces trust DSCP TeraPaths-controlled host router #X #Y DSCP-friendly LAN host 1host nhost to X 2 to Y can be the same device Regional Provider’s Router

11 Current TeraPaths Testbed US ATLAS T2 sites BNL OU UC/IUUMichBU SLAC ESnet UTA I2 NLR NoX StarLight UltraLight MiLR/Merit L2 (dynamic circuit) L3 (MPLS tunnel) L2 and L3 JTW08 weathermap

12 Traffic Regulation (demo) 1 2 2

13 What is the next step?  Side effect of utilizing dynamic circuits: path selection  Common ground with Lambda Station  End sites need domain controllers in order to utilize dynamic circuits and MPLS tunnels transparently and effectively  We would like to:  At the very least, interoperate with all types of domain controllers  Maybe develop a common-core domain controller for end sites, customizable to site requirements through plug-ins?  Deploy domain controllers to all sites willing to participate  All sites serviced by ESnet and Internet2 qualify  Setup a “protoduction” [?] testbed encompassing these sites  Eventually, move system into production