1. LAN QoS and WAN MPLS: Status and Plan Dantong Yu and Shawn Mckee DOE Site Visit December 13, 2004

2. 2 Outline  Introduction: The MPLS & LAN QoS project and personnel Involved.  BNL network configuration.  Program goals and details.  Proposed primitive infrastructure.  Work plans, milestones, schedule.  Current status.

3. 3  This project will investigate the integration and use of LAN QoS and MPLS based differentiated network services in the ATLAS data intensive distributed computing environment as a way to manage the network as a critical resource;.  The Collaboration includes:  Brookhaven National Laboratory (US ATLAS Tier 1, ESNet)  USATLAS Group: Bruce Gibbard, Razvan Popescu, Tom Throwe, Dantong Yu, Xin Zhao  ITD Networking Group: Scott Bradley, Frank Burstein, Vincent Bonafede  Univ. of Michigan [UM] (US ATLAS Network Project Manager, US ATLAS Candidate Tier 2 Center, Internet2, UltraLight Co-PI): Shawn Mckee  The External Collaborators from OSCAR (ESNET), Lambda station (FNAL), and DWMI (SLAC) What is Terapaths ?

4. 4 BNL Campus Network and Immediate Update Plan  The USATLAS/RHIC Computing Facility is attached to access/distribution layer routers.  Campus network consists of Cisco 6509 Series Switches, with Cisco QoS support  Cisco PIX 535 Firewall. (1Gbps)  Will be replaced by firewall service blade for CISCO 6500 series. (5 Gbps), Dec/14/2004

5. 5 Project Goal and Objectives T The primary goal of this project is to investigate the use of QoS/MPLS technology in the ATLAS data intensive distributed computing environment. In addition we intend to: q Develop expertise in QoS technology and Management which will be important to ATLAS and the LHC community more generally. q Dedicate fractions of the available bandwidth via QoS supported by existing network infrastructure to ATLAS Tier 1 data movement and RHIC data replications to assure adequate throughput and limit their disruptive impact upon each other. q Enhance technical contact between the ATLAS tier 1 at BNL and its network partners including the Tier 0 center at CERN, potential ATLAS Tier 2’s and other members of the Grid3+ (OSG-0) community of which it is a part.

6. 6 Proposed Prototype/Primitive Infrastructure GridFtp & SRM LAN/MPLS ESnet Network resource manager MPLS requests Traffic Identification TCP syn/fin packages, addresses, port # Grid AA Network Usage Policy Translator Bandwidth Requests & Releases OSCARS IN(E)GRESS Monitoring Direct Bandwidth Requests SE Second/Third year LAN QoS M10 LAN QoS requests

7. 7 Overall Work Plans  Terapaths envisions a multiple year program to deliver a high- performance, QoS enabled network infrastructure for ATLAS/LHC computing. Each year will determine the following year(s)’s direction.  Phase I: Establish Initial Functionality (08/04 ~ 09/30). z Help to steer the direction of the following two phases.  Phase II:Establish Prototype Production Service (10/05 ~ 07/06).  Depends on the success of Phase 1.  Phase III:Establish Full Production Service, Extend Scope and Increase Functionality (08/06 ~ 07/07). z The level of service and its scope will depends on the available project funding and some additional resources. z Broaden deployment and capability to Tier2s, partners.

8. 8 Goal: Have LAN QoS up and running with manual configuration, integrate with WAN’s MPLS PATH to provide End-to-end QoS (Oct/01~March/31: 6 months)  Milestone 1 and work items: (Jan/10/2005) Enable LAN QoS inside BNL campus network along the path from GridFtp servers and SRM into the boarder routers.  Identify QoS capabilities associated with LAN infrastructure. (ITD)  Study hardware-dependent QoS techniques, options for LAN QoS. (ITD, UM)  Initial deployment on testbed (non-production) routers to prove the concepts. (ITD,UM)  Configure local production network to enable QoS for pre-selected traffics. (ITD)  Milestone 2 and work items: (Jan/31/2005) Test and verify MPLS paths between BNL and LBL, SLAC (network monitoring project), FNAL.  Meet regularly with OSCAR, remote collaborators to request MPLS path setup. (BNL ATLAS, UM, OSCAR)  Send Traffic to MPLS paths provided by OSCAR. (BNL teams) Test and verify MPLS path between BNL and UM (BNL, OSCAR, AND UM). (March/30/2004)

9. 9 6 months goal, milestones and work items  Milestone 3 and work items: (Jan/31/2005) Set up network monitoring tools provided by DWMI, install other monitoring tools when necessary.  Work with Les to attain and deploy the PingER and DWMI tools. (BNL and SLAC)  Install host monitoring tools, such as ganglia and network equipment monitoring tools. (BNL ATLAS and UM)  Milestone 4 and work items: (Jan/31/2005) Integrate the LAN QoS with MPLS paths offered by OSCAR.  Verify LAN QoS capability by injecting regular traffic and prioritized traffic in LAN. (BNL ATLAS and ITD)  Verify WAN MPLS bandwidth provisioning (BNL ATLAS and UM)  Do integration test and document experience. (BNL ATLAS and UM)  Create a simple LAN QoS and MPLS request system based on email exchanges: email list, conference. (BNL ATLAS and OSCAR)

10. 10 6 months goal, milestones and work items  Milestone 5 and work items: (Feb/28/2005) Verify the impact between prioritized traffic and best effort traffic. Learn the effectiveness and efficiency of MPLS/LAN QoS and its impact to overall network performance  Create best effort traffic between BNL and CERN to fill up bottleneck bandwidth for prescribed periods. (BNL ATLAS)  Inject prioritized traffic between BNL and FNAL and monitor the traffic behavior. (BNL ATLAS and ITD)  Test and verify different LAN QoS technologies/policies in different time periods. (BNL ATLAS, ITD and UM)  Milestone 6 and work items: (March/31/2005) Add more applications into MPLS/LAN QoS projects  Add part of USATLAS/RHIC production data transfer into LAN/MPLS paths. (BNL ATLAS, ITD, OSCAR, RHIC)

11. 11 Goal: Automate the MPLS/LAN QoS setup and raise aware of managing end-to-end network resource (April/01~Sep/31/05, 12 months)  Milestone 1: Document any lesson/experience learned from this project. (April/30/2005)  Decide whether LAN QoS/MPLS is useful to LHC physics. (BNL ATLAS, UM)  Build LAN QoS/MPLS expertise in BNL: LAN QoS & MPLS setup, configuration, maintenance and removal. (BNL ATLAS, ITD, UM)  Milestone 2: Implement automatic/manual QoS request system. (May/31/2005)  Traffic Identification: capture TCP syn/fin packages and register to the pending request queue. (BNL ATLAS, UM).  Dynamic generation of QoS requests in the midst of network traffic. (BNL ATLAS, SLAC)  Create Web interface for manually inputting QoS requests. (BNL ATLAS).  Interface into QoS scheduling System. (BNL ATLAS, UM).

12. 12 12 Months goal, milestones and work items  Milestone 3: Implement simple bandwidth scheduling system. (June/30/2005)  Implement first come, first serve for LAN bandwidth. (BNL ATLAS)  Implement interface into OSCAR MPLS scheduler. (BNL ATLAS, ESNet)  Information collector for DWMI. (BNL ATLAS and SLAC)  Milestone 4: Implement automatic LAN QoS configuration systems. (June/30/2005)  Evaluate/Deploy vendor provided LAN management system software (BNL ITD, UM)  Map LAN QoS requests onto network control commands (BNL ITD, UM)

13. 13 12 Months goal, milestones and work items  Milestone 5: System Integration, test, validation and deployment. (August/31/2005)  System integration onto available testbed equipment. (BNL, UM)  Test and Verify with synthetic traffic. (BNL)  Deploy the system into BNL production equipment (BNL, UM).  Milestone 6: Design a remote surveillance video demo to monitor BNL and FNAL SC2005 booths and home institutes and utilize WAN MPLS and LAN QoS. (BNL, FNAL) (Sep/31/2005)  Configure LAN QoS and MPLS paths between BNL and FNAL. (BNL, FNAL, Esnet)  Exchange video and audio streams between these two sites. (BNL, FNAL)  Generate physics event histogram from data stored at remote peers. (BNL, FNAL)  Compare with NON-QoS scenario and document the difference.

14. 14 Establish Prototype Production Service (year 2) T Integrate Grid data transfer (GridFtp) into LAN QoS/MPLS enabled network service. T Effectively couple these network/data transfer services with storage resources managed by SRM, have GridFtp/SRM functionality in “beta”. T Incorporate the resulting system into the ATLAS grid middleware. T Build tools to provide basic authentication, authorization and access control – Depends on funding, Rely on leveraging. T Supply client interfaces which make this service available in a manner transparent to any details of the underlying QoS/MPLS traffic engineering T Leverage MPLS paths/VO level network monitoring services with DWMI project to be developed at SLAC.

15. 15 Extend Scope and Increase Functionality of Prototype Service (year 3) T Inter-network domain MPLS establishment, dynamically creating, adjusting of sub-partitioning MPLS paths to meet time constrained network requirements. q Create site level network resource manager for multiple VOs vying for limited WAN resource. q Provide dynamic bandwidth re-adjusting based resource usage policy and path utilization status collected from network monitoring (DWMI). q Leverage dynamic MPLS establishing services provided by OSCARS: the ESnet On-Demand Secure Circuits and Advance Reservation System. q Create user interface/web services for LHC data transfer applications to request network resource in advance.  Goal: to broaden deployment and capability to tier 1 and tier 2 sites, create services which will be honored/adopted by CERN ATLAS/LHC Tier 0.

16. 16 Status (Testbed and Labeling)  A LAN QoS simulation testbed is being created in a private network environment: consists of four hosts and two CISCO switches (same models as production infrastructure)  Evaluated mechanisms to assign different “cos” to GridFtp data transfers initiated by different VOs.  LAN: source IP addresses and port numbers determine DSCP bits  WAN MPLS: Use IP addresses, port # & DSCP bits for assign MPLS labels at border router. TEnable QoS: “mls qos” TClassify the packets at the receiving ports, 3/1 and 3/9. q Interface fa3/1: Mls qos cos 5 q Interface fa3/9: Mls qos cos 0 TBandwidth Policing: Fair queue tos 5 60 Fair queue tos 0 Class-map match-any green Match ip precedence 0 Class-map match-any red Match ip precedence 5 60% 40% 3/1 3/9

17. 17 Status (Production network configuration)  LAN  packets from ftp servers (aftpexp01~04) and port (21,000~22,000) to FNAL CMS subnet were marked to class 4 and EF at three Cisco switches/routers.  Working on bandwidth allocation/policing for the marked traffic.  WAN  A MPLS path was manually created between the BNL ESNet router to Esnet POP to STARLight. (November/18/2004)  Traffic with correct source, destination ip addresses is correctly labeled and tunneled.  Bandwidth allocation is 50mbps, but policer does not correctly police the traffic.

18. 18 Status (MPLS traffic verification)  Qualified pair of hosts: [dtyu@aftpexp03 ~]$ /usr/sbin/traceroute cmssrv04.fnal.gov traceroute to cmssrv04.fnal.gov 1 anubis.s80.bnl.gov (130.199.80.124) 2 shu.v400.bnl.gov (130.199.136.9) 3 amon (130.199.3.24) 4 esbnl-bnl.es.net (198.124.216.113) 5 aoacr1-oc48-bnl.es.net (134.55.209.129) MPLS Label=515776 CoS=5 TTL=1 S=0 6 chicr1-oc192-aoacr1.es.net (134.55.209.57) MPLS Label=501968 CoS=5 TTL=1 S=0 7 chirt1-ge0-chicr1.es.net (134.55.209.190) 9 198.151.133.213 (198.151.133.213) 10 10cmssrv04.fnal.gov (131.225.207.95)  Un-qualified pair of hosts: [ dtyu@stargrid01 ~]$ /usr/sbin/traceroute cmssrv04.fnal.gov traceroute to cmssrv04.fnal.gov (131.225.207.95), 30 hops max, 38 byte packets 1 anubis.s80.bnl.gov (130.199.80.124) 2 shu.v400.bnl.gov (130.199.136.9) 3 amon.bnl.gov (130.199.3.24) 4 esbnl-bnl.es.net (198.124.216.113) 5 aoacr1-oc48-bnl.es.net (134.55.209.129) 6 chicr1-oc192-aoacr1.es.net (134.55.209.57) 7 fnal-pos-chi.es.net (134.55.209.37) 8 ge2-1.r-s-bdr.fnal.gov (198.49.208.4) 9 vlan313.r-s-hub-fcc.fnal.gov (131.225.15.62) 10 vlan312.r-cms-fcc1.fnal.gov (131.225.15.54) 11 cmssrv04.fnal.gov (131.225.207.95) [dtyu@aftpexp04 ~]$ iperf -c cmssrv16.fnal.gov -p 21000 -w 1MB -P 40 -t 100 -i 5 [SUM] 0.0-102.4 sec 3.08 GBytes 259 Mbits/sec (much higher than allocated the 50mbps bandwidth)

19. 19 Status ( Create best effort traffic between BNL and CERN to fill up bottleneck bandwidth for prescribed periods)  BNL participate into CERN Tier 0, tier 1 service challenge.  Prototype the data movement services that will be needed by LHC  Meet the required performance and reliability.  Network,  Disk to disk file transfer.  Reliable file transfer service.  Mass storage to mass storage file transfer.  Accomplished  Deploy and tune the four Ftp servers on INTEL based commodity PCs with SCSI drives, tune the kernel, local disk file systems.  Disk to disk file transfer between BNL and CERN GridFtp servers with GridFtp tools.  75MB/second writing to BNL disks and 80MB/second reading from BNL disk per host  Two servers filled up the network bottleneck 99%, went beyond OC 12 connection.  Sustained the data transfer rate for more than 12 hours. Stopped because of potential for disrupting other users.  BNL’s service challenge is scheduled during Christmas week.

20. 20 Network Traffic Monitoring for “Robust Service Challenge”

21. 21 Status (Network Monitoring Tools Deployment)  Deployed ganglia based host monitoring tools.  Requested ITD to evaluate Cisco proprietary network managing tools: such as COPS.  Have been working with SLAC DWMI team to deploy network monitoring tools. q IEPM_BW software is being installed on a machine outside the firewall with 100MBits interface, 50% installation was completed. SLAC team provides helps with the system installation. (Xin Zhao, Connie Logg)