Terapaths: MPLS based Data Sharing Infrastructure for Peta Scale LHC Computing Bruce Gibbard and Dantong Yu USATLAS Computing Facility DOE Network Research PI Meeting September 15-17, 2004

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Terapaths Topics  Introduction: what is the Terapaths project  BNL network configuration and upgrade plan  WAN usage at BNL  Related projects at BNL: Grid AA and SRM/dCache  Program goals and details  Work plans and schedule  Summary and current status

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, T This project will investigate the integration and use of MPLS based differentiated network services in the ATLAS data intensive distributed computing environment as a way to manage the network as a critical resource;  The program intends to explore network configurations from common shared infrastructure (current IP networks) to dedicated optical paths point-to-point, using MPLS/QoS to span the intervening possibilities.  The Collaboration includes:  Brookhaven National Laboratory (US ATLAS Tier 1, ESNet)  Univ. of Michigan (US ATLAS Candidate Tier 2 Center, Internet2, UltraLight) What is Terapaths ?

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, BNL Campus Network and Immediate Update Plan  The USATLAS/RHIC Computing Facility is attached to access layer routers.  Campus network consists of Cisco 6509 Series Switch, MPLS support.  Cisco PIX 535 Firewall. (1Gbps)  Will be replaced by firewall service blade for CISCO 6500 series. (5G bps), Sep/Oct 2004  Core router is connected to ESnet Juniper M10 router.  WAN connection will be upgrade to OC 48 (2.5Gbps) in Oct/Nov 2004 time frame.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Network Traffic Monitoring at GridFtp server and routers

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Network Utilization  USATLAS is doing Grid enabled DC2 production and RHIC is sending physics data to the remote collaborators.  Sustain USATLAS data transfer BNL  DC2 sites at ~ 25MB/sec last month.  Goal: sustain data transfer BNL  CERN at ~ 45MBytes/sec in OCT’ 04.  Network Monitoring, performance testing (iperf, GridFtp).  BNL currently offers only best effort network service.  We periodically fill up OC-12 connection: intensive contentions over limited network bandwidth give unhappy users.  Network resource has to be systematically allocated and managed to deliver the most efficient, effective overall system!!!

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Related Projects: BNL Grid AA and SRM/dCache  GUMS (Grid User Management System) is a Grid identity mapping services: Grid credential to local credential.  Part of Privilege project: a joint project between USATLAS/USCMS.  GUMS is in production for RHIC at BNL since May/2004.  We are transforming GUMS into a service which the gatekeeper can contact directly: a preliminary implementation was completed.  Need extension to authorize network resource.  Storage Resource Managers (SRM) is to provide dynamic space allocation and file management on Grid SE. It uses GridFtp for file moving.  BNL USATLAS deployed two flavor of SRMs: Berkeley SRM and dCache.  Capable of inter-operating with each other via web services interface.  Provide access to USATLAS users.  Experience and their characteristics are documented.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Project Goal and Objectives T The primary goal of this project is to investigate the use of this technology in the ATLAS data intensive distributed computing environment. In addition we intend to: q Develop expertise in MPLS based QoS technology which will be important to ATLAS and the LHC community more generally. q Dedicate fractions of the available WAN bandwidth via MPLS to ATLAS Tier 1 data movement, RHIC data replications to assure adequate throughput and limit their disruptive impact upon each other. q Enhance technical contact between the ATLAS tier 1 at BNL and its network partners including the Tier 0 center at CERN, potential ATLAS Tier 2’s and other members of the Grid3+ (OSG- 0) community of which it is a part.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Proposed Prototype/Primitive Infrastructure GridFtp & SRM MPLS Path ESnet Network resource manager MPLS requests Traffic Identification TCP syn/fin packages, addresses, port # Grid AA Network Usage Policy Translator MPLS Bandwidth Requests & Releases OSCARS INGRESS Monitoring Direct MPLS /Bandwidth Requests SE Second/Third year

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Work Plans  Terapaths envisions a multiple year program to deliver a high- performance, QoS enable network infrastructure for ATLAS/LHC computing. Each year will determine the following year(s)’s direction.  Phase I: Establish Initial Functionality (08/04 ~ 07/05). z Help to steer the direction of the following two phases.  Phase II:Establish Prototype Production Service (08/05 ~ 07/06).  Depends on the success of Phase 1.  Phase III:Establish Full Production Service, Extend Scope and Increase Functionality (08/06 ~ 07/07). z The level of service and its scope will depends on the available project funding and some additional resources. z Broaden deployment and capability to Tier2s, partners.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Establish MPLS paths with Initial Partners SLAC LBL Sunnyvale Albuquerque Atlanta New York Chicago PAIX StarLight El Paso SLACLBNLESnet BNL DC TWCAbilene ANLFNAL MPLS CERN MICH

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Establish Initial Functionality Milestones:  Utilize network mock-up on Linux “routers” to test configuration and capability.  Setup MPLS path inside BNL campus network, connect GridFtp servers and SRM into MPLS based network.  Study Impact of MPLS on data transfer service and gain experience of using MPLS.  Study the behavior of MPLS path through firewall.  Setup MPLS paths on routers from multiple vendors (Cisco and Juniper).  Test and verify MPLS paths between BNL and LBL, SLAC (network monitoring project), FNAL and CERN.  Test and verify Inter Domain MPLS path between BNL and University of Michigan.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Contributions at Year 1  Build MPLS expertise in BNL: MPLS setup, configuration, maintenance and removal.  Learn the effectiveness and efficiency of MPLS and its impact to overall network performance: MPLS and non- MPLS.  Decide whether MPLS is useful to LHC physics.  Document any lesson/experience learned from this project.  Raise aware of that network resource can/should be managed.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Establish Prototype Production Service T Integrate Grid data transfer (GridFtp) into MPLS enabled network service. T Effectively couple these network/data transfer services with storage resources managed by SRM, have GridFtp/SRM functionality in “beta”. T Incorporate the resulting system into the ATLAS grid middleware. T Build tools to provide basic authentication, authorization and access control – Depend on funding, Rely on leverage. T Supply client interfaces which make this service available in a manner transparent to any details of the underlying QoS/MPLS traffic engineering T Leverage MPLS paths/VO level network monitoring services with DWMI project to be developed at SLAC.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Extend Scope and Increase Functionality of Prototype Service T Inter-network domain MPLS establishment, dynamically creating, adjusting of sub-partitioning MPLS paths to meet time constrained network requirements. q Create site level network resource manager for multiple VOs vying for limited WAN resource. q Provide dynamic bandwidth re-adjusting based resource usage policy and path utilization status collected from network monitoring (DWMI). q Leverage dynamic MPLS establishing services provided by OSCARS: the ESnet On-Demand Secure Circuits and Advance Reservation System. q Create user interface/web services for LHC data transfer applications to request network resource in advance.  Goal: to broaden deployment and capability to tier 1 and tier 2 sites, create services which will be honored/adopted by CERN ATLAS/LHC Tier 0.

B. Gibbard & D. Yu B. Gibbard & D. Yu DOE Network Research PI Meeting FNAL September 15-17, Summary and Status T Summary: this project will prototype and deploy a QoS capable wide area networking service based on MPLS to support ATLAS/LHC data transfer.  Current Status:  A MPLS simulation testbed is being created in a private network environment.  Evaluate mechanisms to assign different labels to GridFtp data transfers initiated by different VOs.  Different GLOBUS_TCP_PORT_RANGE for multiple VOs, source IP addresses and port numbers determine labels.  Use IP CoS bits to assign labels at border router.  Open Science Grid authentication/Authorization Systems are being developed to provide access control to various resource. (GUMS software development)  Berkeley SRM and dCache/SRM were evaluated and deployed to interface BNL storage resource into Grid.