Lambda Station Matt Crawford, Fermilab co-PI: Don Petravick, Fermilab co-PI: Harvey Newman, Caltech

HEP Computing Labs plus University Community Vast ensembles of commodity equipment Something like a petabyte of IDE disk Storage system to storage system transfer Refresh of 200 TB of state at universities Structured production, “chaotic” analysis

HEP Networking Office of High Energy Physics funds LHCnet, (OC192 triangle Starlight  CERN  MANLAN) Interested in switched optical networking –UltraLight (Caltech) –UltraScience Net (ORNL) –OSCARS MPLS tunnels (ESnet: FNAL  BNL, etc) –FNAL-CERN 875 MB/Sec SS-SS service challenge Interest, testing, and following of improvements to TCP at high bandwidth  delay Given the directions of HEP computing, the ends of “pipes” are likely to be locally, competently engineered networks.

Problem statement Experiments and applications now running, or starting soon, will benefit from data movement capabilities now available only on bleeding-edge networks. These systems are connected to production site networks. Duplicating site infrastructure to connect them to special- purpose networks is an expense to be avoided if possible. Multihoming the endpoints to multiple networks is complicated and expensive and it (nearly) precludes graceful failover when one path is lost. Applications (and operating systems) should not have to be re- customized for every new network technology or high- performance path.

Additional complications Rates are not predictable for real data sources and sinks. –Memory-to-memory is somewhat deterministic, but disk-to- disk has several uncontrolled variables. Applications may use multiple streams for maximum exploitation of high-speed links. Lambda Station must be able to deal in aggregates. Straggler flows persist after bulk of transfer has completed, and continued use of high-volume path may be wasteful at that point. Aggressive protocols for the wide area may have negative impacts on the last mile (site or site’s “uplink”) network.

Lambda Station Function –Schedule use of one or more reservable network paths –Arrange for traffic to be forwarded onto such paths

Interfaces to other systems To application (or to manual request system) To authentication/authorization infrastructure To site’s internal network (dynamic reconfiguration of packet forwarding rules) –Operate at any granularity, down to single flows Site’s border/connection point to reservable path Peer site’s Lambda Station Talk to advanced WANs, through network operator- defined setup protocol, as needed* Monitoring, accounting, status reporting

Block Diagram

Client application interface Application describes the traffic which is to be routed over an alternative path. –Traffic selectors: 6-tuples [ IP version, {src cidr(s)}, {dst cidr(s)}, protocol, {src port(s)}, {dst ports(s)} ] –Transfer rate, total volume, duration, direction –Earliest desired start LS and host agree on packet-selection method - we lean toward DSCP. LS informs application of actual BW allocated and setup status. Host or LS should inform the other of early termination, if it occurs.

Site network interface Configure local site’s internal routing to divert traffic to the alternate path. Graceful teardown – resume normal internal routing before WAN path is torn down. Different version of this module will deal with different varieties of site network. –Each site might plug in its own scripts.

Site-edge router interface Graceful setup – Enable the reserved WAN path before internal routing directs traffic onto it. ACL may be in effect on this device to prevent unauthorized use. ACL very likely to be in effect with respect to incoming traffic from the WAN. –At some sites, this is a path which bypasses firewalls!

LS-to-LS protocol Exchange traffic selectors Coordinate setup & teardown Verify path continuity –Implies that LS can communicate simultaneously over reserved and commodity network paths. Inform of early traffic termination

Interface to advanced WANs Multiple flavors of high-performance WANs are anticipated. –Some WANs may require forwarding state to be created before use. –Some may have their own reservation system, which end systems need not learn to use if it reserves through Lambda Station instead. Lambda Station’s WAN module will parameterize and adapt to each sort of WAN, providing an abstract view. –DOE UltraScience Net, ESNET, LHCNet, UltraLight.

Requirements for Production Robustness –LS must enable production systems to make trial use of advanced networks, and cleanly restore default forwarding behavior upon completion or path failure. Monitoring –Lambda Station must present its own state and history. –Currently it serves this info through its web server. –Investigating MonaLisa (OSG component). Accounting –In many environments, different sub-organizations share the network resource. LS must gather usage information to support accounting.

Provide sample integration With Storage Systems that are components of the USCMS software and computing project. Currently are: –Managed storage elements. SRM / GridFTP protocols. Now implementing LS client calls in SRM/dCache.

Current status Release 1.0 – today. –A stable, usable snapshot of a work in progress. –Based on Perl with SOAP::Lite –Dynamically reconfigures site routers to send traffic over alternate paths –End system applied DSCP tags to special- treatment flows. –Traffic path varied cleanly – unnoticed by application; hiccups in throughput at each change.

Path switching effects

Deployment Scenarios Client capabilities: identifying high-impact traffic Specify src & dst address groups, but no more. 2. Specify src and/or dst ports as well as addresses. 3. Apply DSCP label selected by client 4. Apply DSCP label as directed by Lambda Station. Client capabilities: Lambda Station integration level Lambda Station called manually via web interface 2. SOAP call by wrapper around client application 3. SOAP calls from within the client application Site network capabilities Static router config w/ fixed PBR based on DSCP 2. Router ACLs activated and inactivated by LS 3. Lambda Station constructs and applies ACLs for PBR

Directions Next version being built on Apache Axis –probably will use jClarens –WSDL is sure to evolve IPv6 support is mere placeholder as yet Adding support for Force10 site routers Looking forward to speaking to your lightpath setup directly!

Summary Lambda Station’s role in data-intensive science is to dynamically connect production end-systems to advanced high-performance wide-area networks. –Bring the systems to the network –Bring the network to the systems Prototyping has shown the feasibility of using dynamically selected network paths for traffic between production site networks.

Intriguing sidelights Make asymmetric reservations when possible? (What’s an appropriate bandwidth reservation for an ACK stream?) Does LS set up bandwidth policing? –Would apply when multiple concurrent reservations are active. Is the access authorization a site-local decision? Is there a VO role in this? Does LS have knowledge of QoS facilities available in the site network or the WAN?