Report of the “DOE Workshop on Ultra High-Speed Transport Protocols and Dynamic Provisioning for Large-Scale Science Applications” April 10-11, 2003, Argonne, IL Panel on Future Directions in Networking International Conference on Network Protocols November 5-7, 2003, Atlanta, GA Nagi Rao (Nageswara S.V. Rao) Computer Science and Mathematics Division Oak Ridge National Laboratory Transport Protocols and Dynamic Provisioning for Large-Scale Science Applications

We engineered the Internet, and it works fine for and web; but to do “world-class” scientific research needed in DOE scientific applications, we need to develop a science of networking that delivers usable performance to the applications Allyn Romanow, Cisco Systems

Introduction Organization Details Provisioning Group Notes Transport Group Notes Dynamics and Stabilization of Network Transport Conclusions Outline Opinions expressed in this presentation belong to the author and are not necessarily the official positions of US Department of Energy, Oak Ridge National Laboratory or UT-Battelle LLC.

Next generation of DOE scientific breakthroughs critically depend on large multi-disciplinary geographically dispersed research teams: – high energy physics, climate simulation, fusion energy, genomics, astrophysics, spallation neutron source, and others These applications are Inherently distributed in: Data – archival or on-line Computations – supercomputers or clusters Research teams – experts in different domains Experimental facilities – one of the kind user facilities – they all need to be seamlessly networked Networking for DOE large-science applications

DOE Large-Scale Science Applications and Numerous Other Science Applications - need extreme and acute networking Detailed account of the needs were identified and discussed at DOE High-Performance Network Planning Workshop, August 13-15, 2002, Science Areas Current End2End Throughput 5 years End2End Throughput 5-10 Years End2End Throughput General Remarks High Energy Physics0.5 Gbps E2E100 Gbps E2E1.0 Tbpshigh throughput Climate Data & Computations 0.5 Gbps E2E Gbpsn Tbpshigh throughput SNS NanoScience does not exist1.0 Gbps steady state Tbps & control channels remote control & high throughput Fusion Energy500MB/min (Burst) 500MB/20sec (burst) n Tbpstime critical transport Astrophysics1TB/weekN*N multicast1TB+ & stable streams computational steering & collaborations Genomics Data & Computations 1TB/day100s usersTbps & control channels high throughput & steering Need more than bulk bandwidth

Science Objective: Understand supernova evolutions –Teams of field experts across the country collaborate on computations Experts in hydrodynamics, fusion energy, high energy Universities and national labs –Massive computational code Terabyte in days are generated currently Archived at nearby HPSS Visualized locally on clusters – only archival data Desired capability –Archive and supply massive amounts of data –Collaboratively visualize archival or on-line data –Monitor, visualize and steer computations into regions of interest Astrophysics Computations Visualization channel Control channel

Data Movement Operations –Experimental and computational data Stored across the country Terabytes of data per day –Between users, archives and computers Molecular Dynamics Computations –Supercomputers or clusters –Monitor, visualize, and steer computations Genomics Networking Needs data channel visualization channel steering channel

Experimental Setups and Monitoring of Expensive Facilities (SNS – billion$) –Setup parameters and start experiments –Adjust parameters as needed; stop when necessary Data Movements –Archive and access massive amounts of experimental data Neutron Facilities – SNS, HFIR

DOE faces unique or acute challenges: Small user base with extreme needs –large data transfers at application-level – rates much higher than current backbones –highly controlled end-to-end data streams – unprecedented agility and stability –capabilities must be available to science users – not just to network experts with special networks Commercial and other networks will not adequately meet these acute requirements –Not large enough user base –Very limited business case New advances in Transport and Provisioning hold enormous promise, if suitably fostered and integrated –Flexible and powerful routers/switches, ultra high bandwidth links, new transport protocols can get us partway there –But, need several critical technologies and expertise: end-to-end dynamic provisioning of paths with guaranteed performance transport methods that optimally provide to user applications Current Network Capabilities: Transport and Provisioning

Workshop Goal Address the research, design, development, testing and deployment aspects of transport protocols and network provisioning as well as the application-level capability needed to build operational ultra-speed networks to support emerging DOE distributed large-scale science applications over the next 10 years.

Workshop Focus: Ultra High-Speed Networks to support DOE Large-Science Applications –not a general network research workshop addressing Internet problems Formulate DOE roadmap in the specific areas: –Transport and Provisioning two very critical subareas of network research needed to meet DOE large-science requirements Not in other areas such as security, wireless networks “Working” workshop –Discussions on very specific problems, methods, potential solutions in transport and provisioning areas –Very short introductory presentations –Not just primarily informational or educational

Balanced participation from universities, industry and national laboratories to represent the needs, technologies, research and business aspects Total: 32 National Laboratories: 10 ORNL:3; ANL:2: LANL:2; PNNL:1; SLAC:1; ESnet: 1 Universities: 11 UMass; GaTech(2), Uva, UIC,Indiana U, U Va, U Tennessee, UC Davis, PSC, CalTech Industry:8 Celion, Cienna, Cisco, Juniper, Level3, Lightsand, MCNC, Qwest DOE Headquarters: 3 Working Groups: Provisioning: 14 Transport:15 Participants

Need Focused Efforts in Develop a scalable architecture for fast provisioning Circuit Switched Network Build an application-centric circuit-switched cross country test-bed Coordination and Graceful integration with Applications and Middleware Transport and OS Developers Legacy and evolutionary networks Summary: Provisioning for DOE Large-Science Networks

Recommendation 1: Agile Optical Network infrastructure: –A scalable architecture for fast provisioning of circuit switched dedicated channels specified on-demand by the applications. Recommendation 2: Hybrid Switched Networks: –High capacity (Tbps) switchable channels for Petabyte data transport, a combination of requirements to accommodate burst, real-time streams as well as lower priority traffic, multi-point or shared use, for large file and data transfers, and for low latency and low jitter. Recommendation 3: Dynamically Reconfigured Channels: –Provisioning of dynamically specified end-to-end quality paths for computational steering and time-constrained experimental data analysis. Recommendation 4: Multi-Resolution Quality of Service: –Channels with various types of Quality of Service (QoS) parameters must be supported at various resolutions using GMPLS, service provisioning and channel sharing technologies. Recommendation 5: Experimental Test-Bed Provisioning Recommendations

Limited deployment of ultra-long haul DWDM links; Lack of support for striped/parallel transport both at the core and application levels; Lack of high-speed circuit-switched infrastructure with network control-plane design and synchronous NICs with high-speed and on-demand reconfigurability; and Lack of well-developed methods and application interfaces for scheduling/reserving, allocation, and initiation. DOE applications do not follow the commercial scaling model of large number of users each with smaller bandwidth requirements; Lack of security paradigms for dedicated paths and the infrastructure that to manage them; Lack of a robust multi-cast solution efficiently supported on dedicated channels; High cost of equipment, including the costs of links, routers/switches and other equipment as well as deployment and maintenance; Lack of field-hardening of optical components such as memory/buffer, high-speed switches, Reamplification, Reshaping and Retiming (RRR) equipment, and lambda conversion gear; Lack of effective contention resolution methods for the allocation of channel pools; and Limited interoperability with other data networks, particularly legacy networks. Provisioning: Barriers

Current transport methods are massively inadequate –Unattained Throughput: wizards can achieve several Gbps for certain durations But throughputs are needed at application user level –Cannot provide sustained and stable streams for control operations –TCP has complicated dynamics – hard to use in finer control operations Need focused efforts in: –Optimal transport methods to exploit provisioning to meet requirements Transport -Tbps throughputs Support stable and agile control channels –Comprehensive theory of transport: synergy and extensions of traditional disciplines Stochastic control, non-linear control, statistics, optimization, protocol engineering –Strict algorithmic design Modular, autonomic, adaptive, composable Integration and interactions: –DOE deployment, wider adoption, legacy integration –Experiment and test-bed –Instrumentation and Diagnostics tools web100/Net100 Statistical Inference and optimized data collection Summary: Transport for DOE Large-Science Networks

Scientists (must) view the network as they view a computer as a resource But they are becoming (not always willingly) network experts – “wizard gap” at all levels but “gray matter” tax must be low Advance the state of network protocols to make them plug-and- play for the application users -need significant effort Time-to-solution in networking area is currently too high – TCP tuning for Gbps throughputs took years Peak is not enough – need sustained throughputs at application level Transport Group Notes

Recommendation 1: Transport Protocols and Implementations –Transport methods for dedicated channels and IP networks for achieving high throughput, steering and control. The transport methods include TCP-, UDP- and SAN- based methods together with newer approaches. Recommendation 2: Transport Customization and Interfacing –Transport methods optimized to single and multiple hosts as well as channels of different modes. –Transport methods suitably interfaced with storage methods to avoid impedance mismatches that could degrade the end-to-end transport performance. Recommendation 3: Stochastic Control Methods –Stochastic control theoretic methods to design protocols with well-understood and/or provable stability properties. Recommendation 4: Monitoring and Estimation Methods –Monitoring and statistical estimation techniques to monitor the critical transport variables and dynamically adjust them to ensure transport stability and efficiency. Recommendation 5: Experimental Test-Bed Transport Group Recommendations: 1-5 years

Recommendation 1: Modular Adaptive Composable and Optimized Transport Modules: –Highly dynamic and adaptive methods to dynamically compose transport methods to match the application requirements and the underlying provisioning. Recommendation 2: Stochastic and Control Theoretic Design and Analysis: –Stochastic control theoretic methods for composable transport methods to analyze them as well as to guide their design to ensure stability and effectiveness Recommendation 3: Graceful Integration with Middleware and Applications: –Application data and application semantics must be mapped into transport methods to optimally meet application requirements –boundary between middleware and transport must be made transparent to applications. Recommendation 4: Vertical Integration of Applications, Transport and Provisioning: –Vertical integration of resource allocation policies (cost and utility) with transport methods to present a unified view and interface to the applications. Transport Group Recommendations: 5-10 years

There is a need for systematic scientific approaches to the design, analysis and implementation of the transport methods and to network provisioning. On-Demand Bandwidth and Circuit Optimization: –Dynamic optimization and scheduling methods to allocate the bandwidth pipes to applications. –A comprehensive approach for on-line estimation and allocation of the “bandwidths” –Signaling to provide the required timeliness and reliability of the allocated channels. –Scientific, systematic understanding to integrate the components for bandwidth allocation, channel scheduling, channel setup and teardown, and performance monitoring. Comprehensive Theory of Transport: –Rigorous transport design methods tailored to the underlying provisioning modes. –A synergy and extensions of a number of traditional disciplines. –New stochastic control methods may be required to design suitable transport control methods. –Non-linear control theoretic methods to analyze delayed feedback. –Statistical theory for designing rigorous measurements and tests. –Optimization theory to obtain suitable parameters for tuning protocols. Strict Algorithmic Design and Implementation: –Strict algorithmic design methods to efficiently implement the designed protocols. –Implementations must be modular, autonomic, adaptive, and composable. Statistical Inference and Optimized Data Collection: –Due to the sheer data volumes, it is inefficient to collect measurements from all nodes all the time for the purposes of diagnosis, optimization and performance tuning. –Systematic inferencing methods to identify the critical and canonical sets of measurements needed. –Statistical design of experiments to ensure that the measurements are strategic and optimal. Science of High-Performance Networking

High-Performance Network Test-beds: Recommended by both groups State-of-the-art Components: software and hardware networking components, including routers/switches, high bandwidth long-haul links, protocols and application interface modules. Integrated Development Environments: mechanisms to integrate a wide spectrum of network technologies including high throughput protocol, dynamic provisioning, interactive visualization and steering, and high performance cyber security measures Smooth Technology Transition: transition of network technologies from research stages to production stages by allowing them to mature in such an environment. Characteristics of ultra high-speed network test-bed: 1.Interconnection of at least three science facilities with large-scale science applications; 2.Geographical coverage adequate to capture optical characteristics, transport protocols dynamics, and application behaviors comparable to that of real-word applications; 3.Integration with appropriate middleware; 4.Scalable network measurement tools; and 5.Well-defined technology transfer plan.

Integration, Interaction, and Interfacing Applications are empowered to “tune” the network Application 1Application 2Application 3 IP provisioning Dynamic lambda switching Net100 modules Non-TCP protocols Stabilize module Control modules UltraNet Middleware Protocols Network-Aware Applications Supernova: Large data stream Control stream Molecular dynamics visualization HEP data transfers

While not on the original agenda, security issues have significant impact on application performance – DOE sites have very strict firewalls Securing Operational and Development Environments: –authentication, validation and access controls –data speeds of multiple tens of Gbps or higher –new security methods for on-demand dedicated channels. Effects of Security Measures on Performance: –impact of security measures on application performance. –graceful interoperation of science applications under secured network environments. Proactive Countermeasures: –protect bandwidth allocation, and signaling to setup and tear down the paths –vulnerability of new transport protocols to certain attacks Network Security Issues