Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 26, 2001

Introduction Applications experience network performance from a end customer perspective Providing end-to-end performance has two aspects –Bandwidth Reservation –Performance Tuning We have been working to improve actual end-to- end throughput using Performance Tuning This work allows applications to fully exploit reserved bandwidth

Improve Network Performance Poor network performance arises from a subtle interaction between many different components at each layer of the OSI network stack Physical Data Link Network Transport Application

TCP Bandwidth Limits – Mathis Equation Based on characteristics from physical layer up to transport layer. Hard Limits TCP Bandwidth, Max Packet Loss

Packet Loss and MSS If the minimum link bandwidth between two hosts is OC-12 (622 Mbps), and the average round trip time is 20 msec, the maximum packet loss rate necessary to achieve 66% of the link speed (411 Mbps) is approximately %, which represents only 2 packets lost out of every 100,000 packets. If MSS is increased from 1500 bytes to 9000 bytes (Jumbo frames), limit on TCP BW will rise by a factor of 6.

The Results

Web100 Collaboration

SOURCE: Harimath Sivakumar, Stuart Bailey, Robert L. Grossman. “PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks,” SC2000: High-Performance Network and Computing Conference, Dallas, TX, 11/00 Parallel TCP Connections…a clue

Why Does This Work? Assumption is that network gives best effort throughput for each connection But end-to-end performance is still poor, even after tuning the host, network, and application Parallel Sockets are being used in GridFTP, Netscape, Gnutella, Atlas, Storage Resource Broker, etc.

Packet Loss Bolot* found that Random losses are not always due to congestion –local system configuration (txqueuelen in Linux) –Bad cables (noisy) Packet losses occur in bursts TCP throttles transmission rate on ALL packet losses, regardless of the root cause Selective Acknowledgement (SACK) helps, but only so much * Jean-Chrysostome Bolot. “Characterizing End-to-End packet delay and loss in the Internet.”, Journal of High Speed Networks, 2(3): , 1993.

Expression for Parallel Socket Bandwidth

Example MSS = 4418, RTT = 70 msec, p = 1/10000 for all connections Number of Connections Aggregate Bandwidth Mb/sec Mb/sec Mb/sec 44 (100)200 Mb/sec 5 5 (100)250 Mb/sec

Measurements To validate theoretical model, minute transmissions performed from U-M to NASA AMES in San Jose, CA Bottleneck was OC-12, MTU= runs MSS=4366, 1 to 20 sockets 2 runs MSS=2948, 1 to 20 sockets 2 runs MSS=1448, 1 to 20 sockets Iperf used for transfer, Web100 used to collect TCP observations on sender side

Actual: MSS 1448 Bytes

Actual: MSS 2948 Bytes

Actual: MSS 4366 Bytes

Observations Knee in the curve is the point at which aggregate throughput reaches the delay*bandwidth product of the pipe between sender and receiver. On a long link (trans-Atlantic), the pipe can hold a lot of data, since the delay (RTT) is so large. Parallel sockets should only work if there are are no router drops. TCP will try to fill the pipe on a single stream.

Sunnyvale – Denver Abilene Link Initial Tests Yearly Statistics

Abilene Weather Map

Other stuff Internet2 measurements on Cleveland Abilene node Interesting results from ns-2 simulation – is it just a simulation artifact? Working on a loss model for Abilene that differentiates between router drops and random drops

Conclusion High Performance Network Throughput is possible with a combination of host, network and application tuning along with using parallel TCP connections Parallel TCP Sockets mitigate negative effects of packet loss in random congestion regime Effects of Parallel TCP Sockets similar to using larger MSS Using Parallel Sockets is aggressive, but as fair as using large MSS