TransPAC HPCC Engineer TCP performance John Hicks TransPAC HPCC Engineer Indiana University APAN 19 Meeting – Bangkok, Thailand 27-January-2005

Standard TCP Low performance on fast long distance paths AIMD (add a=1 pkt to cwnd / RTT, decrease cwnd by factor b=0.5 in congestion) RTT (~70ms) RTT ms Reno Throughput Mbps 700 1200 s Information courtesy of Les Cottrell from the SLAC group at Stanford

Standard TCP problems It has long been recognized that TCP does not provide good performance for applications on networks with a high bandwidth delay product. Some things that contribute to this include: Slow linear increase by one packet per RTT. Too drastic multiplicative decrease. Packet level oscillation due to packet loss.

Standard TCP improvements One approach to improving TCP performance is to adjust the TCP window size to be the bandwidth delay product of the network. This approach usually requires a network expert. Hard to achieve in practice. Another approach is to stripe TCP over several standard TCP network connections. This is approach plateaus as the number of sockets increase.

Improving data rates There are basically three categories of projects to improve data transfer rates: Rate limited Examples include: (Robert Grossman) SABUL, RBUDP, and (Steve Wallace) Tsunami. Congestion window limited Examples include: (Steven Low ) FAST, (Tom Kelly) Scalable, (Sally Floyd) Highspeed, and (Injong Rhee) BicTCP. Hardware changes Examples include: XCP One proving ground for these activities is the Bandwidth Challenge at the Supercomputing Conference

One of the SC04 BWC winners All Roads Lead Through Chicago to Pittsburgh Performance Award: High Speed TeraByte Transfers for Physics California Institute of Technology, Stanford Linear Accelerator Lab and Fermi National Lab. Over 100 Gb/s per second aggregate memory to memory bandwidth utilizing the greatest number of networks “Caltech, SLAC, Fermilab, CERN, Florida and Partners in the UK, Brazil and Korea Set 101 Gigabit Per Second Mark During the SuperComputing 2004 Bandwidth Challenge” This group used FAST TCP as a transfer mechanism.

FAST TCP Based on TCP Vegas Uses both queuing delay and packet losses as congestion measures Developed at Caltech by Steven Low and collaborators Code available at: http://http://netlab.caltech.edu/FAST/ Information courtesy of Les Cottrell from the SLAC group at Stanford

FAST TCP at SC04

Scalable TCP Uses exponential increase everywhere (in slow start and congestion avoidance) Multiplicative decrease factor b = 0.125 Introduced by Tom Kelly of Cambridge Information courtesy of Les Cottrell from the SLAC group at Stanford

Highspeed TCP Behaves like Reno for small values of cwnd Above a chosen value of cwnd (default 38) a more aggressive function is used Uses a table to indicate by how much to increase cwnd when an ACK is received Available with web100 Introduced by Sally Floyd Information courtesy of Les Cottrell from the SLAC group at Stanford

Binary Increase Control TCP (BIC TCP) Combine: An additive increase used for large cwnd A binary increase used for small cwnd Developed by Injong Rhee at NC State University Information courtesy of Les Cottrell from the SLAC group at Stanford

For More Information Supercomputing 2004, Bandwidth Challenge http://scinet.supercomp.org/2004/bwc/ FAST TCP http://netlab.caltech.edu/FAST/ Scalable TCP http://www-lce.eng.cam.ac.uk/~ctk21/scalable/ Highspeed TCP http://www.icir.org/floyd/hstcp.html Binary Increase Control (BIC) TCP http://www.csc.ncsu.edu/faculty/rhee/export/bitcp/

Thank you John Hicks Indiana University jhicks@iu.edu