1. FAST TCP in Linux Cheng Jin David Wei Steven Low California Institute of Technology

2. netlab.caltech.edu Outline Overview of FAST TCP. Design details. SC2002 experiments. On-going experiments.

3. netlab.caltech.edu Motivation Clear and present need in the HENP community. High capacity network infrastructure available today. Existing TCP protocol does not perform well.

4. netlab.caltech.edu FAST vs Linux TCP Distance = 10,037 km; Delay = 180 ms; MTU = 1500 B; Duration: 3600 s Linux TCP Experiments: Jan 28-29, 2003 1333173.182 Linux TCP txqlen=100 3909319.352 Linux TCP txqlen=10000 781851.861 Linux TCP txqlen=100 753 387 111 Transfer (GB) 1,7972 FAST 19.11.2002 9259.281 FAST 19.11.2002 2662.671 Linux TCP txqlen=10000 Throughput (Mbps) Bmps Peta Flows 18.03

5. netlab.caltech.edu Aggregate Throughput Linux TCP Linux TCP FAST Average utilization 19% 27% 92% FAST Standard MTU of 1500 bytes Utilization averaged over one hour txq=100txq=10000 95% 16% 48% Linux TCP Linux TCP FAST 2G 1G

6. netlab.caltech.edu Summary of Changes RTT estimation: high-resolution timer. Fast convergence to equilibrium. Delay monitoring in equilibrium. Pacing cwnd to smooth out traffic.

7. netlab.caltech.edu FAST TCP Flow Chart Slow Start Fast Convergence Equilibrium Loss Recovery Normal Recovery Time-out

8. netlab.caltech.edu RTT Estimation Measure queueing delay. Kernel timestamp with s resolution. Exponential averaging of RTT samples.

9. netlab.caltech.edu Fast Convergence Monitor the per-ack queueing delay to avoid overshoot. Window increment scales with the “distance” from equilibrium.

10. netlab.caltech.edu Equilibrium Define an equilibrium neighborhood around the target queueing delay. Small cwnd adjustment in large time- scale -- one per RTT. Per-ack delay monitoring to enable timely detection of changes in equilibrium.

11. netlab.caltech.edu Pacing What do we pace?  Increment to window. Time-Driven vs. event-driven.  Trade-off between complexity and performance.  Timer resolution is important.

12. netlab.caltech.edu Time-Based Pacing cwnd increments are scheduled at fixed intervals. dataackdata

13. netlab.caltech.edu Event-Based Pacing Detect sufficiently large gap between consecutive bursts and delay cwnd increment until the end of each such burst.

14. SCinet Caltech-SLAC experiments http://netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Experiment SunnyvaleBaltimore Chicago Geneva 3000km 1000km 7000km C. Jin, D. Wei, S. Low FAST Team and Partners Internet: distributed feedback system R f (s) R b ’ (s) x p TCP AQM Theory FAST TCP Standard MTU Peak window = 14,255 pkts Throughput averaged over > 1hr 925 Mbps single flow/GE card 9.28 petabit-meter/sec 1.89 times LSR 8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR 21TB in 6 hours with 10 flows Implementation Sender-side modification Delay based Highlights 1 2 1 2 7 9 10 Geneva-Sunnyvale Baltimore-Sunnyvale FAST I2 LSR #flows

15. netlab.caltech.edu Network (Sylvain Ravot, Caltech/CERN)

16. netlab.caltech.edu FAST BMPS Internet2 Land Speed Record FAST 12 1 2 7 9 10 Geneva-Sunnyvale Baltimore-Sunnyvale #flows FAST Standard MTU Throughput averaged over > 1hr

17. netlab.caltech.edu Aggregate Throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST Standard MTU of 1500 bytes Utilization averaged over one hour 1hr 6hr 1.1hr6hr

18. netlab.caltech.edu Caltech-SLAC Entry Rapid recovery after possible hardware glitch Power glitch -- Reboot 100-200Mbps ACK traffic

19. SCinet Caltech-SLAC experiments http://netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Prototype C. Jin, D. Wei Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA) Experiment/facilities  Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh  CERN: O. Martin, P. Moroni  Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip  DataTAG: E. Martelli, J. P. Martin-Flatin  Internet2: G. Almes, S. Corbato  Level(3): P. Fernes, R. Struble  SCinet: G. Goddard, J. Patton  SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams  StarLight: T. deFanti, L. Winkler  TeraGrid: L. Winkler Major sponsors ARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF Acknowledgments

20. netlab.caltech.edu Fairness Experiment I

21. netlab.caltech.edu Fairness Experiment II

22. netlab.caltech.edu 10GbE Mystery Frequent packet loss with 1500-byte MTU. None with larger MTUs. Packet loss even when cwnd is capped at 300 - 500 packets. Routers have large queue size of 4000 packets. Packets captured at both sender and receiver using tcpdump.

23. netlab.caltech.edu How Did the Loss Happen? loss detected

24. netlab.caltech.edu Future Work Redesigning a beta version. Stability and fairness experiments. More 10 GbE WAN tests. Extensive testing on shared links.