MB-NG Review High Performance Network Demonstration 21 April 2004 Richard Hughes-Jones The University of Manchester, UK

It works ? So what’s the Problem with TCP TCP has 2 phases: Slowstart & Congestion Avoidance AIMD and High Bandwidth – Long Distance networks Poor performance of TCP in high bandwidth wide area networks is due in part to the TCP congestion control algorithm - cwnd congestion window For each ack in a RTT without loss: cwnd -> cwnd + a / cwnd - Additive Increase, a=1 For each window experiencing loss: cwnd -> cwnd – b (cwnd) - Multiplicative Decrease, b= ½ Time to recover from 1 packet loss ~100 ms rtt: 2

Investigation of new TCP Stacks High Speed TCP a and b vary depending on current cwnd using a table a increases more rapidly with larger cwnd – returns to the ‘optimal’ cwnd size sooner for the network path b decreases less aggressively and, as a consequence, so does the cwnd. The effect is that there is not such a decrease in throughput. Scalable TCP a and b are fixed adjustments for the increase and decrease of cwnd a = 1/100 – the increase is greater than TCP Reno b = 1/8 – the decrease on loss is less than TCP Reno Scalable over any link speed. Fast TCP Uses round trip time as well as packet loss to indicate congestion with rapid convergence to fair equilibrium for throughput. HSTCP-LP High Speed (Low Priority) – backs off if rtt increases BiC-TCP – Additive increase large cwnd; binary search small cwnd H-TCP – after congestion standard then switch to high performance ●●● 3

Comparison of TCP Stacks TCP Response Function Throughput vs Loss Rate – steeper: faster recovery Drop packets in kernel MB-NG rtt 6ms DataTAG rtt 120 ms 4

Multi-Gigabit flows at SC2003 BW Challenge Three Server systems with 10 GigEthernet NICs Used the DataTAG altAIMD stack 9000 byte MTU Send mem-mem iperf TCP streams From SLAC/FNAL booth in Phoenix to: Chicago Starlight rtt 65 ms window 60 MB Phoenix CPU 2.2 GHz 3.1 Gbit hstcp I=1.6% Amsterdam SARA rtt 175 ms window 200 MB 4.35 Gbit hstcp I=6.9% New TCP stacks are very Stable Both used Abilene to Chicago 5

Transfer Applications – Throughput [1] 2Gbyte file transferred RAID0 disks Manc – UCL GridFTP See alternate 600/800 Mbit and zero Apache web server + curl-based client See steady 720 Mbit 6

Transfer Applications – Throughput [2] 2Gbyte file transferred RAID5 - 4disks Manc – RAL bbcp Mean 710 Mbit/s Mean ~710 GridFTP See many zeros Mean ~620 7

Topology of the MB – NG Network Manchester Domain UKERNA Development Network man02 UCL Domain lon01 Boundary Router Cisco 7609 Boundary Router Cisco 7609 man01 lon02 Edge Router Cisco 7609 man03 lon03 ral02 RAL Domain Key Gigabit Ethernet 2.5 Gbit POS Access MPLS Admin. Domains ral01 ral02 Boundary Router Cisco 7609 8

High Throughput Demo London Manchester Dual Zeon 2.2 GHz Monitor TCP with Web100 lon01 man03 Cisco 7609 Cisco GSR Cisco GSR Cisco 7609 Drop Packets 1 GEth 2.5 Gbit SDH MB-NG Core 1 GEth Send data with TCP 9

Standard to HS-TCP No loss, but output queue filled by sender 10

HS-TCP to Scalable No loss, but output queue filled by sender 11

Standard, HS-TCP, Scalable Drop 1 in 25,000 12

Standard Reno TCP Drop 1 in 106 13

Focus on Helping Real Users: Throughput CERN -SARA Using the GÉANT Backup Link 1 GByte disk-disk transfers Blue is the Data Red is the TCP ACKs Standard TCP Average Throughput 167 Mbit/s Users see 5 - 50 Mbit/s! High-Speed TCP Average Throughput 345 Mbit/s Scalable TCP Average Throughput 340 Mbit/s Technology link to EU Projects: DataGrid DataTAG & GÉANT 14

BaBar Case Study: Host, PCI & RAID Controller Performance RAID0 (striped) & RAID5 (stripped with redundancy) 3Ware 7506 Parallel 66 MHz 3Ware 7505 Parallel 33 MHz 3Ware 8506 Serial ATA 66 MHz ICP Serial ATA 33/66 MHz Tested on Dual 2.2 GHz Xeon Supermicro P4DP8-G2 motherboard Disk: Maxtor 160GB 7200rpm 8MB Cache Read ahead kernel tuning: /proc/sys/vm/max-readahead Disk – Memory Read Speeds Memory - Disk Write Speeds 15

Topology of the MB – NG Network Manchester Domain UKERNA Development Network man02 UCL Domain lon01 Boundary Router Cisco 7609 Boundary Router Cisco 7609 man01 lon02 HW RAID Edge Router Cisco 7609 man03 lon03 ral02 RAL Domain Key Gigabit Ethernet 2.5 Gbit POS Access MPLS Admin. Domains ral01 HW RAID ral02 Boundary Router Cisco 7609 16

BaBar Data: Throughput on MB–NG kit RAID5 - 4disks RAL - Manc Includes small files ~Kbytes bbftp 1 stream with compression With bb diag bbftp 6 streams bbftp 1 stream no compression 10 * 2 G byte files – each peak is a 20 G byte transfer bbftp 1 stream Files ≥ 1 Mbyte 17

Helping Real Users Radio Astronomy VLBI PoC with NRNs & GEANT 1024 Mbit/s 24 on 7 NOW 18

VLBI Project: Throughput Jitter & 1-way Delay 1472 byte Packets man -> JIVE FWHM 22 µs (B2B 3 µs ) 1472 byte Packets Manchester -> Dwingeloo JIVE 1-way Delay – note the packet loss (points with zero 1 –way delay) 19

Case Study: ATLAS LHC Tests streaming built Events from Level3 Trigger to remote compute farm in real time 500 Mbit to 1 Gbit CERN – Man Investigation of use of new high performance TCPs Testing concepts in the ATLAS Offline Computing model More Mesh than Star: CERN Tier0 to Tier 1s Tier 2s to all Tier 1s Tests planned over production networks: Lancaster-Manchester NNW SuperJANET4 Lancaster-Manchester to CERN 20

21

Scalable TCP DataTAG Drop 1 in 106 22

HS-TCP DataTAG Drop 1 in 106 23

Standard Reno TCP DataTAG Drop 1 in 106 Transition highspeed to Standard TCP @ 520s 24

Summary Multi-Gigabit transfers are possible and stable Demonstrated that new TCP stacks help performance DataTAG has made major contributions to understanding of high-speed networking There has been significant technology transfer between DataTAG and other projects Now reaching out to real users. But still much research to do: Achieve performance – Protocol vs implementation issues Stability / Sharing issues Optical transports & hybrid networks 25

10 Gigabit: Tuning PCI-X 16080 byte packets every 200 µs Intel PRO/10GbE LR Adapter PCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s mmrbc 1024 bytes 2048 bytes 4096 bytes 5.7Gbit/s 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update 26

DataTAG Testbed 27

BaBar Case Study: Disk Performance BaBar Disk Server Tyan Tiger S2466N motherboard 1 64bit 66 MHz PCI bus Athlon MP2000+ CPU AMD-760 MPX chipset 3Ware 7500-8 RAID5 8 * 200Gb Maxtor IDE 7200rpm disks Note the VM parameter readahead max Disk to memory (read) Max throughput 1.2 Gbit/s 150 MBytes/s) Memory to disk (write) Max throughput 400 Mbit/s 50 MBytes/s) [not as fast as Raid0] 28

RAID Controller Performance Read Speed Write Speed RAID 0 RAID 5 29

BaBar: Serial ATA Raid Controllers RAID5 3Ware 66 MHz PCI ICP 66 MHz PCI 30

VLBI Project: Packet Loss Distribution Measure the time between lost packets in the time series of packets sent. Lost 1410 in 0.6s Is it a Poisson process? Assume Poisson is stationary λ(t) = λ Use Prob. Density Function: P(t) = λ e-λt Mean λ = 2360 / s [426 µs] Plot log: slope -0.0028 expect -0.0024 Could be additional process involved 31

The performance of the end host / disks BaBar Case Study: RAID BW & PCI Activity 3Ware 7500-8 RAID5 parallel EIDE 3Ware forces PCI bus to 33 MHz BaBar Tyan to MB-NG SuperMicro Network mem-mem 619 Mbit/s Disk – disk throughput bbcp 40-45 Mbytes/s (320 – 360 Mbit/s) PCI bus effectively full! User throughput ~ 250 Mbit/s Read from RAID5 Disks Write to RAID5 Disks 32