NASA EOS Active Network Performance Testing Using Web100 Andy Germain Swales Aerospace 1 August

24 June 2002 Andy Germain EOS Active Testing Overview End-to-end user level test –Active testing, no visibility into network internals Communities –EOS Internal Network: 9 Sites, 8 Sources, 13 Sinks "Production" Flows, dedicated bandwidth –EOS Science Users: About 50 sites, tested from EOS DAACs "QA" and Science flows, often via Abilene –CEOS: About 20 International sites Earth Observation data sharing Purposes –Verify that networks as implemented meet SLA and/or requirements –Assess whether networks can support intended applications –Resolve user complaints: Network problems -- or elsewhere?? –Determine bottlenecks -- seek routing alternatives –Provide a basis for allocation of additional resources Results at

24 June 2002 Andy Germain Test Process Test script runs hourly to each site: Traceroute (1 way) –Number of hops -- route stability Hops Chart Pings –100 pings prior to thruput test and/or 100/300 during –Round Trip Time RTT Chart –Packet Loss Packet Loss Chart TCP Throughput –Iperf Thruput Chart –keeps send buffer full for 30 Seconds –Netstat packets retransmitted (if pings blocked)

24 June 2002 Andy Germain EOS DAAC NASA Nodes SCFs QA Other Key: ORST UCSB Ariz LANL Wisc Miami SUNY-SB BU GSFC LaRC EDC MSFC, NSSTC NCAR Mont JPL Toronto Colo St. Niagara ASF Chicago Other Nodes SLAC NSIDC NMEX CCRS UVA UMD GPN NGDC, NOAA USF RSS EOS Performance Test Sites Texas UCSD Wash Mich NOAA Ohio Penn State NCDC MIT

24 June 2002 Andy Germain EOSDIS Mission Partner CEOS PI: QA/IST EOS International Test Sites GSFC CCRS JPL NASDA (ADEOS, TRMM, Aura, Aqua) CSIRO ESRIN INPE (Aqua), IDN CONAE IRE-RAS Israel ASF NSIDC EDC LaRC MITI (Terra) CAO (SAGE III) RAL, OXFORD (Aura) Toronto (Terra) UCL (Terra) JRC AIT, RFD, GISTDA KNMI (Aura)

24 June 2002 Andy Germain Uses of Web100 One of our sources at GSFC runs Web100 –King = "GSFC MAX" –Connected to MAX by GigE Typical use is in problem solving –DTB, Triage Window size (easier to use than tcpdump) Vs. circuit limitations vs. packet loss –Also ANLiperf Window size again Plan: extract packet drops from web100, not pings or netstats

24 June 2002 Andy Germain A recent case Sending data from LaRC to JPL via a project dedicated 20 mbps ATM VC. –Problem surfaced after firewall was installed Portus "proxy" firewall RTT of 60 ms requires 150 KB windows –To fill pipe with a single TCP stream Iperf worked well – a single stream typically got over 15 mbps But ftp got < 8 mbps

24 June 2002 Andy Germain A recent case (2) The problem, of course, was window size –Looked like it was the ftp application, since iperf performance showed that O/S was OK –But which end? Ran ftps from both nodes to web100 node –Used DTB to capture window size –Problem: small disk quota FTPs were quick FTP data session not established until ftp started So had to be quick to capture data with DTB –DTB showed one site had 64 kb windows But problem was in O/S (IRIX), not ftp –Tcp_recvspace and tcp_sendspace –Iperf can exceed O/S defaults!

24 June 2002 Andy Germain Case #2 Another case of limited thruput –This time iperf was limited –from one source to several destinations –Limit inverse to RTT window size –But source and dest clearly used large windows Testing to Web100 box showed source was not using extended windows TCPdump on source showed it was! Problem turned out to be PIX firewall –Nop'd out the WSCALE field!

24 June 2002 Andy Germain Case #3 Iperf from GSFC to Tokyo XP –Via MAX, Abilene, Seattle, TransPac Thruput appears to ramp up linearly for about 5 minutes (when no loss) –Then becomes window limited: 1 MB 188 ms RTT 42.5 mbps –Repeatable (more or less) –Low or no packet loss Web100 Triage usually reports 100% path limited –But can't show early part of session (?) What causes this ramp-up ???

24 June 2002 Andy Germain Traceroute traceroute to perf.jp.apan.net ( ), 30 hops max, 38 byte packets 1 enpl-rtr1-ge ( ) ms ms ms ( ) ms ms ms ( ) ms ms ms 4 gsfc-wash.maxgigapop.net ( ) ms ms ms 5 dcne-so3-1-0.maxgigapop.net ( ) ms ms ms 6 abilene-wash-oc48.maxgigapop.net ( ) ms ms ms 7 nycm-wash.abilene.ucaid.edu ( ) ms ms ms 8 clev-nycm.abilene.ucaid.edu ( ) ms ms ms 9 ipls-clev.abilene.ucaid.edu ( ) ms ms ms 10 kscy-ipls.abilene.ucaid.edu ( ) ms ms ms 11 dnvr-kscy.abilene.ucaid.edu ( ) ms ms ms 12 sttl-dnvr.abilene.ucaid.edu ( ) ms ms ms 13 TRANSPAC-PWAVE.pnw-gigapop.net ( ) ms ms ms ( ) ms ms ms 15 perf.jp.apan.net ( ) ms ms ms

24 June 2002 Andy Germain Typical ramp up Client connecting to perf.jp.apan.net, TCP port 5002 TCP window size: 1000 KByte (WARNING: requested 500 KByte) [ 3] local port 3623 connected with port 5002 [ ID] Interval Transfer Bandwidth [ 3] sec 808 KBytes 4.4 Mbits/sec [ 3] sec 856 KBytes 12.1 Mbits/sec [ 3] sec 1.4 MBytes 12.3 Mbits/sec [ 3] sec 1.7 MBytes 12.6 Mbits/sec [ 3] sec 2.0 MBytes 14.8 Mbits/sec [ 3] sec 1.7 MBytes 15.2 Mbits/sec [ 3] sec 1.7 MBytes 15.2 Mbits/sec [ 3] sec 2.8 MBytes 21.1 Mbits/sec [ 3] sec 2.5 MBytes 22.4 Mbits/sec [ 3] sec 2.8 MBytes 23.8 Mbits/sec