1. The Performance Bottleneck Application, Computer, or Network Richard Carlson Internet2 Part 1 Richard Carlson Internet2 Part 1

2. Outline Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next

3. Basic Premise Application’s performance should meet your expectations! If they don’t you should complain!

4. Questions How many times have you said: What’s wrong with the network? Why is the network so slow? Do you have any way to find out? Tools to check local host Tools to check local network Tools to check end-to-end path

5. Underlying Assumption When problems exist, it’s the networks fault!

6. NDT Demo First

7. Simple Network Picture Bob’s Host Network Infrastructure Carol’s Host

8. Switch 1 Switch 2Switch 3 Network Infrastructure R1 R3 R4 R2 R7 R6 R9 R8 R5 Switch 4

9. Possible Bottlenecks Network infrastructure Host computer Application design

10. Network Infrastructure Bottlenecks Links too small Using standard Ethernet instead of FastEthernet Links congested Too many hosts crossing this link Scenic routing End-to-end path is longer than it needs to be Broken equipment Bad NIC, broken wire/cable, cross-talk Administrative restrictions Firewalls, Filters, shapers, restrictors

11. Host Computer Bottlenecks CPU utilization What else is the processor doing? Memory limitations Main memory and network buffers I/O bus speed Getting data into and out of the NIC Disk access speed

12. Application Behavior Bottlenecks Chatty protocol Lots of short messages between peers High reliability protocol Send packet and wait for reply before continuing No run-time tuning options Use only default settings Blaster protocol Ignore congestion control feedback

13. TCP 101 Transmission Control Protocol (TCP) Provides applications with a reliable in-order delivery service The most widely used Internet transport protocol Web, File transfers, email, P2P, Remote login User Datagram Protocol (UDP) Provides applications with an unreliable delivery service RTP, DNS

14. Summary – Part 1 Problems can exist at multiple levels Network infrastructure Host computer Application design Multiple problems can exist at the same time All problems must be found and fixed before things get better

15. Summary – Part 2 Every problem exhibits the same symptom The application performance doesn’t meet the users expectations!

16. Outline Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next

17. Real Life Examples I know what the problem is Bulk transfer with multiple problems

18. Example 1 - SC’04 experience Booth having trouble getting application to run from Amsterdam to Pittsburgh Tests between remote SGI and local PC showed throughput limited to < 20 Mbps Assumption is: PC buffers too small Question: How do we set WinXP send/receive window size

19. SC’04 Determine WinXP info http://www.dslreports.com/drtcp

20. SC’04 Confirm PC settings DrTCP reported 16 MB buffers, but test program still slow, Q: How to confirm? Run test to SC NDT server (PC has Fast Ethernet Connection) Client-to-Server: 90 Mbps Server-to-Client: 95 Mbps PC Send/Recv window size: 16 Mbytes (wscale 8) NDT Send/Recv window Size: 8 Mbytes (wscale 7) Reported TCP RTT: 46.2 msec approximately 600 Kbytes of data in TCP buffer Min window size / RTT: 1.3 Gbps

21. SC’04 Local PC Configured OK No problem found Able to run at line rate Confirmed that PC’s TCP window values were set correctly

22. SC’04 Remote SGI Run test from remote SGI to SC show floor (SGI is Gigabit Ethernet connected). Client-to-Server: 17 Mbps Server-to-Client: 16 Mbps SGI Send/Recv window size: 256 Kbytes (wscale 3) NDT Send/Recv window Size: 8 Mbytes (wscale 7) Reported RTT: 106.7 msec Min window size / RTT: 19 Mbps

23. SC’04 Remote SGI Results Needed to download and compile command line client SGI TCP window is too small to fill transatlantic pipe (19 Mbps max) User reluctant to make changes to SGI network interface from SC show floor NDT client tool allows application to change buffer (setsockopt() function call)

24. SC’04 Remote SGI (tuned) Re-run test from remote SGI to SC show floor. Client-to-Server: 107 Mbps Server-to-Client: 109 Mbps SGI Send/Recv window size: 2 Mbytes (wscale 5) NDT Send/Recv window Size: 8 Mbytes (wscale 7) Reported RTT: 104 msec Min window size / RTT: 153.8 Mbps

25. SC’04 Debugging Results Team spent over 1 hour looking at Win XP config, trying to verify window size Single NDT test verified this in under 30 seconds 10 minutes to download and install NDT client on SGI 15 minutes to discuss options and run client test with set buffer option

26. SC’04 Debugging Results 8 Minutes to find SGI limits and determine maximum allowable window setting (2 MB) Total time 34 minutes to verify problem was with remote SGIs’ TCP send/receive window size Network path verified but Application still performed poorly until it was also tuned

27. Example 2 – SCP file transfer Bob and Carol are collaborating on a project. Bob needs to send a copy of the data (50 MB) to Carol every ½ hour. Bob and Carol are 2,000 miles apart. How long should each transfer take? 5 minutes? 1 minute? 5 seconds?

28. What should we expect? Assumptions: 100 Mbps Fast Ethernet is the slowest link 50 msec round trip time Bob & Carol calculate: 50 MB * 8 = 400 Mbits 400 Mb / 100 Mb/sec = 4 seconds

29. Initial SCP Test Results

30. Initial Test Results This is unacceptable! First look for network infrastructure problem Use NDT tester to examine both hosts

31. Initial NDT testing shows Duplex Mismatch at one end

32. NDT Found Duplex Mismatch Investigating this it is found that the switch port is configured for 100 Mbps Full-Duplex operation. Network administrator corrects configuration and asks for re-test

33. Duplex Mismatch Corrected

34. SCP results after Duplex Mismatch Corrected

35. Intermediate Results Time dropped from 18 minutes to 40 seconds. But our calculations said it should take 4 seconds! 400 Mb / 40 sec = 10 Mbps Why are we limited to 10 Mbps? Are you satisfied with 1/10 th of the possible performance?

36. Default TCP window settings

37. Calculating the Window Size Remember Bob found the round-trip time was 50 msec Calculate window size limit 85.3KB * 8 b/B = 698777 b 698777 b /.050 s = 13.98 Mbps Calculate new window size (100 Mb/s *.050 s) / 8 b/B = 610.3 KB Use 1MB as a minimum

38. Resetting Window Value

39. With TCP windows tuned

40. Steps so far Found and fixed Duplex Mismatch Network Infrastructure problem Found and fixed TCP window values Host configuration problem Are we done yet?

41. SCP results with tuned windows

42. Intermediate Results SCP still runs slower than expected Hint: SCP uses internal buffers Patch available from PSC

43. SCP Results with tuned SCP

44. Final Results Fixed infrastructure problem Fixed host configuration problem Fixed Application configuration problem Achieved target time of 4 seconds to transfer 50 MB file over 2000 miles

45. Why is it hard to Find/Fix Problems? Network infrastructure is complex Network infrastructure is shared Network infrastructure consists of multiple components

46. Shared Infrastructure Other applications accessing the network Remote disk access Automatic email checking Heartbeat facilities Other computers are attached to the closet switch Uplink to campus infrastructure Other users on and off site Uplink from campus to gigapop/backbone

47. Other Network Components DHCP (Dynamic Host Resolution Protocol) At least 2 packets exchanged to configure your host DNS (Domain Name Resolution) At least 2 packets exchanged to translate FQDN into IP address Network Security Devices Intrusion Detection, VPN, Firewall

48. Network Infrastructure Large complex system with potentially many problem areas

49. Why is it hard to Find/Fix Problems? Computers have multiple components Each Operating System (OS) has a unique set of tools to tune the network stack Application Appliances come with few knobs and limited options

50. Computer Components Main CPU (clock speed) Front & Back side bus Main Memory I/O Bus (ATA, SCSI, SATA) Disk (access speed and size)

51. Computer Issues Lots of internal components with multi- tasking OS Lots of tunable TCP/IP parameters that need to be ‘right’ for each possible connection

52. Why is it hard to Find/Fix Problems? Applications depend on default system settings Problems scale with distance More access to remote resources

53. Default System Settings For Linux 2.6.13 there are: 11 tunable IP parameters 45 tunable TCP parameters 148 Web100 variables (TCP MIB) Currently no OS ships with default settings that work well over trans-continental distances Some applications allow run-time setting of some options 30 settable/viewable IP parameters 24 settable/viewable TCP parameters There are no standard ways to set run-time option ‘flags’

54. Application Issues Setting tunable parameters to the ‘right’ value Getting the protocol ‘right’

55. How do you set realistic Expectations? Assume network bandwidth exists or find out what the limits are Local LAN connection Site Access link Monitor the link utilization occasionally Weathermap MRTG graphs Look at your host config/utilization What is the CPU utilization

56. Ethernet, FastEthernet, Gigabit Ethernet 10/100/1000 auto-sensing NICs are common today Most campuses have installed 10/100 switched infrastructure Access network links are currently the limiting factor in most networks Backbone networks are 10 Gigabit/sec

57. Site Access and Backbone Campus access via Regional ‘GigaPoP’ Confirm with campus admin Abilene Backbone 10 Gbps POS links coast-to-coast Other Federal backbone networks Other Commercial network Other institutions, sites, and networks

58. Tools, Tools, Tools Ping Traceroute Iperf Tcpdump Tcptrace BWCTL NDT OWAMP AMP Advisor Thrulay Web100 MonaLisa pathchar NPAD Pathdiag Surveyor Ethereal CoralReef MRTG Skitter Cflowd Cricket Net100

59. Active Measurement Tools Tools that inject packets into the network to measure some value Available Bandwidth Delay/Jitter Loss Requires bi-directional traffic or synchronized hosts

60. Passive Measurement Tools Tools that monitor existing traffic on the network and extract some information Bandwidth used Jitter Loss rate May generate some privacy and/or security concerns

61. Abilene Weather Map

62. MRTG Graphs

63. Windows XP Performance

64. Outline Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next

65. Focus on 3 tools Existing NDT tool Allows users to test network path for a limited number of common problems Existing NPAD tool Allows users to test local network infrastructure while simulating a long path Emerging PerfSonar tool Allows users to retrieve network path data from major national and international REN network

66. Network Diagnostic Tool (NDT) Measure performance to users desktop Identify real problems for real users Network infrastructure is the problem Host tuning issues are the problem Make tool simple to use and understand Make tool useful for users and network administrators

67. NDT user interface Web-based JAVA applet allows testing from any browser Command-line client allows testing from remote login shell

68. NDT test suite Looks for specific problems that affect a large number of users Duplex Mismatch Faulty Cables Bottleneck link capacity Achievable throughput Ethernet duplex setting Congestion on this network path

69. Duplex Mismatch Detection Developing analytical model to describe how network operates (no prior art?) Expanding model to describe UDP and TCP flows Test models in LAN, MAN, and WAN environments NIH/NLM grant funding

70. Four Cases of Duplex Setting FD-FD FD-HD HD-FDHD-HD

71. Bottleneck Link Detection What is the slowest link in the end-2-end path? Monitors packet arrival times using libpacp routine Use TCP dynamics to create packet pairs Quantize results into link type bins (no fractional or bonded links) Cisco URP grant work

72. Normal congestion detection Shared network infrastructures will cause periodic congestion episodes Detect/report when TCP throughput is limited by cross traffic Detect/report when TCP throughput is limited by own traffic

73. Faulty Hardware/Link Detection Detect non-congestive loss due to Faulty NIC/switch interface Bad Cat-5 cable Dirty optical connector Preliminary works shows that it is possible to distinguish between congestive and non-congestive loss

74. Full/Half Link Duplex setting Detect half-duplex link in E2E path Identify when throughput is limited by half- duplex operations Preliminary work shows detection possible when link transitions between blocking states

75. Finding Results of Interest Duplex Mismatch This is a serious error and nothing will work right. Reported on main page and on Statistics page Packet Arrival Order Inferred value based on TCP operation. Reported on Statistics page, (with loss statistics) and order: value on More Details page

76. Finding Results of Interest Packet Loss Rates Calculated value based on TCP operation. Reported on Statistics page, (with out-of- order statistics) and loss: value on More Details page Path Bottleneck Capacity Measured value based on TCP operation. Reported on main page

77. Additional Functions and Features Provide basic tuning information Basic Features Basic configuration file FIFO scheduling of tests Simple server discovery protocol Federation mode support Command line client support Created sourceforge.net project page

78. NPAD/pathdiag A new tool from researchers at Pittsburgh Supercomputer Center Finds problems that affect long network paths Uses Web100-enhanced Linux based server Web based Java client

79. Long Path Problem E2E application performance is dependant on distance between hosts Full size frame time at 100 Mbps Frame = 1500 Bytes Time = 0.12 msec In flight for 1 msec RTT = 8 packets In flight for 70 msec RTT = 583 packets

80. Switch 1 Switch 2 Switch 3 Long Path Problem R1 R3 R4 R2 R7 R6 R9 R8 R5 Switch 4 H1 H2 H3 X 1 msec H1 – H2 70 msec H1 – H3

81. TCP Congestion Avoidance Cut number of packets by ½ Increase by 1 per RTT LAN (RTT=1msec) In flight changes to 4 packets Time to increase back to 8 is 4msec WAN (RTT = 70 msec) In flight changes to 292 packets Time to increase back to 583 is 20.4 seconds

82. PerfSonar – Next Steps in Performance Monitoring New Initiative involving multiple partners ESnet (DOE labs) GEANT (European Research and Education network) Internet2 (Abilene and connectors)

83. PerfSonar – Router stats on a path Demo ESnet tool https://performance.es.net/cgi-bin/perfsonar-trace.cgi Paste output from Traceroute into the window and view the MRTG graphs for the routers in the path Author: Joe Metzger ESnet

84. Traceroute Visualizer

85. The Wizard Gap* * Courtesy of Matt Mathis (PSC)

86. Google it! Enter “tuning tcp” into the google search engine. Top 2 hits are: http://www.psc.edu/networking/perf_tune.html http://www-didc.lbl.gov/TCP-tuning/TCP-tuning.html

87. PSC Tuning Page

88. LBNL Tuning Page

89. Internet2 Land Speed Record Challenge to community to demonstrate how to run fast – long distance flows 2000 record – 751 Mbps over 5,262 km 2005 record- 7.2 Gbps over 30,000 km

90. Conclusions Applications can fully utilize the network All problems have a single symptom All problems must be found and fixed before things get better Some people stop investigating before finding all problems Tools exist, and more are being developed, to make it easier to find problems

91. Extra Material

92. Outline Why there is a problem What can be done to find/fix problems Tools you can use Ramblings on what’s next

93. Introduction Where have we been and where are we headed? Technology and hardware Transport Protocols

94. Basic Assumption The Internet was designed to improve communications between people

95. What does the future hold? Moore’s Law shows no signs of slowing down The original law says the number of transistors on a chip doubles every 18 months Now it simply means that everything gets faster

96. PC Hardware CPU processing power (flops) is increasing Front/back side bus clock rate is increasing Memory size is increasing HD size is increasing too For the past 10 years, every HD I’ve purchased cost $130

97. Scientific Workstation PC or Sparc class computer Fast CPU 1 GB RAM 1 TB disk 10 Gbps NIC Today’s cost ~ $5,000

98. Network Capability LAN networks (includes campus) MAN/RON network WAN network Remember the 80/20 rule

99. Network NIC costs 10 Mbps NICs were $50 - $150 circa 1985 100 Mbps NICS were $50 - $150 circa 1995 1,000 Mbps NICS are $50 - $150 circa 2005 10 Gbps NICs are $1,500 - $2,500 today Note today 10/100/1000 cards are common and 10/100 cards are < $10

100. Ethernet Switches Unmanaged 5 port 10/100 switch ~ $25.00 Unmanaged 5 port 10/100/1000 switch ~ $50 Managed switches have more ports and are more expensive ($150 - $400 per port)

101. Network Infrastructure Campus Regional National International

102. Campus Infrastructure Consists of switches, routers, and cables Limited funds make it hard to upgrade

103. Regional Infrastructure Many states have optical networks Illinois has I-Wire Metro area optical gear is ‘reasonably’ priced Move by some to own fiber Flexible way to cut operating costs, but requires larger up-front investment

104. National Infrastructure Commercial vendors have pulled fiber to major metro areas NLR – n x 10 Gbps Abilene - 1 x 10 Gbps (Qwest core) FedNets - (DoE, DoD, and NASA all run national networks) CA*net – n x 10 Gbps Almost 500 Gbps into SC|05 conference in Seattle

105. International Infrastructure Multiple trans-atlantic 10 Gbps links Multiple trans-pacific 10 Gbps links Gloriad

106. Interesting sidebar China’s demand for copper, aluminum, and steel have caused an increase in theft Man hole covers Street lamps Parking meters Phone cable One possible solution is to replace copper wires with FTTH solutions

107. Transport Protocol TCP Reno has know problems with loss at high speeds Linear growth following packet loss No memory of past achievements TCP research groups are actively working on solutions: HighSpeed-TCP, Scaleable-TCP, Hamilton- TCP, BIC, CUBIC, FAST, UDT, Westwood+ Linux (2.6.13) has run-time support for these stacks

108. What drives prices? Electronic component prices are driven by units produces Try buying a brand NEW i386 CPU Try upgrading your PC’s CPU NIC’s are no different