1. Network Path and Application Diagnostics
Matt Mathis
John Heffner
Ragu Reddy
7/17/06
PathDiag ppt
(Corrected)

2. Outline NPAD/Pathdiag - Why should you care?
What are the real performance problems?
Automatic diagnosis
Deployment

3. NPAD/Pathdiag - Why should you care?
One click automatic performance diagnosis
Designed for (non-expert) end users
Accurate end-systems and last mile diagnosis
Eliminate most false pass results
Accurate distinction between host and path flaws
Accurate and specific identification of most flaws
Basic networking tutorial info
Help the end user understand the problem
Help train 1st tier support (sysadmin or netadmin)
Backup documentation for support escalation
Empower the user to get it fixed
The same reports for users and admins

4. Recalibrate user expectations
Long history of very poor network performance
Users do not know what to expect
Users have become completely numb
Goal for new baseline user expectations:
1 Gigabyte in less than 2 minutes (~67 Mb/s)
Everyone should be able to reach these rates by default
People who can’t should know why or be very angry

5. The Wizard Gap

6. The Wizard Gap Updated Experts have topped out end systems & links
10 Gb/s NIC bottleneck
40 Gb/s “link” bandwidth (striped)
Median I2 bulk rate is 3 Mbit/s
See
Current Gap is about 3000:1
Closing the first factor of 30 should now be “easy”

7. TCP tuning requires expert knowledge
By design TCP/IP hides the ‘net from upper layers
TCP/IP provides basic reliable data delivery
The “hour glass” between applications and networks
This is a good thing, because it allows:
Invisible recovery from data loss, etc
Old applications to use new networks
New application to use old networks
But then (nearly) all problems have the same symptom
Less than expected performance
The details are hidden from nearly everyone

8. TCP tuning is painful debugging
All problems reduce performance
But the specific symptoms are hidden
Any one problem can prevent good performance
Completely masking all other problems
Trying to fix the weakest link of an invisible chain
General tendency is to guess and “fix” random parts
Repairs are sometimes “random walks”
Repair one problem at time at best
The solution is to instrument TCP

9. The Web100 project Instrumentation and autotuning for TCP New insight
TCP has the ideal diagnostic vantage point
TCP-ESTATS-MIB now past IETF WG last-call
Will be a standard track RFC soon
Prototypes for Linux ( and Windows Vista
TCP Autotuning
Automatically adjusts TCP buffers
Linux default maximum window size is 4 M Bytes
Announced for Vista - details unknown
New insight
Nearly all symptoms scale with round trip time

10. Nearly all symptoms scale with RTT
For example
TCP Buffer Space, Network loss and reordering, etc
On a short path TCP can compensate for the flaw
Local Client to Server: all applications work
Including all standard diagnostics
Remote Client to Server: all applications fail
Leading to faulty implication of other components
Talk through the diagram: typical campus, clean backbone, S, LC and RC

11. The confounded problems
For nearly all network flaws
The only symptom is reduced performance
But the reduction is scaled by RTT
Therefore, flaws are undetectable on short paths
False pass for even the best conventional diagnostics
Leads to faulty inductive reasoning about flaw locations
Diagnosis often relies on tomography and complicated inference techniques
This is the real end-to-end problem
Ask for questions

12. The NPAD solution: For applications (and upper layers)
Bench test over an (emulated) ideal long path
Topic of a future talk
“Pathdiag” tests short path sections to localize a flaw
Use Web100 to collect detailed statistics
Loss, delay, queuing properties, etc
Use models to extrapolate results to the full path
Assume that the rest of the path is ideal
You have to specify the end-to-end performance goal
Data rate and RTT
Pass/Fail on the basis of the extrapolated performance

13. Deploy as a Diagnostic Server
Use pathdiag in a Diagnostic Server (DS)
Specify End to End target performance
From server (S) to client (C) (RTT and data rate)
Measure the performance from DS to C
Use Web100 in the DS to collect detailed statistics
On both the path and client
Extrapolate performance assuming ideal backbone
Pass/Fail on the basis of extrapolated performance
Reversed data from eariler slide

14. Demo
Laptop
PSC

15. Key NPAD/pathdiag features
Results are intended for end-users
Provides a list of specific items to be corrected
Failed tests are showstoppers for fast apps
Includes explanations and tutorial information
Clear differentiation between client and path problems
Accurate escalation to network or system admins
The reports are public and can be viewed by either
Coverage for a majority of OS and last-mile network flaws
Most of the remaining flaws can be detected with pathdiag in the client or traceroute
Eliminates nearly all(?) false pass results

16. More features Tests becomes more sensitive as the path gets shorter
Conventional diagnostics become less sensitive
Depending on models, perhaps too sensitive
New problem is false fail (e.g. queue space tests)
Flaws no longer completely mask other flaws
A single test often detects several flaws
E.g. find both OS and network flaws in the same test
They can be repaired concurrently
Archived DS results include raw web100 data
Can reprocess with updated reporting SW
New reports from old data
Critical feedback for the NPAD project
We really want to collect “interesting” failures

17. NPAD/pathdiag deployment
Why should a campus networking organization care?
“Zero effort” solution to miss-tuned end-systems
Accurate reports of real problems
You have the same view as the user
Saves time when there really is a problem
You can document reality for management
Suggestion:
require pathdiag reports for all performance problems

18. What about impact of the test traffic?
NPAD/pathdiag is single threaded
Only one test at a time
Same load as any well tuned TCP application
Protected by TCP “fairness”
Large flows are generally “softer” than small flows
Large flows are easily disturbed by small flows

19. Impact Automatically diagnose first level problems
Easily expose all path bottlenecks that limit performance to less than 100 Mb/s
Easily expose all end-system/OS problems that limit performance to less than 100 Mb/s
(Will become moot as autotuning is deployed)
Empower the users to apply the proper motivation
Still need to recalibrate user expectations
Less than 1 gigabyte / 2 minutes is too slow
Many paths should support 5 gigabytes/minute
Less than 1 Gb/s

20.

Download and install
User documentation:
Follow the link to “Installing a Server”
Easily customized with a site specific skin
Designed to be easily upgraded with new releases
Roughly every 2 months
Improving reports through ongoing field experience
Drops into existing NDT servers
Plans for future integration
Enjoy!

21. Backup slides

22. Blast from the past
Same base algorithm as “Windowed Ping” [Mathis, INET’94]
Aka “mping”
See
Killer diagnostic in use at PSC in the early 90s
Stopped working with the advent of “fast path” routers
Use a simple fixed window protocol
Scan window size in 1 second steps
Measure data rate, loss rate, RTT, etc as window changes