1. 30 May 2001 Campus Measurement Matt Zekauskas, matt@internet2.edu Internet2 Campus Workshop Atlanta, GA

2. Campus Measurement30-May-2001 2 Outline Existing measurement tools & projects A sample performance problem Vision for infrastructure to solve problems Steps campuses can take today

3. Campus Measurement30-May-2001 3 My Bias I once ran a (corporate) campus network My recent focus: measurements Most recently: end-to-end performance I have been helping solve some wide- area performance problems Caveat: I haven’t been doing the fixing (My view is from the center)

4. Campus Measurement30-May-2001 4 We Need Your Help I have probably missed something Give us the benefit of your experience fixing problems operating a network …also join a working group!

5. Campus Measurement30-May-2001 5 Measurement Goals Solving performance problems Network operations Network engineering Network research Operational data performance, flows, anomalies Network characterization how used? load response? SLS?

6. Campus Measurement30-May-2001 6 Measurements from the Center Active Measurement within Abilene Measurement using entire Internet2 infra. Passive SNMP stats (esp. core Abilene links) “IOS” stats (for QoS) Characterization of traffic (on the way) –Netflow; OCxMON

7. Campus Measurement30-May-2001 7 Measurement Projects Surveyor (one-way delay, loss, routing) www.advanced.org/surveyor On many Internet2 campuses (70 sites) Abilene presence AMP (round-trip delay, loss, routing) moat.nlanr.net/AMP At even more Internet2 campuses (120 sites) PMA (passive, packet traces) moat.nlanr.net/PMA 1 min, 8 times a day, ~13 sites

8. Campus Measurement30-May-2001 8 Measurement Projects PingER (round-trip delay, routing) http://www-iepm.slac.stanford.edu/pinger/ Long term data from a few locations to many High-energy physics focus NIMI http://www.ncne.nlanr.net/nimi/ Designed to be platform for experiments Undergoing some redesign/revitalization ~ 60 sites?

9. Campus Measurement30-May-2001 9 Usefulness AMP, Surveyor, Pinger If at your campus, a view from your campus If at destination, a view of destination Look for campus connected to same gigaPoP if not at local or destination [“Phase 0” measurement points for e2eperf] Routing, congestion problems

10. Campus Measurement30-May-2001 10 Usefulness PMA If at your campus, can look at traces for anomalies Not as useful for on-demand debugging (but don’t ignore ability to take traces)

11. Campus Measurement30-May-2001 11 Surveyor on One Slide Continuous measurement One-way delay and loss 1/sec on Poisson Schedule 12 Byte UDP packets Traceroutes at 1/600 sec 72 Machines http://hartman.advanced.org/IPPMApplet/report/ Report.html -- Java, close to real-time http://ippm-db.advanced.org/plots/ -- static

12. Campus Measurement30-May-2001 12

13. Campus Measurement30-May-2001 13

14. Campus Measurement30-May-2001 14 AMP Like Surveyor, but Round-trip latency instead of one-way –Easier to deploy Working on more comprehensive set of “alarms” Potentially more available

15. Campus Measurement30-May-2001 15

16. Campus Measurement30-May-2001 16 An “Application-Level” Example Pioneer http://pelle.internet2.edu:8080/pioneer/ Synthesis of existing infrastructure Focus: video conferencing tests Goal: use this to tell if video likely to work

17. Campus Measurement30-May-2001 17 Abilene Abilene goal to be an exemplar Measurements open Tests possible to router nodes Web-mediated on-demand measurements Throughput tests routinely through backbone …as well as existing utilization, etc.

18. Campus Measurement30-May-2001 18 Active within Abilene Each Router Node has a PC Now 10 of 11 are OC3-ATM attached missing: Houston No GPS working towards GPS within CDMA solution

19. Campus Measurement30-May-2001 19 Ad-hoc Active on Abilene With OC-3, can do moderate throughput testing (e.g., iperf UDP & TCP). ~90 Mbps Adding on-demand tests in support of performance debugging Contact me (matt@internet2.edu) if you want to perform an ad-hoc test

20. Campus Measurement30-May-2001 20 Passive - Utilization The Abilene NOC takes Packets in,out Bytes in,out Drops/Errors..for all interfaces, publishes internal links & peering points (at 5 min intervals)..via SNMP polling – every 3 sec http://hydra.uits.iu.edu/~abilene/traffic/

21. Campus Measurement30-May-2001 21

22. Campus Measurement30-May-2001 22

23. Campus Measurement30-May-2001 23 Passive – Characterization Some sparse via NLANR/MOAT http://moat.nlanr.net/PMA/ Starting some NetFlow measurements QoS AS-AS information for K-20 & ITN Intend to do some characterization

24. Campus Measurement30-May-2001 24 Others via Abilene NOC BGP Peering MSDP (multicast source discovery) logging See: http://www.abilene.iu.edu/ -> Operational Status

25. Campus Measurement30-May-2001 25 Multicast-specific Multicast measurements Not fully understood Debugging is an art Tools Mtrace ‘sdr’ announcements in backbone Mhealth, Mantra via UCSB http://www.nmsl.cs.ucsb.edu/ http://www.cs.ucsb.edu/~almeroth/

26. Campus Measurement30-May-2001 26 JPL/Caltech – GSFC The situation Using Abilene Tuned hosts Things work locally Therefore it MUST be Abilene Tests show good flows router-router Intermediate tests point towards CA Bad fiber connection!

27. Campus Measurement30-May-2001 27 Vision I Ongoing monitoring to test major elements, and (some, important) end-to- end paths. Elements: gigaPoP links, peering, … Utilization Delay Loss Occasional throughput Multicast connectivity

28. Campus Measurement30-May-2001 28 Vision II There are many more paths end to end than can be monitored. Diagnostic tools available on-demand (with authorization) Show routes Perform flow tests (perhaps app tests) Parse/debug flows (a-la tcpdump or OCXmon with heuristic tools)

29. Campus Measurement30-May-2001 29 For TCP (and Streaming) Eliminating loss is the goal Focus on noncongestive losses TCP: 100 Mbit Ethernet coast-to-coast: Full size packets… need 10 -6 P loss [Mathis] Less than 1 loss every 83 seconds http://www.psc.edu/~mathis/papers/JTechs200105/ GigE/655: 10 -8, 1 loss every 497 seconds

30. Campus Measurement30-May-2001 30 Enabling Divide & Conquer and Ongoing Monitoring Backbone 1 GigaPoP A Campus GigaPoP B Backbone 2 Wall Jack P P

31. Campus Measurement30-May-2001 31 Some Commercial Tools Caveat: only a partial list, give me more! Spirent (nee Netcom/Adtech): working on a box for ‘end-to-end’ measurements SmartBits: test at low & high rates, QoS; test components or end-to-end path NetIQ: Chariot/Pegasus Ixia (like SmartBits/Spirent); Agilent Brix Networks (like Surveyor, for ‘QoS’)

32. Campus Measurement30-May-2001 32 Some Noncommercial Tools Iperf: dast.nlanr.net/Projects/iperf See also http://www-itg.lbl.gov/nettest/ Flowscan: http://www.caida.org/tools/utilities/flowscan/ http://net.doit.wisc.edu/~plonka/FlowScan/ SLAC’s traceroute perl script: http://www.slac.stanford.edu/comp/net/wan-mon/traceroute- srv.html One large list: http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html

33. Campus Measurement30-May-2001 33 What You Can Do Export SNMP data I can keep an “internet2 list”, would like it to be public [Current Measurement WG project] Monitor loss as well as throughput Performance test point at campus edge Netperf or iperf, so can be from anywhere Traceroute “looking glass” Commercial (e.g., NetIQ) complements I’m willing to keep a master list [MWG project] Portable performance test point

34. Campus Measurement30-May-2001 34 For TCP Tuning Keep an eye out for Web100: http://www.web100.org/ NCNE Tuning Page: http://www.psc.edu/networking/perf_tune.html http://www.ncne.nlanr.net/research/tcp/

35. Campus Measurement30-May-2001 35 What You Can Do If you have a Cisco router at your edge, use NetFlow and cflowd + FlowScan to see your traffic characteristics RTFM / RMON probes See also Joe St.Sauver’s presentation from the last “Joint Techs” meeting: http://www.ncne.nlanr.net/training/techs/2001/0514/presentations /200105-sauver1.html

36. Campus Measurement30-May-2001 36 A Summer Project Measurement box at edge Spend month or two with mobile box, checking throughput/loss/.. from every point. Eliminate noncongestive losses Develop a baseline to get a complete picture of the campus: map the campus networks

37. Campus Measurement30-May-2001 37 NTP everywhere! If GPS, get good NTP distribution Allow correlation among campuses

38. Campus Measurement30-May-2001 38 Plug: Internet2 Measurement Working Group Activities Measurement architecture Encourage common Measurements, tools Parameters Reporting Work with (at least) management, QoS, multicast; End-to-end Performance Initiative

39. Campus Measurement30-May-2001 39 Contact Information Matt Zekauskas, matt@internet2.edu Measurements Working Group http://www.internet2.edu/measurement/ End-to-end interest list listserv@internet2.edu subscribe e2e-interest

40. Campus Measurement30-May-2001 40 (Some) URLs Http://www.internet2.edu/measurement/ http://www.advanced.org/surveyor/ http://moat.nlanr.net/ http://dast.nlanr.net/ http://www.ncne.nlanr.net/ http://www.ncne.org/ http://www.caida.org/ http://www.web100.org/ http://www.auckland.ac.nz/net/Internet/rtfm/ http://www.slac.stanford.edu/xorg/icfa/ntf/home.html http://www.merit.edu/ipma/

41. www.internet2.edu