Global Observatory for Advanced Network Operations APAN Hawaii meeting, January 2004 Yoshinori Kitatsuji, Jin Tanaka & Kazunori Konishi APAN Tokyo XP

Outline Background Lessons learned from high performance experiments Comparison between APAN JP NOC pages and Abilene NOC/Observatory Discussion on Global Observatory

Background High performance demonstration become to be held constantly through a year. Some demonstrations begins to be done without notifications. Know-how for provisioning and troubleshoot has been accumulated by network engineers. Operators struggle with deployment of new service and technology. Measurement for new items such as IGP stability Performance experiment such as SLAC, etc. Conventional services should be maintained Needs to share high technology and know-how to operators toward new services with higher technology. Introduction of new tools with advanced functions enables operators to tackle new service easily. View of collected data leads to the smooth and stable operation at a result.

Experiences of High Performance Experiment Support Osaka University HD over IPv6 transmission to NPACI, SC2003 and OptIPuter Osaka Univ., JGNv6, Tokyo XP, Abilene and SCInet/SDSC, 100+ Mbit/s University of Washington HD over IP transmission at APAN Busan meeting, Aug 2003 Tyco DC, IEEAF-WIDE, Tokyo XP, JGN, Genkai XP and KOREN, 200+ Mbit/s National Institute of Advanced Industrial Science and Technology Japan SC2003 Bandwidth Challenge Tokyo XP/SINET Abilene and SCInet, 3.8Gbit/s University Tokyo SC2003 Bandwidth Challenge Tokyo Univ., Tokyo XP, WIDE, Tyco DC(Seattle), NTT Communications, Abilene, SCInet, 7.8Gbit/s SURFNET/TransLIGHT/APAN Tokyo XP Performance test has just started. Bottleneck is 1Gbit/s between Japan and Netherlands.

Lessons Learned from Experiments Importance of understanding the availability on the path Performance test should be performed hop by hop. Resource sharing technique is required. Effects of multiple TCP connections and rate control at source stations. Dynamic rate control mechanism is one of the next key technologies items. VLAN ID assignment policy applied to HPRENs connections is not discussed yet. It is anticipated that connections among three continentals accelerates VLAN ID consumption. A VLAN for Tokyo XP/TransLIGHT/SURFnet was already been assigned. Importance of management of used ports and assigned VLAN was pointed out Much time was spent to establish the VLAN between Tokyo and Netherlands. It’s seems that few people can administrate VLAN and networks connected are not clarified.

Comparison between APAN-JP NOC page and Abilene NOC/Observatory-1 Network Monitoring Function APAN-JP NOC Abilene NOC/Observatory Our Impression Useful for Trouble shooting Useful for operation View current alerts on network -Alertmon Network Monitor ☆☆☆☆ View geographical network usage data -Animated Traffic Map ☆☆ Show traffic graph of router interface All Links Traffic Graphs by MRTG RRD Connector Graphs (5- minute avg) ☆☆ Show high-performance traffic graph of router interface er-interfaces/ er-interfaces/ & Traffic Report by RRDTraffic Report by RRD RRD Connector Graphs (1- minute avg) ☆☆☆ Aggregate traffic graphs of the whole network or routers -Aggregate Traffic on Abilene ☆☆☆ View router CPU utilization CPU Utilization by MRTG Contained in Visible Network ToolsetVisible Network Toolset ☆☆☆☆☆ View router Memory utilization Memory Utilization by MRTG Contained in Visible Network ToolsetVisible Network Toolset ☆☆ View router Temperature measurement Temperature Measure by MRTG Contained in Visible Network ToolsetVisible Network Toolset ☆☆☆

Comparison between APAN-JP NOC page and Abilene NOC/Observatory-2 Network Monitoring Function APAN-JP NOC Abilene NOC/Observatory Our Impression Useful for Trouble shooting Useful for operation Collect & analyze XML Network data -Visible Network Toolset ☆☆☆ Archive text reports of daily aggregate statistics -Daily Connector Statistics ☆☆ View recent outage and availability reports -Weekly Reports ☆☆☆☆☆ View daily reports generated from the Netflow data collected from routers -Netflow Reports ☆☆☆☆☆ View errors and discards on the link Error Packets Error Packets & Discard FrameDiscard FrameAnimated Traffic Map ☆☆ Show daily snapshots of the BGP table and BGP events -Ixia IxTraffic BGP tables ☆☆ Show time-series graph of the number of BGP routes per peer Number of BGP routes - ☆☆☆☆☆ View the VLAN information VLAN View- ☆☆☆☆☆

View traffic of the network displayed on the map Excels in grasping the traffic Usage of the whole network. APAN JP is considering the introduction of this tool in the near future. Requirements Hardware: Traffic Graph Server (Ready) Library: GD Library (Ready) Software: Collect and analyze XML Network data Useful for remote trouble shooting from the other networks. Requirements Know-how of JUNOScript Hardware: WWW Server with huge hard disk (Ready) Software: View daily reports generated from the Netflow data collected from routers Useful for routing troubleshoot, traffic analysis and DoS detection. APAN JP is considering the realization with cflow. Requirements Software: cflowd Router configuration

Comparison between APAN-JP NOC page and Abilene NOC/Observatory-3 Operation Tools Function APAN-JP NOC Abilene NOC/Observatory Our Impression Useful for Trouble shooting Useful for operation Get the result of operational commands to a router from the web page Traceroute ServiceCore Node Router Proxy ☆☆☆ Visualize the tree structure of a mroute -Multicast Route Viewer ☆☆ Operation manual searching engine A necessary manual can be searched with a keyword from about 150 manuals (secure page) ? ☆☆☆☆☆ Show the DC power usage polled from power controllers - Rack Power Draws ☆☆☆☆☆ Ticket system for managing problem and maintenance Request TrackerRequest Tracker (secure page)Helpdesk System ☆☆☆☆ Check the allocation of IP address Automatically check by icmp (secure page) ? ☆☆☆

Get the result of operational commands to a router from the web page It allows APAN participants to execute “show” commands on multiple routers in APAN Tokyo XP via a web interface. Enable to conduct more complete diagnosis of a problem without contacting NOC. Requirements Hardware: WWW server (Ready) Software: routeproxy Router configuration Show the DC power usage polled from power controllers Enable to show an overview of power usage in rack. No need to measure the power usage of each rack on the regular schedule. Requirements Hardware: WWW Server (Ready), SNMP responseable AC/DC power controller Software: NET-SNMP (Ready)

Comparison between APAN-JP NOC page and Abilene NOC/Observatory-4 Advanced Services Function APAN-JP NOC Abilene NOC/Observatory Our Impression Useful for Trouble shooting Useful for operation Monitor various aspects of multicast at the router level - MANTRA Multicast Monitoring ☆☆☆☆☆ Test multicast connectivity between multiple hosts -Multicast Beacon Server ☆☆☆ Display the amount of IPv6 traffic to and from the tunnel router -IPv6 Traffic Graphs ☆☆ Determine the Provider Independent (PI) address based on the location - Provider Independent Addressing ☆☆☆

Test of multicast connectivity between multiple hosts Provides measurement data for the current multicast traffic in a group by RTP. Requirements Hardware: WWW Server Perl Module: Net-RTP-0.4, Net::Domain Software: Multicast Beacon

Comparison between APAN-JP NOC page and Abilene NOC/Observatory-5 Observatory Function APAN-JP NOC Abilene NOC/Observatory Our Impression Useful for Trouble shooting Useful for operation Provide one way latency information for each link -Abilene Latency Tables ☆☆☆☆☆ Provide one way latency statistics - Abilene Worst Ten Performing Latency Measurements ☆☆ Provide Throughput information for each link -Abilene Throughput Tables ☆☆☆ Provide Throughput statistics - Abilene Worst Ten Performing Throughput Measurements ☆☆ Provide Router Node Data - Connection Technologies Table ☆☆ NTP Service Stratum 2 Server ☆☆☆☆

Provide one way latency information for each link Network performance of each link can be checked Vital when trouble-shooting delay-related problems Requirements NTP on endpoints Hardware: FreeBSD or Linux Server with OWAMP (One-way Active Measurement Protocol) OWAMP: Provide Throughput Information for each link Draw the throughput statistics and ranking from the data which Iperf reported. Throughput (Mbps) / Jitter (%) / Packet Loss (%), Date & Time Requrements Hardware: WWW Server (Ready) Software: Iperl

Discussion: Global Observatory for Advanced Researches over HPRENs We would like to work for a program challenge that supports the collection and dissemination of network data over HPRENs. With observatory servers such as Abilene Observatory, NOC provides network engineers a view of the operational data associated with a global-scale network, and also research communities the fundamental properties of basic network protocols. With provide rack space and fundamental operation service by NOC, Advanced research projects can collect data to be opened to the other researchers. NOC will provide an account in measurement devices and the ways to export data. NOC manages resources for the operations, advanced researches and the conventional services.

Issues towards global operations and/or sharing measurement data among HPRENs Non-uniform networks are connected. High load performance test should be considered. Long Distance Long latency. Time synchronization accuracy tends to be NTP level. Time difference Burden of Troubleshoot depends on how much management information are opened to the community. Troubleshoot escalation level for nighttime operators should be specified at every site. Scalability Automated operations of the tools will be more important, because the number of measurement or monitoring tools will be increased. Software update methods and software version management will also be more troublesome.

Requirements for Global Observatory Basic data collection. The comparison between APAN-JP NOC and Abilene NOC/Observatory is described in the front pages. Advanced features to be developed for advanced researchers. TCP performance test scheduler High resolution flow based traffic grapher for TCP/UDP performance test Preparations to accept an applications of co-located project Observatory Servers Cooperate with Abilene Observatory project. Preparation for Co-Located project Available rack space & power and account in related routers and switches. Application: form, organizing evaluation team. Drafting Acceptable Co-location Policy (Example) Accounts should be provided to NOC operators and engineers. Permission of exporting date to specified researchers.

TCP Performance Test Scheduler Increasing TCP high performance transmission experiment from multiple group or organization. SLAC, CRL, NASA, Indiana University, Demonstrations Experiments are getting congested. Some experiments require original TCP stack Dedicated machine is required for this kind of experiments. Traffic generated by each experiment can easily fill up the link between the end hosts. Multiple experiments running at the same time degrade the performance.

Architecture of TCP Performance Scheduler R R R R R R R R Performance machine with original TCP stack (iperf, netperf, etc) Scheduler Host A Host B 1.At beginning, Scheduler collects topology data by SNMP and routing information base from routers by OSPF SLAs and recognizes topology tree. 2.Run iperf wrapper program to send a ticket to Scheduler for running the performance test. 3.Scheduler checks the queuing tickets and responds when test can start. At this stage tickets performance time is registered as a reservation. 4.If the start time is accepted, the host responds whether to run or quit. 5.Scheduler update a reservation ticket based on wrapper program response. If wrapper program requests to continue, complete ticket registration. 6.Wrapper program run iperf test. 4. Response whether continue or no 2. & 3. test request and runable time response 1. Collect data to recognize topology tree at Scheduler 5. Register reserving tickets for test in the future or discard Iperf test path 6. Run performance test

High Resolution Flow Base Traffic Grapher Why Flow measurement? Performance test wants traffic viewers shows the traffic variation based on flows at intermediate routers. Data collection by Netflow/Cflowd shows traffic tendency because 1/ /1000 packet sampling analysis is done on high speed routers in stead of full packet checking. It’s not enough! Why High resolutions? It’s easy for users to get average speed of TCP performance test at end stations, but hard to understand variation of speed in short period such as millisecond granularity. Control of variable burst traffic might help to troubleshoot TCP performance degradation. AIST and University of Tokyo was awarded in SC2003 bandwidth challenge. Multiple flows should be measured at the same time over multiple links. Measurement performance should be investigated.

Requirements of High Resolution Flow Base Traffic Grapher Devices will be installed on multiple links User and operator want to compare the performance over multiple paths Time synchronization between the devices. Topology recognition is important. Operation of the multiple devices is the key. On demand measurement To respond for the requests from multiple users without high burden, measurement should be short period (Ex. 10 second or 1 minute graphs) Gigabit ethernet will be ready soon, but SONET/SDH… Development with OC48/OC192MON equipping DAG card sold by Endace.

Architecture of High Resolution Flow Base Traffic Grapher R R R R R R R R Performance machine with original TCP stack (iperf, netperf, etc) Host A Host B Measurement device 1.At beginning, Manager program to control measurement devices collects topology data by SNMP and routing information base from routers by OSPF SLAs. Manager recognizes topology tree. 2.User requests the measurement to Manager via WEB interface by giving start and end of path, port numbers, start time and period. 3.Manager analyzes topology from path information and request Measurement devices to start to measure flows at the specified time. 4.Measurement Devices responds traffic flow graph to Manager. 5.User see the traffic graph page constructed by Manager, via the WEB browser. 2. Request Traffic graphes via http 1. Collect data to recognize topology tree at Scheduler 3. & 4. Request Measuremet device to measure and respond traffic graph 5. Show graphes on browser Iperf test path