KEK Network Qi Fazhi

KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS GbE KEK GW 2 x GbE KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch B Netscreen Firewall 2 x GbE 4 x GbE NOOD Switch to other places NOOD Switches to Buildings 4 x GbE 10GbE Tskuba Center 10GbE Other Univ. HEP Lab. GbE 4 x GbE up to: 14 Vlan, Based MAC Static & DHCP(for wireless) IDS GbE

KEK ’ s firewall architecture L2/L3 Switch for outside connections Central L2/L3 Switch Firewall 10GbE 10GbE/GbE 10GbE 2 x GbE Security monitor High Throughput Access Route Regular flows IDS GbE

KEK universities/institutes in Asia-Pacific universities in Japan SLAC, FNAL, BNL, CERN, DESY, SDSC,..... SuperSINET APAN/Transpac APAN

Since SuperSINET’s links to US/EU SuperSINET NII’s router at NewYork OC48 x 4 Gèant Abilene ESnet OC48 GbE ISP ( 10Gbps ) MAN LAN 10GbE Switch POS Channel 10GbE GbE x 3 IEEAF Atlantic SLAC is on ESnet CERN is on GEANT

WAN Performance Measurement and Monitor -- Study and Work at KEK Qi Fazhi

Study WAN Performance Terminology WAN Performance Measurement Tools How to get larger throughput

Terminology / Elements be relationship to WAN Performance Bandwidth –Physical bandwidth, or capacity (C) –Available bandwidth (A) Throughput –Maximum throughput –Achievable throughput Latency / RTT

TERMINOLOGY Latency Packet Loss Bandwidth: the speed that a network element can forward traffic. (is independent of end hosts and protocol type) –Physical/capacity:maximum number of bits per second a network element can transfer(is determined by the slowest network element along the path) –Available:the capacity minus utilization over a given time interval. Throughput: amount of data that is successfully sent from one host to another via a network (may be limited by every component along the path from source host to destination host) –Maximum throughput –Achievable throughput

Network Performance Measurement Tools Iperf/Netperf: tools for measuring end-to-end TCP/UDP performance pipechar: hop-by-hop bottleneck analysis tool pchar: hop-by-hop performance measurement tool traceroute: lists all routers from current host to remote host

How to get larger throughput Decrease Latency / RTT Packet Loss Choose the best pathRouting coordination Larger Throughput TCP Tuning ? Latency / RTT Packet Loss Parallel stream TCP

TCP Tuning Why? –Bandwidth increases –TCP shortage How? –Modify the TCP Windows Size –Modify the System Buffer Size –…… –Some New Protocol: FAST

TCP TCP is adaptive It is constantly trying to go faster It slows down when it detects a loss How much it sends is controlled by windows When it sends is controlled by received ACK’s(or timeouts)

Summary of techniques to maximize TCP WAN throughput congestion window (CWND):TCP uses it to determine how many packets can be sent at one time, The larger the congestion window size, the higher the throughput. TCP “ slow start ” and “ congestion avoidance ” algorithms determine the size of the congestion window The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket (System) For each socket, there is a default value for the buffer size, which can be changed by the program using a system library call just before opening the socket. Kernel enforced maximum buffer size. Why Tuning?

Bandwidth*Delay Product Bandwidth * Delay = number of bytes in flight to fill path The optimal buffer size is twice the bandwidth*delay product of the link( recover from errors ) buffer size = 2 * bandwidth * delay = bandwidth * RTT bandwidth :get it from pipechar or pchar pipecharpchar RTT:get it from Ping Example:ping time is 50 ms ; the end-to-end network consists of all 100 BT Ethernet and OC3 (155 Mbps).then TCP buffers should be.05 sec * (100 Mbits / 8 bits) = 625 KBytes most of today UNIX OS's by default have a maximum TCP buffer size of only 256 KB! So we should do some tuning …… buffer Windows throughput BW*Delay

Important Points About TCP Throughput congestion window : determine how many packets can be sent at one time,the larger the congestion window size, the higher the throughput. the size of the congestion window : slow start ( constantly trying to go faster ) and congestion avoidance ( slows down when it detects a loss ) maximum congestion window : is related to the amount of buffer space that the kernel allocates for each socket buffer size : 2*Bandwidth*Delay Products （ BDP ） = bandwidth * RTT Summary

Routing coordination KEK IHEP Before Change Status –Go through Russia and USA KEK->IHEP: KEK->Sinet->Abilene->RBNet->CSTnet->IHEP IHEP->KEK: IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK –RTT is very large After Change Status –KEK->IHEP: KEK->SINet->APAN-JP->CERnet->NSFC->CSTnet->IHEP –IHEP->KEK (not changed): IHEP->CSTnet-> RBNet ->Abilene ->Sinet ->KEK –For the reason of not changing IHEP->KEK route,the RTT is still large,but there are some changes in the throughput

KEK-IHEP Route Tuning KEK2505 Sinet2907 Abilene APAN_JP 7660 RBnet5568 Dargon_tap 9407 Cernet 4538 NSFC 9406 CSTNet 7497 AS1239 AS9405 AS3356 Level 3 Sprint APAN_JP&APAN_CN Link IHEP3460

Parallel TCP Stream KEK->IHEP One TCP StreamParallel TCP Streams

WAN Performance Measurement/Monitor Target –Packet loss & latency –Throughput Tools Mechanism WORK

Test PC Data Remote WWW Traceroute/Ping iperf netperf HTTP Ping/iperf/netperf Mechanism (Base on www) CMDResult

System Structure WAN Performance Measurement and Monitor WAN Performance MeasurementWAN Performance Monitor Performance data collect Data File Performance data analysis Performance measurement resultPerformance measurement command

The System Interface

Measurement: (RTT & Packet Loss/traceroute & ping)

Traceroute and Ping Result

Traceroute and Ping Result with graph ms

Measurement: Throughput with Iperf

Measurement Result: Throughput with Iperf

Measurement: Throughput with Netperf

Measurement Result: Throughput with Netperf

Monitor: RTT and Packet loss with ping

Monitor Result: RTT and Packet loss with ping

Monitor: Throughput with iperf

Monitor Result: Throughput with iperf

README INSTALL –creat a user account: monitor –cd ~monitor –tar xvf monitor.tar –edit /usr/local/apache/conf/httpd.conf –edit the ~monitor/ping_list.txt and ~monitor/iperf_list.txt –edit the /etc/crontab –

Useful links