Presentation is loading. Please wait.

Presentation is loading. Please wait.

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard.

Published byMartin Gregory Modified over 9 years ago

Similar presentations

Presentation on theme: "CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard."— Presentation transcript:

1 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/net/

2 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Measurements (1) uLatency uRound trip times measured using Request-Response UDP frames uLatency as a function of frame size Slope gives sum of individual data transfer rates end-to-end Mem copy + pci + Gig Ethernet + pci + mem copy Intercept indicates processing times + HW latencies uHistograms of ‘singleton’ measurements uTells us about: Behavior of the IP stack The way the HW operates Performance of the LAN / MAN / WAN

3 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Measurements (2) uUDP Throughput uSend a controlled stream of UDP frames spaced at regular intervals uVary the frame size and the frame transmit spacing uAt the receiver record The time of first and last frames received The number packets received, the number lost, number out of order The received inter-packet spacing is histogramed The time each packet is received  provides packet loss pattern CPU load, Number of interrupts uUse the Pentium CPU cycle counter for times and delay Few lines of user code uTells us about: Behavior of the IP stack The way the HW operates Capacity and Available throughput of the LAN / MAN / WAN

4 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Works uHow it works: The ‘Zero_stats’ request also provides an interlock against concurrent tests. Zero stats OK done ●●● Get remote statistics Send statistics Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time Signal end of test OK done n bytes Number of packets Wait time time 

5 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Latency: Tests B2B PCs with Gigabit Ethernet uNIC SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Latency low & well behaved Latency Slope 0.0252 us/byte Expect: PCI0.00758 GigE0.008 PCI0.00758 0.0236 us/byte www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt

6 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Latency: Tests B2B PCs with Gigabit Ethernet uNIC Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 Latency high but well behaved Indicates Interrupt coalescence Slope 0.0187 us/byte Expect: PCI0.00188 GigE0.008 PCI0.00188 0.0118 us/byte

7 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:32 bit 33 MHz 100Mbit NICs RedHat 5.2 Kernel 2.2.14 LAN 2 PC B2B FWHM ~3  s No change with packet size Man – UCL LAN – MAN – SuperJANET4 – MAN – LAN London Man at 130Mbit on ATM 155Mbit links Latency: Histograms

8 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: Intel Pro/1000 on B2B P4DP6 Max throughput 950Mbit/s Some throughput drop for packets >1000 bytes Loss NIC dependent Loss not due to user  Kernel moves Traced to discards in the receiving IP layer ??? Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

9 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: SysKonnect SK-9843 B2B Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Max throughput 690Mbit/s No packet loss Packet loss during drop

10 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester European Topology: NRNs & Geant SuperJANET4 CERN UvA Manc SURFnet

11 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit Throughput on the Production WAN Manc - RAL 570 Mbit/s 91% of the 622 Mbit access link between SuperJANET4 and RAL 1472 bytes propagation ~21  s Manc-UvA (SARA) 750 Mbit/s SJANET4 + Geant + SURFnet Manc – CERN 460 Mbit/s CERN PC had a 32 bit PCI bus

12 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: SLAC - Man SLAC – Manc 470 Mbit/s 75% of the 622 Mbit access link SuperJANET4 peers with ESnet at 622Mbit in NY

13 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Packet Loss: Amsterdam – Chicago Lambda (16b of 18) WS L2 fast->slow L2 slow->fast fast high RTT Cees de Laat & R. H-J Paris Mar 02 BW ~500 Mbit/s Keep 2 Drop 1 L2 Buffers full Packet Number Time  s

14 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Frame Jitter (Nov 01) Interrupt Coalescence Lon MAN ~130 Mbit Via CERN

15 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester TCPstream uTCP Throughput vs : msg size, transmit spacing, n-streams, window size u1 control TCP link n test streams uMemory – memory uOnly 1 process – no threads – no context switching uStreams treated equally not round Robin but next available stream uAt the receiver record The time of first and last message received The amount of data ( CPU load, Number of interrupts ) uUse the Pentium CPU cycle counter for times and delay Few lines of user code uTells us about: Behavior of the IP stack Available TCP throughput of the LAN / MAN / WAN

16 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit TCP on the Production WAN (01) Throughput vs TCP buffer size TCP window sizes in Mbytes calculated from RTT*bandwidth

17 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit TCP on the Production WAN (10) Throughput vs n-streams Default buffer size slope = ~25 Mbit/s/stream up to 9 streams then 15 Mbit/s/stream With larger buffers rate of increase per stream is larger Plateaus at about 7 streams giving a total throughput of ~400 Mbit/s

18 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester More Information Some URLs  UDPmon / TCPstream kit + writeup http://www.hep.man.ac.uk/~rich/net  ATLAS Investigation of the Performance of 100Mbit and Gigabit Ethernet Components Using Raw Ethernet Frames http:// www.hep.man.ac.uk/~rich/atlas/atlas_net_note_draft5.pdf  DataGrid WP7 Networking: http://www.gridpp.ac.uk/wp7/index.html  Motherboard and NIC Tests: www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt  PPNCG Home page with Stop Press: http://ppncg.rl.ac.uk/  PPNCG Page for monitoring Grid, HEP & Astronomy Sites http://icfamon.dl.ac.uk/ppncg/astronomy.html  IEPM PingER home site: http://www-iepm.slac.stanford.edu/  IEPM-BW site: http://www-iepm.slac.stanford.edu/bw

19 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester The Measurements (4) uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC Gigabit Ethernet Probe CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display

20 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 SK300 1400 bytes sent Wait 100 us ~8 us for send or receive Gigabit Ethernet frame 11.6  s Send PCI Receive PCI Send data mem  NIC Receive data NIC  mem

21 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 SK301 1400 bytes sent Wait 20 us Sk303 1400 bytes sent Wait 10 us Frames are back-to-back Can drive at line speed Cannot go any faster ! Gig Eth frames back to back

22 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 IT66M212 1400 bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv IT66M212 1400 bytes sent Wait 11 us Packets lost Action of pause packet?

23 CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP4010 1400 bytes sent Wait 8 us ~5.14us on send PCI bus PCI bus ~68% occupancy ~ 2 us on PCI for data recv Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 3-5: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

Download ppt "CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard."

Similar presentations

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot
IGRID2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets 256-500 Mbit/s on the production network.

IGRID2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.
MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.
DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)
Fundamentals of Computer Networks ECE 478/578

Fundamentals of Computer Networks ECE 478/578
CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/

CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester
ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets 256-500 Mbit/s on the production network.

ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.
JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.
ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.

ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.
Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.
Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.
GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.

GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.
CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of.

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of.
CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.
Astronomy Sysman Meeting 29/30 April 02 R. Hughes-Jones Manchester 1 Introduction to the PPNCG Networking for the PPARC Community  Introduction to the.

Astronomy Sysman Meeting 29/30 April 02 R. Hughes-Jones Manchester 1 Introduction to the PPNCG Networking for the PPARC Community  Introduction to the.
IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.

IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.
DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)
JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.
T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then.

T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester then.

Similar presentations

Ads by Google