CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard.

Slides:

Advertisements

Similar presentations

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot

Advertisements

IGRID2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

Fundamentals of Computer Networks ECE 478/578

CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester

ER2002 Radio Astronomy VLBI Demo. uWeb based demonstration sending VLBI data A controlled stream of UDP packets Mbit/s on the production network.

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.

ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

Astronomy Sysman Meeting 29/30 April 02 R. Hughes-Jones Manchester 1 Introduction to the PPNCG Networking for the PPARC Community  Introduction to the.

IEEE Real Time 2007, Fermilab, 29 April – 4 May R. Hughes-Jones Manchester 1 Using FPGAs to Generate Gigabit Ethernet Data Transfers & The Network Performance.

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.

T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester then.

CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.

R. Hughes-Jones Video_TAG Meeting Oct 1999 Some Studies of H.323 VideoConference Traffic on LANs and WANs Studies done for PPNCG & PIPVIC2 Lab Tests of.

DataTAG Meeting CERN 7-8 May 03 R. Hughes-Jones Manchester 1 High Throughput: Progress and Current Results Lots of people helped: MB-NG team at UCL MB-NG.

EDG WP7 Networking Demonstration uDemonstration sending HEP data CERN to SARA Multiple streams of TCP packets Tuned TCP parameters: ifconfig eth0 txqueuelen.

PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.

© 2006 Open Grid Forum Interactions Between Networks, Protocols & Applications HPCN-RG Richard Hughes-Jones OGF20, Manchester, May 2007,

Slide: 1 Richard Hughes-Jones CHEP2004 Interlaken Sep 04 R. Hughes-Jones Manchester 1 Bringing High-Performance Networking to HEP users Richard Hughes-Jones.

ESLEA Bedfont Lakes Dec 04 Richard Hughes-Jones Network Measurement & Characterisation and the Challenge of SuperComputing SC200x.

CdL was here DataTAG CERN Sep 2002 R. Hughes-Jones Manchester 1 European Topology: NRNs & Geant SuperJANET4 CERN UvA Manc SURFnet RAL.

GridPP Collaboration Meeting May 2002 R. Hughes-Jones Manchester 1 Networking in Under 30 Minutes ! Richard Hughes-Jones, University of Manchester.

July 2000 PPNCG Meeting R. Hughes-Jones Performance Measurements of LANs MANs and SuperJANET III This is PRELIMINARY uBaseline data for Grid development.

MB - NG MB-NG Meeting UCL 17 Jan 02 R. Hughes-Jones Manchester 1 Discussion of Methodology for MPLS QoS & High Performance High throughput Investigations.

02 nd April 03Networkshop Managed Bandwidth Next Generation F. Saka UCL NETSYS (NETwork SYStems centre of excellence)

GGF4 Toronto Feb 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress Mar 02)

13th-14th July 2004 University College London End-user systems: NICs, MotherBoards, TCP Stacks & Applications Richard Hughes-Jones.

Sven Ubik, Petr Žejdl CESNET TNC2008, Brugges, 19 May 2008 Passive monitoring of 10 Gb/s lines with PC hardware.

KEK Network Qi Fazhi KEK SW L2/L3 Switch for outside connections Central L2/L3 Switch A Netscreen Firewall Super Sinet Router 10GbE 2 x GbE IDS.

R. Hughes-Jones ESnet Meeting Kyoto July 2000 Networking for HEP in the UK uIntroduction to the PPNCG uNetwork Topologies uMonitoring Activities uPeering.

Slide: 1 Richard Hughes-Jones e-VLBI Network Meeting 28 Jan 2005 R. Hughes-Jones Manchester 1 TCP/IP Overview & Performance Richard Hughes-Jones The University.

Srihari Makineni & Ravi Iyer Communications Technology Lab

High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.

1 Network Performance Optimisation and Load Balancing Wulf Thannhaeuser.

ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 1 VLBI Data Transfer Tests Recent and Current Work. Richard Hughes-Jones The University.

MB - NG MB-NG Meeting Dec 2001 R. Hughes-Jones Manchester MB – NG SuperJANET4 Development Network SuperJANET4 Production Network Leeds RAL / UKERNA RAL.

ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 1 Protocols DCCP and dccpmon. Richard Hughes-Jones The University of Manchester.

Robin HJ & R. Hughes-Jones Manchester Sep 1999 Gigabit Ethernet in Ptolemy Status Sep 99 : Stars that exist : –GigEChipTranslate between GigEPacket and.

Online-Offsite Connectivity Experiments Catalin Meirosu *, Richard Hughes-Jones ** * CERN and Politehnica University of Bucuresti ** University of Manchester.

PFLDNet Workshop February 2003 R. Hughes-Jones Manchester Some Performance Measurements Gigabit Ethernet NICs & Server Quality Motherboards Richard Hughes-Jones.

Collaboration Meeting, 4 Jul 2006, R. Hughes-Jones Manchester 1 Collaborations in Networking and Protocols HEP and Radio Astronomy Richard Hughes-Jones.

High bit rate tests between Manchester and JIVE Looking at data rates attainable with UDP along with packet loss and reordering statistics Simon Casey,

DataGrid WP7 Meeting Amsterdam Nov 01 R. Hughes-Jones Manchester 1 UDPmon Measuring Throughput with UDP  Send a burst of UDP frames spaced at regular.

Sep. 17, 2002BESIII Review Meeting BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep , 2002.

MB - NG MB-NG Meeting UCL 17 Jan 02 R. Hughes-Jones Manchester 1 Discussion of Methodology for MPLS QoS & High Performance High throughput Investigations.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.

Networks ∙ Services ∙ People Richard-Hughes Jones eduPERT Training Session, Porto A Hands-On Session udpmon for Network Troubleshooting 18/06/2015.

Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 1 vlbi_udp Throughput Performance and Stability. Richard Hughes-Jones The University.

L1/HLT trigger farm Bologna setup 0 By Gianluca Peco INFN Bologna Genève,

Connect communicate collaborate Performance Metrics & Basic Tools Robert Stoy, DFN EGI TF, Madrid September 2013.

Recent experience with PCI-X 2.0 and PCI-E network interfaces and emerging server systems Yang Xia Caltech US LHC Network Working Group October 23, 2006.

Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI Richard Hughes-Jones.

DataGrid WP7 Meeting Jan 2002 R. Hughes-Jones Manchester Initial Performance Measurements Gigabit Ethernet NICs 64 bit PCI Motherboards (Work in progress)

MB MPLS MPLS Technical Meeting Sep 2001 R. Hughes-Jones Manchester SuperJANET Development Network Testbed – Cisco GSR SuperJANET4 C-PoP – Cisco GSR.

R. Hughes-Jones Manchester

Networking for grid Network capacity Network throughput

Data Transfer Node Performance GÉANT & AENEAS

Mar 2001 ATLAS T2UK Meeting R. Hughes-Jones

MB-NG Review High Performance Network Demonstration 21 April 2004

MB – NG SuperJANET4 Development Network

Presentation transcript:

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon and TCPstream Tools to understand Network Performance Richard Hughes-Jones The University of Manchester

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Measurements (1) uLatency uRound trip times measured using Request-Response UDP frames uLatency as a function of frame size Slope gives sum of individual data transfer rates end-to-end Mem copy + pci + Gig Ethernet + pci + mem copy Intercept indicates processing times + HW latencies uHistograms of ‘singleton’ measurements uTells us about: Behavior of the IP stack The way the HW operates Performance of the LAN / MAN / WAN

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Measurements (2) uUDP Throughput uSend a controlled stream of UDP frames spaced at regular intervals uVary the frame size and the frame transmit spacing uAt the receiver record The time of first and last frames received The number packets received, the number lost, number out of order The received inter-packet spacing is histogramed The time each packet is received  provides packet loss pattern CPU load, Number of interrupts uUse the Pentium CPU cycle counter for times and delay Few lines of user code uTells us about: Behavior of the IP stack The way the HW operates Capacity and Available throughput of the LAN / MAN / WAN

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDPmon: The Works uHow it works: The ‘Zero_stats’ request also provides an interlock against concurrent tests. Zero stats OK done ●●● Get remote statistics Send statistics Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time Signal end of test OK done n bytes Number of packets Wait time time 

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Latency: Tests B2B PCs with Gigabit Ethernet uNIC SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Latency low & well behaved Latency Slope us/byte Expect: PCI GigE0.008 PCI us/byte

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Latency: Tests B2B PCs with Gigabit Ethernet uNIC Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel Latency high but well behaved Indicates Interrupt coalescence Slope us/byte Expect: PCI GigE0.008 PCI us/byte

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:32 bit 33 MHz 100Mbit NICs RedHat 5.2 Kernel LAN 2 PC B2B FWHM ~3  s No change with packet size Man – UCL LAN – MAN – SuperJANET4 – MAN – LAN London Man at 130Mbit on ATM 155Mbit links Latency: Histograms

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: Intel Pro/1000 on B2B P4DP6 Max throughput 950Mbit/s Some throughput drop for packets >1000 bytes Loss NIC dependent Loss not due to user  Kernel moves Traced to discards in the receiving IP layer ??? Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: SysKonnect SK-9843 B2B Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel Max throughput 690Mbit/s No packet loss Packet loss during drop

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester European Topology: NRNs & Geant SuperJANET4 CERN UvA Manc SURFnet

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit Throughput on the Production WAN Manc - RAL 570 Mbit/s 91% of the 622 Mbit access link between SuperJANET4 and RAL 1472 bytes propagation ~21  s Manc-UvA (SARA) 750 Mbit/s SJANET4 + Geant + SURFnet Manc – CERN 460 Mbit/s CERN PC had a 32 bit PCI bus

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Throughput: SLAC - Man SLAC – Manc 470 Mbit/s 75% of the 622 Mbit access link SuperJANET4 peers with ESnet at 622Mbit in NY

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Packet Loss: Amsterdam – Chicago Lambda (16b of 18) WS L2 fast->slow L2 slow->fast fast high RTT Cees de Laat & R. H-J Paris Mar 02 BW ~500 Mbit/s Keep 2 Drop 1 L2 Buffers full Packet Number Time  s

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester UDP Frame Jitter (Nov 01) Interrupt Coalescence Lon MAN ~130 Mbit Via CERN

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester TCPstream uTCP Throughput vs : msg size, transmit spacing, n-streams, window size u1 control TCP link n test streams uMemory – memory uOnly 1 process – no threads – no context switching uStreams treated equally not round Robin but next available stream uAt the receiver record The time of first and last message received The amount of data ( CPU load, Number of interrupts ) uUse the Pentium CPU cycle counter for times and delay Few lines of user code uTells us about: Behavior of the IP stack Available TCP throughput of the LAN / MAN / WAN

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit TCP on the Production WAN (01) Throughput vs TCP buffer size TCP window sizes in Mbytes calculated from RTT*bandwidth

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester Gigabit TCP on the Production WAN (10) Throughput vs n-streams Default buffer size slope = ~25 Mbit/s/stream up to 9 streams then 15 Mbit/s/stream With larger buffers rate of increase per stream is larger Plateaus at about 7 streams giving a total throughput of ~400 Mbit/s

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester More Information Some URLs  UDPmon / TCPstream kit + writeup  ATLAS Investigation of the Performance of 100Mbit and Gigabit Ethernet Components Using Raw Ethernet Frames  DataGrid WP7 Networking:  Motherboard and NIC Tests:  PPNCG Home page with Stop Press:  PPNCG Page for monitoring Grid, HEP & Astronomy Sites  IEPM PingER home site:  IEPM-BW site:

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester The Measurements (4) uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC Gigabit Ethernet Probe CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel SK bytes sent Wait 100 us ~8 us for send or receive Gigabit Ethernet frame 11.6  s Send PCI Receive PCI Send data mem  NIC Receive data NIC  mem

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel SK bytes sent Wait 20 us Sk bytes sent Wait 10 us Frames are back-to-back Can drive at line speed Cannot go any faster ! Gig Eth frames back to back

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel IT66M bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv IT66M bytes sent Wait 11 us Packets lost Action of pause packet?

CAIDA Bandwidth Estimation Meeting San Diego June 2002 R. Hughes-Jones Manchester PCI: Intel Pro/1000 on P4DP6 ITP bytes sent Wait 8 us ~5.14us on send PCI bus PCI bus ~68% occupancy ~ 2 us on PCI for data recv Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 3-5: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel