2. Copyright© 2000 OPNET Technologies, Inc. R.W. Dobinson, S. Haas, K. Korcyl, M.J. LeVine, J. Lokier, B. Martin, C. Meirosu, F. Saka, K. Vella Testing and Modeling Ethernet Switches and Networks for Use in ATLAS High-level Triggers

3. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 2 Overview Ethernet as a candidate technology for ATLAS LVL2 ATLAS - the networking requirement Parameterized model of an Ethernet switch Large network modeling results using the parameterized switch model GBE measurements Fast Ethernet tester/ROB emulator

4. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 3 ATLAS LVL2 system network 1564 buffers (distributed 2 Mbyte image) ~500 500MIPS processors analyze data from 5% of buffers CENTRAL Gigabit switch Concentrating switches Gigabit Ethernet Fast Ethernet Trigger rate 100 kHz Readout buffers (ROB) Processing nodes

5. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 4 ATLAS LVL2 network traffic Network nodes 1564 ROBs (Readout Buffers ) 500-600 Processor CPUs Message scenario LVL1 accepts @ 75 kHz [scalable to100 kHz] Supervisor CPU selects processor CPU for each event accepted by LVL1 Processor CPU sends ROB_REQUEST to selected (~5%) of ROBs ROBs reply with ROB_REPLY containing designated ROI data Traffic @ ROB: ~11000 request-response/sec Worst case data rate: 4 MB/s Traffic @ processor node: ~14000 request-response/sec Average data rate: 6.6 MB/s

6. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 5 Modeling Parameterized model of a switch  Motivation - Limited number of physical parameters Fast execution Direct determination by measurement  commodity Ethernet switches  parameterized model of a switch  comparison of parameterized model and results from measurements Parameterized model in large Ethernet network  ATLAS HEP experiment at LHC in CERN  second level trigger (LVL2) for ATLAS experiment

7. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 6 Commodity Ethernet switches Most of the Ethernet switches on the market today have a hierarchical architecture ports modules backplane Store-and-forward mode of operation frame is fully stored in module with an input port for inter-module transfer the frame is subsequently stored in a module with an output port module ports chassis backplane

8. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 7 Parameterized model for switch: inter-module communication Buffer manager MAC Input buffer P1 Buffer manager Output buffer P2 Backplane P6 P7 P4 P7 P3 Parameters: P1 - Input Buffer Length [ #frames] P2 - Output Buffer Length [ # frames] P3 - Max ToBackplane Throughput [MB/s] P4 - Max FromBackplane Throughput [MB/s] P6 - Max Backplane Throughput [MB/s] P7 - Inter-module Transfer Bandwidth [ MB/s] P9 - Inter-module Fixed Overhead [µs] (not shown) Input moduleOutput module

9. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 8 Switch parameters from latency and rate measurements Measure latency as function of packet size (ping-pong) Measure packet rate as function of packet size No switch, intra-module, inter-module

10. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 9 Modeling calculations Switch parameters determined from simple ping-pong and streaming measurements described Switch model implemented as C++ code Event-driven simulations Object-oriented switch model embedded in OPNET (commercial package) Calculations also carried out using the same model within Ptolemy

11. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 10 parameterized model vs measurement latency vs throughput

12. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 11 parameterized model vs measurement lost frame rate vs input data rate

13. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 12 ATLAS LVL2 network performance studies Central switch: generic - fully non-blocking

14. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 13 Modeling results The parameterized model uses a limited number of parameters, which can be measured on COTS switches The parameterized model has been successfully verified on a number of switches The parameterized switch model was used in modeling the large Ethernet network for the ATLAS second level trigger system The model of the network can be used to verify applicability of different switches currently on the market and their impact on the network performance

15. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 14 CPU utilization measurements Ping-pong configuration at different packet rates Measure CPU utilization for various packet sizes, rates ATLAS LVL2 REQUIREMENT

16. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 15 Limitations of PC-based measurements PC with standard software unable to drive switches at line speed 100% for FE only for packet size > 500 Byte [half duplex] 35% for GBE for any packet size - limited by host transfer Parameterization of a single switch needs to be tested over the entire operational envelope Number of nodes limited to 40 in present tests Scaling up to ~2000 nodes is not credible Model needs to be verified on a larger test bed Conclusion: Dedicated traffic generators needed for FE, GBE in order to fully characterize switches to be used in ATLAS HLT Need larger test bed for greater confidence in model applied to full ATLAS HLT system

17. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 16 Rates: GB Ethernet NIC development Commercial NIC - Alteon ACENIC Inadequate performance using manufacturer’s firmware Modifications - Custom software developed All traffic local to NIC (host memory not involved) Full duplex line speed achieved for any size packet PCI BUS CPU 2 PCI interface CPU 1 MAC Ext Mem 0.5/1 Mbyte Mem DMA1 DMA2 Phy GE TIGON CHIP PCI BUS TIGON CHIP

18. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 17 8 Gigabit switch tester in a single chassis

19. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 18 Latency vs throughput for 8-port GBE switch

20. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 19 Fast Ethernet tester Custom board, built at CERN Programmable network traffic to characterize network switches IP, Ethernet, including QoS Why build it instead of buying? Economics Reprogrammable for ATLAS purposes

21. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 20 FE tester: capabilities & status 32 ports full-duplex fast Ethernet Parallel port connection to host MAC function: FPGA All FPGAs programmed in Handel C Handel C is a high-level language with ‘C’ syntax Enhancements for FPGAs (e.g., arbitrary width bit fields) Simulation facility built into implementation Output packets generated, time stamped Incoming packets time stamped, CRC checked Queue dwell times histogrammed All of this full duplex @ full line speed

22. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 21 Fast Ethernet tester board

23. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 22 FE tester as ROB emulator Accepts MESH ROB_DATA_REQUESTs Generates MESH ROB_DATA_REPLYs Programmable latency in ROB Response size depends on ROI size and detector type Response contents not meaningful Measure latencies and queue depths

24. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 23 Large scale test bed using ROB emulator Use 8 of these boards [11K CHF each] to provide 256 emulated ROB ports Add: - 64 PCs - supervisors + farm nodes Switch fabric We have a ~15% scale test bed of Atlas LVL2 Previous test bed: order of magnitude smaller

25. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 24 Future activity Measurements - Improved switch characterization Modeling - Parameterized model of GE switch (central switch) Detailed modeling of ATLAS LVL2 Modeling of unified network to handle LVL2 traffic + Event Filter (LVL3 trigger)

26. DAQ 2000 Oct 2000 Testing & Modeling Ethernet Switches for ATLAS HLT 25 Summary TCP/IP is very expensive in terms of CPU utilization at rates required MESH will likely be used in ATLAS LVL2 trigger GBE needs custom firmware in NIC to get required performance Modeling in good agreement with single switch measurements Large scale test bed minimizes uncertainty associated with behavior extrapolated from a few nodes Hardware ROB emulation provides opportunity to see network behavior under realistic conditions