4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 1 Evaluation of the LDC Computing Platform for Point 2 SuperMicro X7DBE-X Andrey Shevel CERN PH-AID ALICE DAQ (Internal report) CERN November 2006

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 2 Purpose  Background:  Test the machine Elonex Supermicro based on motherboard X7DBE-X (2xDual Core Xeon 3.0 GHz) as LDC with 6 D-RORCs  Evaluation program:  Linux installation.  DATE V5 installation.  D-RORC throughput with DATE.  Perform a range of runs with internal D-RORC generator and receiving through DATE.  Perform a range of long runs (two days each) in configuration with the test machine as LDC and another machine as GDC.  Quick comparison with another machine based on motherboard X6DHE-XB (Dual Xeon 2.8 MHz).

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 3 Architecture of motherboard

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 4 Supermicro X7DBE-X 4U (tower) Chassis 2x USB front, 2x USB back Serial, PS/2 Periphery ATI Technologies Inc ES1000 Graphics 120 GB, SATA Disk 2x GbE, Intel 631xESC/632xESB Network 3x 64-bit 133MHz PCI-X, 3x PCI-X (64-bit 133 MHz) 3x PCI-X (64-bit 100 MHz) I/O 2 GB (max 32GB), 533 MHz,DDR2 Memory 1333 MHzFSB Intel 5000P (BlackFord) Chipset 2 x Dual Core Intel Xeon, cache L2 – total 4 MB CPU pcald44.cern.ch

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 5 Software Installation  Linux  SLC4.4 with kernel EL.cernsmp no specific problems: USB boot successful, BIOS – everything is OK  To use D-RORC modules the BIOS setting is needed to be changed: all PCI slots were set to 100 MHz  DATE 5.25 installation  MySql  mysqltcl  BWidget  SMI  DIM  Drivers: rorc and physmem have been installed as rpm with additional scripts: DATE_basic_Install-SLC4x.bash, DATE_User_Install-SLC4x.bash, DATE_DriversInstall.bash  No specific problems

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 6 Common info about measurement  Total main memory volume = 2 GB 1 GB for Linux (in /etc/grub.conf) 1 GB for physmem  6 cards D-RORC rev 4 (internal data generator).  All measurement were done with DATE No received event data were recorded to HD. The data from the database were analyzed with scripts. The pictures were built up with gnuplot.  In total there were performed 390 measurement runs. Because the figures (rates, bandwidth) are fluctuating every run was continued at least 5 minutes.  Two points: random event size with max=10**4 and max=10**6 Bytes were measured during two days each (LDC on test machine, GDC on different machine).  All measurements with randomly distributed event size were performed with event checking in DATE.  D-RORC Page Size was 10**5 Bytes.

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 7 Event rate of one card

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 8 Bandwidth of readout of one card

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 9 Example of editDb page

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 10 Example of Memory Banks page

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 11 Event rate for several cards

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 12 Bandwidth for several cards

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 13 Event rate: ConsistencyCheckLevel

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 14 Bandwidth: ConsistencyCheckLevel

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 15 Event rate: NumberOfCards, ConsistencyCheckLevel

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 16 Bandwidth: NumberOfCards, ConsistencyCheckLevel

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 17 Event rate: D-RORC Page Size

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 18 Bandwidth: D-RORC Page Size

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 19 X6DHE-XB vs X7DBE-X Parameter nameX6DBE-XBX7DBE-X CPU type 2x Xeon 2.8 GHz, 1 MB2x Xeon 5160 dual core, 3.0 GHz, 4 MB L2 cache Chipset E7520 Intel 5000P (BlackFord) FSB 800 MHz1333 MHz Memory 2 GB, DDR 3332 GB, DDR MHz PCI I/O 2x PCI-X (64/133) 4x PCI-X (64/100) 1x PCI-e (x4) 3x PCI-X (64-bit 100 MHz) 3x PCI-X (64-bit 133 MHz) Network 2x GbE, BCM57212x GbE, Intel 631xESB/632xESB SYSBENCH Memory bandwidth 1.5 GB/s3.4 GB/s RootMarks

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 20 X6DHE-XB vs X7DBE-X (continuation) Parameter nameX6DBE-XBX7DBE-X Event rate for 1 D- RORC with the size 1 KB 88.9 KHz87.2 KHz Event rate for 6 D-RORCs with size 1 KB 6 KHz12.2 KHz Bandwidth of readout for 6 D- RORCs with size 10**6 Bytes 1.58 GB/sec1.98 GB/sec

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 21 Recommendation and conclusion  Recommendation: if you need to allocate D-RORC cards in the machine and like to have maximum bandwidth for X7DBE-X : it is better to avoid to use PCI slot 4 & PCI slot 5 at the same time for high bandwidth information sources.  Conclusion  In the test it was shown bandwidth (1.96 GB/sec) when all 6 D-RORC cards are in operation and event fragment size is more 10**5 bytes when ConsistencyCheckLevel=0.  X7DBE-X shows increased event rate and bandwidth with 6 D-RORCs, also it has more CPU power in comparison to X6DHE-XB.  The configuration LDC on test machine and GDC on another machine shown about the speed 118 MB/sec with no error during two days test.  Randomly distributed event fragment size with maximum 10**6 bytes gives total bandwidth 1.53 GB/sec during two days run.  The machine on the base X7DBE-X is excellent to be used as LDC with 6 D-RORC cards.

4 Dec 2006 Testing the machine (X7DBE-X) with 6 D-RORCs 22 Thank you! Klaus Schossmaier, Sylvain Chapeland, Ulrich Fuchs and PH/AID crew