Evaluation of the 2-way Opteron 1U system Klaus Schossmaier CERN EP-AID Computing Seminar 3 September 2003 Performance test of PCs based on AMD platforms

2-way Opteron 1U systemComputing Seminar, 3 Sep Welcome  The EP-AID group constitutes the core team of the Data Acquisition (DAQ) System for the LHC Experiment ALICE  Development of the DAQ software called DATE  Deployment of the DAQ system  Computing fabric: ~500 nodes, GbE switches, ~400 optical links  Permanent recording during Pb-Pb run: 1.25 GB/s during 1 month/year  Selecting products  Evaluation of Linux platforms  Intel: Pentium II, Pentium III, Xeon  AMD: AthlonMP (32bit), Opteron (64 bit)

2-way Opteron 1U systemComputing Seminar, 3 Sep Overview  In March 2003 we received the Newisys 2100 evaluation system (“Beachhead”)  This 2-way 1U enterprise-class server is based on AMD Opteron processors (x86-64 architecture)  Our evaluation was done under NDA:  Hardware, chassis, BIOS  Linux operating systems  Development tools  Benchmarking  Applications

2-way Opteron 1U systemComputing Seminar, 3 Sep Chassis & Periphery  Rack mountable  Size: 42mm x 428mm x 725mm  Robust construction  Easy to mount the rack module  Easy to open the cover (no screws)  Solid connectors  Not very hot or noisy!  Good documentation  Labeling, e.g. MAC addresses  Newisys “Getting Started Guide”  Newisys support website

2-way Opteron 1U systemComputing Seminar, 3 Sep Hardware – Front View 2x hard disk bays: - empty GB SCSI hot swap CD-ROM drive Floppy drive Operator panel for the SP 4x 1GB DIMM DDR266 SDRAM 4x 1GB DIMM DDR266 SDRAM 2x Opteron 240, 1 MB L2 cache, 1388 MHz 2x Opteron 244, 1 MB L2 cache, 1793 MHz Power supply 465W (March – May) (June – now)

2-way Opteron 1U systemComputing Seminar, 3 Sep Hardware – Rear View PCI-X slot 64/133 full-length PS/2 mouse keyboard SVGA video Trident Blade 3D 8 MB 2x 10/100BaseT for the SP Service processor (SP) 2x 100/1000BaseT BCM5703X chip PCI-X slot 64/100 half-length AMD-8131 PCI-X bridgeAMD-8111 Southbridge

2-way Opteron 1U systemComputing Seminar, 3 Sep Booting & BIOS  Booting/Resetting/Shutdown is fast (~10 sec)  PhoenixBIOS 4.0 Release 6.0  Easy to enter the BIOS setup  Well designed configuration menus  Upgrade via the phlash16 flash-utility or the SP  System BIOS Revision   initial   bug fix   Opteron 1.8 GHz   Opteron “C” step

2-way Opteron 1U systemComputing Seminar, 3 Sep Linux Operating Systems  CERN Red Hat Linux ( kernel)  Installation via boot floppy and NFS  Everything works without any change in 32-bit mode  UnitedLinux 1.0 ( kernel)  Consortium of Conectiva, SCO, SuSE, Turbolinux  Installation via CD-ROM (4 disks) using YaST  Runs in 64-bit mode  Red Hat = GinGin64  Installation via CD-ROM (6 disks) failed

2-way Opteron 1U systemComputing Seminar, 3 Sep Development Tools  GNU 2.96 C/C++ compiler  default for Red Hat Linux installations  only for 32-bit executables  GNU C/C++ compiler  default for UnitedLinux installations  for 64-bit executables, 32-bit compatibility libraries  Portland Group Compiler Technology (PGI) Workstation 5.0-beta Fortan/C/C++ compiler  commercial product  for 64-bit executables  Other development tools (e.g. numerical libraries, debuggers, code migration, performance analyzers) exist, but not tested.

2-way Opteron 1U systemComputing Seminar, 3 Sep Benchmarking  CPU benchmarking  Ubench 0.32, gcc 2.96, 32-bit mode  Ubench 0.32, gcc 3.2.1, 64-bit mode  bench/stress rootmarks, 64-bit mode  Memory bandwidth  Stream 4.0, gcc 2.96, 32-bit mode  Stream 4.0, gcc 3.2.1, 64-bit mode  Ethernet performance  Iperf 1.2, gcc 2.96, 32-bit mode  Disk performance  Bonnie, 64-bit mode

2-way Opteron 1U systemComputing Seminar, 3 Sep Benchmark - CPU (Ubench) 64-bit GHz1.4 GHz 1.8 GHz 2.4 GHz

2-way Opteron 1U systemComputing Seminar, 3 Sep Benchmark - CPU (ROOT)  Done by Fons Rademakers (EP-SFT)  Porting of ROOT and CINT was trivial  OO data analysis framework with ~ locs  Porting to x86-64 under United Linux 1.0 in one afternoon  Code was already 64-bit safe (no special x86-64 ifdef’s)  No problem with gcc  Performance is very good Opteron 1.4 GHz 1 MB cache, 8 GB RAM Opteron 1.8 GHz 1 MB cache, 8 GB RAM Itanium2 1.0 GHz 3 MB cache, 2 GB RAM P4 Xeon 3.06 GHz 512 KB cache, 2 GB RAM 750 rootmarks g (-O2) 950 rootmarks g (-O2) 497 rootmarks g (-O2) 750 rootmarks g (-O2) 550 rootmarks 32-bit binary compiled on P4 with g (-O2) 1020 rootmarks ecc 7.1 (-O)

2-way Opteron 1U systemComputing Seminar, 3 Sep Benchmark - Memory 1x Stream:2x Stream:4x Stream: 2x Opteron, 1.8 GHz, HyperTransport:1006 – 1671 MB/s975 – 1178 MB/s924 – 1133 MB/s 2x Xeon, 2.4 GHz, 400 MHz FSB:1202 – 1404 MB/s561 – 785 MB/s365 – 753 MB/s 4x DIMM 1GB DDR266 Avent Techn. 4x DIMM 1GB DDR266 Avent Techn.

2-way Opteron 1U systemComputing Seminar, 3 Sep Benchmark - Ethernet, Disk  Network benchmarking  2x Broadcom BCM5703X on motherboard  10/100/1000 Mbps Ethernet, auto-negotiation  bcm5700 kernel module, 64-bit mode  up to 105 MB/s, stable traffic during 4 days  20%+18% sys CPU as TCP/IP receiver  20%+2% sys CPU as TCP/IP sender  Disk benchmarking  Seagate ST336753LC, 15kRPM  LSI53C1030 dual channel Ultra320 SCSI controller  26.7 MB/s for writing a 6 GB file  36.2 MB/s for reading a 6 GB file

2-way Opteron 1U systemComputing Seminar, 3 Sep Service Processor (SP)  Dedicated SP on-board  PowerPC based  Own IP name/address  Front panel  Command line interface  Web-server  Remote administration  System status  Boot/Reset/Shutdown  Flash the BIOS

2-way Opteron 1U systemComputing Seminar, 3 Sep Applications - DAQ  Front-end computer / Event-Builder  DATE v4.6  Red Hat  32-bit mode, binary compatible!  Event-Building: 113 MB/s with 2 sources, 80% CPU  Porting to 64-bit mode needs to be done  DAQ services  NFS server  Run control, message logger, database  Online monitoring: performance, data quality

2-way Opteron 1U systemComputing Seminar, 3 Sep Applications – non DAQ  MySQL for x86-64  64-bit mode, binary distribution  UnitedLinux 1.0  ~3300 remote INSERT queries (80 chars) per sec  Web Servers  HEP computing  Data analysis  Simulation tools  Visualizations

2-way Opteron 1U systemComputing Seminar, 3 Sep Résumé  Merits  Design: chassis, hardware, BIOS  Execution: seamless 32-bit & 64-bit mode  Benchmarking: CPU, memory, Ethernet, disk  Applications: DAQ services, analysis, simulations  Improvements  Linux operating systems: more in 64-bit mode  Tool chain: more non-commercial tools  Questions  Cost: processor and whole system  Roadmap: AMD’s 64-bit processors  Thanks  AMD: Pierre-Yves Ferrard, Francesco Torricelli, Bill Zipoy  CERN ALICE: Pierre Vande Vyvre

2-way Opteron 1U systemComputing Seminar, 3 Sep References  Web pages      Technical Reports  “Performance tests of PCs based on the AMD AthlonMP”, ALICE-INT  A report about the 2-way Opteron system is forthcoming  Further Talks  15 Sep 2003: “Linux platforms for the ALICE DAQ”  Access to our Opteron machine  Please ask me for an account