1. Evaluation of the 2-way Opteron 1U system Klaus Schossmaier CERN EP-AID Computing Seminar 3 September 2003 Performance test of PCs based on AMD platforms

2. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 2 Welcome  The EP-AID group constitutes the core team of the Data Acquisition (DAQ) System for the LHC Experiment ALICE  Development of the DAQ software called DATE  Deployment of the DAQ system  Computing fabric: ~500 nodes, GbE switches, ~400 optical links  Permanent recording during Pb-Pb run: 1.25 GB/s during 1 month/year  Selecting products  Evaluation of Linux platforms  Intel: Pentium II, Pentium III, Xeon  AMD: AthlonMP (32bit), Opteron (64 bit)

3. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 3 Overview  In March 2003 we received the Newisys 2100 evaluation system (“Beachhead”)  This 2-way 1U enterprise-class server is based on AMD Opteron processors (x86-64 architecture)  Our evaluation was done under NDA:  Hardware, chassis, BIOS  Linux operating systems  Development tools  Benchmarking  Applications

4. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 4 Chassis & Periphery  Rack mountable  Size: 42mm x 428mm x 725mm  Robust construction  Easy to mount the rack module  Easy to open the cover (no screws)  Solid connectors  Not very hot or noisy!  Good documentation  Labeling, e.g. MAC addresses  Newisys “Getting Started Guide”  Newisys support website

5. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 5 Hardware – Front View 2x hard disk bays: - empty - 36.7GB SCSI hot swap CD-ROM drive Floppy drive Operator panel for the SP 4x 1GB DIMM DDR266 SDRAM 4x 1GB DIMM DDR266 SDRAM 2x Opteron 240, 1 MB L2 cache, 1388 MHz 2x Opteron 244, 1 MB L2 cache, 1793 MHz Power supply 465W (March – May) (June – now)

6. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 6 Hardware – Rear View PCI-X slot 64/133 full-length PS/2 mouse keyboard SVGA video Trident Blade 3D 8 MB 2x 10/100BaseT for the SP Service processor (SP) 2x 100/1000BaseT BCM5703X chip PCI-X slot 64/100 half-length AMD-8131 PCI-X bridgeAMD-8111 Southbridge

7. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 7 Booting & BIOS  Booting/Resetting/Shutdown is fast (~10 sec)  PhoenixBIOS 4.0 Release 6.0  Easy to enter the BIOS setup  Well designed configuration menus  Upgrade via the phlash16 flash-utility or the SP  System BIOS Revision  00.09.06  initial  00.09.08  bug fix  01.08.00  Opteron 1.8 GHz  01.09.00  Opteron “C” step

8. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 8 Linux Operating Systems  CERN Red Hat 7.3.2 Linux (2.4.18 kernel)  Installation via boot floppy and NFS  Everything works without any change in 32-bit mode  UnitedLinux 1.0 (2.4.19 kernel)  Consortium of Conectiva, SCO, SuSE, Turbolinux  Installation via CD-ROM (4 disks) using YaST  Runs in 64-bit mode  Red Hat 8.0.95 = GinGin64  Installation via CD-ROM (6 disks) failed

9. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 9 Development Tools  GNU 2.96 C/C++ compiler  default for Red Hat Linux installations  only for 32-bit executables  GNU 3.2.1 C/C++ compiler  default for UnitedLinux installations  for 64-bit executables, 32-bit compatibility libraries  Portland Group Compiler Technology (PGI) Workstation 5.0-beta Fortan/C/C++ compiler  commercial product  for 64-bit executables  Other development tools (e.g. numerical libraries, debuggers, code migration, performance analyzers) exist, but not tested.

10. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 10 Benchmarking  CPU benchmarking  Ubench 0.32, gcc 2.96, 32-bit mode  Ubench 0.32, gcc 3.2.1, 64-bit mode  bench/stress rootmarks, 64-bit mode  Memory bandwidth  Stream 4.0, gcc 2.96, 32-bit mode  Stream 4.0, gcc 3.2.1, 64-bit mode  Ethernet performance  Iperf 1.2, gcc 2.96, 32-bit mode  Disk performance  Bonnie, 64-bit mode

11. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 11 Benchmark - CPU (Ubench) 64-bit 74955 97749 52122 66714 88064 79798 1.0 GHz1.4 GHz 1.8 GHz 2.4 GHz

12. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 12 Benchmark - CPU (ROOT)  Done by Fons Rademakers (EP-SFT)  Porting of ROOT and CINT was trivial  OO data analysis framework with ~900000 locs  Porting to x86-64 under United Linux 1.0 in one afternoon  Code was already 64-bit safe (no special x86-64 ifdef’s)  No problem with gcc 3.2.1  Performance is very good Opteron 1.4 GHz 1 MB cache, 8 GB RAM Opteron 1.8 GHz 1 MB cache, 8 GB RAM Itanium2 1.0 GHz 3 MB cache, 2 GB RAM P4 Xeon 3.06 GHz 512 KB cache, 2 GB RAM 750 rootmarks g++ 3.2.1 (-O2) 950 rootmarks g++ 3.2.1 (-O2) 497 rootmarks g++ 3.2.3 (-O2) 750 rootmarks g++ 3.2.3 (-O2) 550 rootmarks 32-bit binary compiled on P4 with g++ 3.2.2 (-O2) 1020 rootmarks ecc 7.1 (-O)

13. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 13 Benchmark - Memory 1x Stream:2x Stream:4x Stream: 2x Opteron, 1.8 GHz, HyperTransport:1006 – 1671 MB/s975 – 1178 MB/s924 – 1133 MB/s 2x Xeon, 2.4 GHz, 400 MHz FSB:1202 – 1404 MB/s561 – 785 MB/s365 – 753 MB/s 4x DIMM 1GB DDR266 Avent Techn. 4x DIMM 1GB DDR266 Avent Techn.

14. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 14 Benchmark - Ethernet, Disk  Network benchmarking  2x Broadcom BCM5703X on motherboard  10/100/1000 Mbps Ethernet, auto-negotiation  bcm5700 kernel module, 64-bit mode  up to 105 MB/s, stable traffic during 4 days  20%+18% sys CPU as TCP/IP receiver  20%+2% sys CPU as TCP/IP sender  Disk benchmarking  Seagate ST336753LC, 15kRPM  LSI53C1030 dual channel Ultra320 SCSI controller  26.7 MB/s for writing a 6 GB file  36.2 MB/s for reading a 6 GB file

15. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 15 Service Processor (SP)  Dedicated SP on-board  PowerPC based  Own IP name/address  Front panel  Command line interface  Web-server  Remote administration  System status  Boot/Reset/Shutdown  Flash the BIOS

16. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 16 Applications - DAQ  Front-end computer / Event-Builder  DATE v4.6  Red Hat 7.3.2  32-bit mode, binary compatible!  Event-Building: 113 MB/s with 2 sources, 80% CPU  Porting to 64-bit mode needs to be done  DAQ services  NFS server  Run control, message logger, database  Online monitoring: performance, data quality

17. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 17 Applications – non DAQ  MySQL 4.0.13 for x86-64  64-bit mode, binary distribution  UnitedLinux 1.0  ~3300 remote INSERT queries (80 chars) per sec  Web Servers  HEP computing  Data analysis  Simulation tools  Visualizations

18. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 18 Résumé  Merits  Design: chassis, hardware, BIOS  Execution: seamless 32-bit & 64-bit mode  Benchmarking: CPU, memory, Ethernet, disk  Applications: DAQ services, analysis, simulations  Improvements  Linux operating systems: more in 64-bit mode  Tool chain: more non-commercial tools  Questions  Cost: processor and whole system  Roadmap: AMD’s 64-bit processors  Thanks  AMD: Pierre-Yves Ferrard, Francesco Torricelli, Bill Zipoy  CERN ALICE: Pierre Vande Vyvre

19. 2-way Opteron 1U systemComputing Seminar, 3 Sep 2003 19 References  Web pages  www.amd.com/opteron www.amd.com/opteron  www.x86-64.org www.x86-64.org  www.hypertransport.org www.hypertransport.org  www.newisys.com www.newisys.com  Technical Reports  “Performance tests of PCs based on the AMD AthlonMP”, ALICE-INT-2003-020  A report about the 2-way Opteron system is forthcoming  Further Talks  15 Sep 2003: “Linux platforms for the ALICE DAQ”  Access to our Opteron machine  Please ask me for an account