LQCD benchmarks on cluster architectures M. Hasenbusch, D. Pop, P LQCD benchmarks on cluster architectures M. Hasenbusch, D. Pop, P. Wegner (DESY Zeuthen), A. Gellrich, H.Wittig (DESY Hamburg) CHEP03, 25 March 2003 Category 6: Lattice Gauge Computing Motivation PC Cluster @DESY Benchmark architectures DESY Cluster E7500 systems Infiniband blade servers Itanium2 Benchmark programs, Results Future Conclusions, Acknowledgements Peter Wegner, DESY CHEP03, 25 March 2003 1

PC Cluster Motivation LQCD, Stream Benchmark, Myrinet Bandwidth 32/64-bit Dirac Kernel, LQCD (Martin Lüscher, (DESY) CERN, 2000): P4, 1.4 GHz, 256 MB Rambus, using SSE1(2) instructions incl. cache pre-fetch Time per lattice point: 0.926 micro sec (1503 Mflops [32 bit arithmetic]) 1.709 micro sec (814 Mflops [64 bit arithmetic]) Stream Benchmark, Memory Bandwidth: P4(1.4 GHz, PC800 Rambus) : 1.4 … 2.0 GB/s PIII (800MHz, PC133 SDRAM) : 400 MB/s PIII(400 MHz, PC133 SDRAM) : 340 MB/s Myrinet, external Bandwidth: 2.0+2.0 Gb/s optical-connection, bidirectional, ~240 MB/s sustained Peter Wegner, DESY CHEP03, 25 March 2003 2

Benchmark Architectures - DESY Cluster Hardware Nodes Mainboard Supermicro P4DC6 2 x XEON P4, 1.7 (2.0) GHz, 256 (512) kByte Cache 1 Gbyte (4x 256 Mbyte) RDRAM IBM 18.3 GB DDYS-T18350 U160 3.5” SCSI disk Myrinet 2000 M3F-PCI64B-2 Interface Network Fast Ethernet Switch Gigaline 2024M, 48x100BaseTX ports + GIGAline2024 1000BaseSX-SC Myrinet Fast Interconnect M3-E32 5 slot chassis, 2xM3-SW16 Line cards Installation Zeuthen: 16 dual CPU nodes, Hamburg: 32 dual CPU nodes Peter Wegner, DESY CHEP03, 25 March 2003 3

Benchmark Architectures DESY Cluster i860 chipset problem Xeon Processor Xeon Processor 400MHz System Bus AGP4X Graphics Dual Channel RDRAM* 3.2 GB/s MRH >1GB/s + + P64H 800MB/s MCH 3.2GB/s MRH 800MB/s + + P64H Intel® Hub Architecture 266 MB/s 64 bit PCI ATA 100 MB/s (dual IDE Channels) 6 channel audio 64 bit PCI ICH2 LAN Connection Interface Up to 4 GB of RDRAM 133 MB/s PCI Slots (66 MHz, 64bit) PCI Slots (33 MHz, 32bit) 10/100 Ethernet bus_read (send) = 227 MBytes/s bus_write (recv) = 315 MBytes/s of max. 528 MBytes/s 4 USB ports External Myrinet bandwidth: 160 Mbytes/s 90 Mbytes/s bidirectional

Benchmark Architectures – Intel E7500 chipset Peter Wegner, DESY CHEP03, 25 March 2003 5

Benchmark Architectures - E7500 system Par-Tec (Wuppertal) 4 Nodes: Intel(R) Xeon(TM) CPU 2.60GHz 2 GB ECC PC1600 (DDR-200) SDRAM Super Micro P4DPE-G2 Intel E7500 chipset PCI 64/66 2 x Intel(R) PRO/1000 Network Connection Myrinet M3F-PCI64B-2 Peter Wegner, DESY CHEP03, 25 March 2003 6

Benchmark Architectures Leibniz-Rechenzentrum Munich (single cpu tests): Pentium IV 3,06GHz. with ECC Rambus Pentium IV 2,53GHz. with Rambus 1066 memory Xeon, 2.4GHz. with PC2100 DDR SDRAM memory (probably FSB400) Megware: 8 nodes dual XEON, 2.4GHz, E7500 2GB DDR ECC memory Myrinet2000 Supermicro P4DMS-6GM University of Erlangen: Itanium2, 900MHz, 1.5MB Cache, 10GB RAM zx1 chipset (HP) Peter Wegner, DESY CHEP03, 25 March 2003 7

Benchmark Architectures - Infiniband Megware: 10 Mellanox ServerBlades Single Xeon 2.2 GHz 2 GB DDR RAM ServerWorks GC-LE Chipsatz InfiniBand 4X HCA RedHat 7.3, Kernel 2.4.18-3 MPICH-1.2.2.2 und OSU-Patch für VIA/InfiniBand 0.6.5 Mellanox Firmware 1.14 Mellanox SDK (VAPI) 0.0.4 Compiler GCC 2.96 Peter Wegner, DESY CHEP03, 25 March 2003 8

Dirac Operator Benchmark (SSE) 16x163, single P4/XEON CPU MFLOPS Dirac operator Linear Algebra Peter Wegner, DESY CHEP03, 25 March 2003 9

Parallel (1-dim) Dirac Operator Benchmark (SSE), even-odd preconditioned, 2 x 163 , XEON CPUs, single CPU performance MFLOPS Myrinet2000 i860: 90 MB/s E7500: 190 MB/s Peter Wegner, DESY CHEP03, 25 March 2003 10

Performance comparisons (MFLOPS): Single node Dual node Parallel (1-dim) Dirac Operator Benchmark (SSE), even-odd preconditioned, 2 x 163 , XEON CPUs, single CPU performance, 2, 4 nodes Performance comparisons (MFLOPS): Single node Dual node SSE2 non-SSE 446 328 (74%) 330 283 (85%) Parastation3 software non-blocking I/O support (MFLOPS, non-SSE): blocking non-blocking I/O 308 367 (119%) Peter Wegner, DESY CHEP03, 25 March 2003 11

Maximal Efficiency of external I/O MFLOPs (without communication) MFLOPS (with communication) Maximal Bandwidth Efficiency Myrinet (i860), SSE 579 307 90 + 90 0.53 Myrinet/GM (E7500), SSE 631 432 190 + 190 0.68 Myrinet/ Parastation (E7500), SSE 675 446 181 + 181 0.66 Parastation (E7500), non-blocking, non-SSE 406 368 hidden 0.91 Gigabit, Ethernet, non-SSE 390 228 100 + 100 0.58 Infiniband 370 297 210 + 210 0.80 Peter Wegner, DESY CHEP03, 25 March 2003 12

4 single CPU nodes, Gbit Ethernet, non-blocking switch, full duplex Parallel (1-dim) Dirac Operator Benchmark (SSE), even-odd preconditioned, 2 x 163 , XEON/Itanium2 CPUs, single CPU performance, 4 nodes 4 single CPU nodes, Gbit Ethernet, non-blocking switch, full duplex P4 (2.4 GHz, 0.5 MB cache) SSE: 285 MFLOPS 88.92 + 88.92 MB/s non-SSE: 228 MFLOPS 75.87 + 75.87 MB/s Itanium2 (900 MHz, 1.5 MB cache) non-SSE: 197 MFLOPS 63.13 + 63.13 MB/s Peter Wegner, DESY CHEP03, 25 March 2003 13

Infiniband interconnect up to 10GB/s Bi-directional Sys Mem HCA Mem Cntlr Host Bus CPU Switch: Simple, low cost, multistage network Link: High Speed Serial 1x, 4x, and 12x    Sys Mem CPU Mem Cntlr Host Bus Link Switch HCA TCA I/O Cntlr Target Channel Adapter: Interface to I/O controller SCSI, FC-AL, GbE, ... TCA I/O Cntlr Host Channel Adapter: Protocol Engine Moves data via messages queued in memory Chips : IBM, Mellanox PCI-X cards: Fujitsu, Mellanox, JNI, IBM http://www.infinibandta.org Peter Wegner, DESY CHEP03, 25 March 2003 14

Infiniband interconnect    Peter Wegner, DESY CHEP03, 25 March 2003 15

Infiniband vs Myrinet performance, non-SSE (MFLOPS): Parallel (2-dim) Dirac Operator Benchmark (Ginsparg-Wilson-Fermions) , XEON CPUs, single CPU performance, 4 nodes Infiniband vs Myrinet performance, non-SSE (MFLOPS): XEON 1.7 GHz Myrinet, i860 chipset XEON 2.2 GHz Infiniband, E7500 chipset 32-Bit 64-Bit 8x83 lattice, 2x2 processor grid 370 281 697 477 16x163 lattice, 2x4 processor grid 338 299 609 480 Peter Wegner, DESY CHEP03, 25 March 2003 16

Future - Low Power Cluster Architectures ? Peter Wegner, DESY CHEP03, 25 March 2003 17

Future Cluster Architectures - Blade Servers ? NEXCOM – Low voltage blade server 200 low voltage Intel XEON CPUs (1.6 GHz – 30W) in a 42U Rack Integrated Gbit Ethernet network Mellanox – Infiniband blade server Single XEON Blades connected via a 10 Gbit (4X) Infiniband network MEGWARE, NCSA, Ohio State University Peter Wegner, DESY CHEP03, 25 March 2003 18

Conclusions PC CPUs have an extremely high sustained LQCD performance using SSE/SSE2 (SIMD+pre-fetch), assuming a sufficient large local lattice Bottlenecks are the memory throughput and the external I/O bandwidth, both components are improving (Chipsets: i860  E7500  E705  …, FSB: 400MHz  533 MHz  667 MHz  …, external I/O: Gbit-Ethernet  Myrinet2000  QSnet  Inifiniband  …) Non-blocking MPI communication can improve the performance by using adequate MPI implementations (e.g. ParaStation) 32-bit Architectures (e.g. IA32) have a much better price performance ratio than 64-bit architectures (Itanium, Opteron ?) Large low voltage dense blade clusters could play an important role in LQCD computing (low voltage XEON, CENTRINO ?, …) Peter Wegner, DESY CHEP03, 25 March 2003 19

Acknowledgements Acknowledgements We would like to thank Martin Lüscher (CERN) for the benchmark codes and the fruitful discussions about PCs for LQCD, and Isabel Campos Plasencia (Leibnitz-Rechenzentrum Munich), Gerhard Wellein (Uni Erlangen), Holger Müller (Megware), Norbert Eicker (Par-Tec), Chris Eddington (Mellanox) for the opportunity to run the benchmarks on their clusters. Peter Wegner, DESY CHEP03, 25 March 2003 20