1. IEEE Workshop on HSLN 16 Nov 2004 SCI Networking for Shared-Memory Computing in UPC: Blueprints of the GASNet SCI Conduit Hung-Hsun Su, Burton C. Gordon, Sarp Oral, and Alan D. George HPN and UPC Groups HCS Research Laboratory University of Florida

2. IEEE Workshop on HSLN 16 Nov 2004 2 Outline Introduction Design Results Updates Conclusions

3. IEEE Workshop on HSLN 16 Nov 2004 3 Introduction Goal: Enable and evaluate UPC computing over SCI networks Parallel programming models have emerged to provide programmers alternative ways in solving complex and computationally intensive problems  Message passing  Shared memory  Global address space (GAS) Message-passing and shared-memory programming models are the two most popular approaches GAS model quickly gaining momentum  Programming can be very complex in message-passing model  Shared-memory model does not work well on server clusters  GAS model attempts to provide the best of both worlds Ease of programming Support for various system architectures

4. IEEE Workshop on HSLN 16 Nov 2004 4 Introduction Unified Parallel C (UPC)  Partitioned GAS language  Parallel extension to the ISO C standard  Smaller learning curve for people with C experience Available on various platforms  Vendor supplied Cray T3D, T3E HP AlphaServer SC, SMP, and PA-RISC SMP Under development: IBM, Sun  Open source Berkeley UPC (BUPC)  Wide variety of architectures and operating systems  Large-scale multiprocessors  PC clusters  Clusters of shared-memory multiprocessors Intrepid GCC-UPC  SCI Origin 2000, 3000, IRIX, ALTIX Michigan Tech UPC (MUPC)  Linux clusters

5. IEEE Workshop on HSLN 16 Nov 2004 5 Introduction Global Address Space Networking (GASNet)  Communication system developed at UCB/LBNL  Language-independent and network-independent communication middleware  Provides high-performance comm. primitives aimed at supporting GAS languages  Used by Berkeley UPC, Titanium (parallel extension of Java), and Intrepid GCC-UPC  Supports execution on UDP, MPI, Myrinet, Quadrics, InfiniBand, and IBM LAPI

6. IEEE Workshop on HSLN 16 Nov 2004 6 Introduction Scalable Coherent Interface (SCI) – IEEE Standard 1596-1992  High-performance interconnect for system-area-network (SAN) clusters and embedded systems Allows memory on each node of system to be accessed by every other node on network Uses point-to-point links  1D, 2D, 3D Low-latency transfer  Single-digit microseconds for remote write  Tens of microseconds for remote read High data rate  5.3 Gb/s Good fit for GASNet  Dolphin SISCI API Standard set of API calls provided by hardware vendor to access and control SCI hardware Two modes of transfer  PIO  Low-latency, shared-memory operation  Requires memory importation (maps a portion of virtual memory to remote memory region)  DMA  High-bandwidth, zero-copy operation  No memory importation required

7. IEEE Workshop on HSLN 16 Nov 2004 7 Design GASNet is divided into two layers  Core API Narrow interface based on Active Messages (AM)  Extended API Provides medium- and high-level operations on remote memory and collective operations Reference version available using the Core API  Successful implementation of Core API sufficient for a complete GASNet conduit Core API Design  Support the three types of AM messages Short: header only Medium: header with payload stored in buffer space Long: header with payload stored in designated address  Uses remote write exclusively to achieve maximum performance  Communication regions Command region  Buffer space for incoming AM requests and replies  Able to hold k pairs of request/reply  Pairing ensures a deadlock-free system  Size of each request/reply buffer space = longest AM header size + max. medium payload size  PIO transfer mode

8. IEEE Workshop on HSLN 16 Nov 2004 8 Design Communication regions (cont.)  Control region Buffer space for message flags  Allow nodes to check for incoming message locally  Message-ready flags  One for each request/reply message  Indicates existence of a particular incoming request/reply message  Message-exist flag  One per node  Indicates existence of any incoming message PIO transfer mode  Payload region Corresponds to range of remotely accessible memory as specified by the GAS languages (user-defined) DMA transfer mode  Achieves higher bandwidth  Improves scalability of system

9. IEEE Workshop on HSLN 16 Nov 2004 9 Design AM communication 1.Obtain free slot Tracked locally using array of flags 2.Package AM header 3.Transfer data Short AM  PIO write (Header) Medium AM  PIO write (Header)  PIO write (Medium payload) Long AM  PIO write (header)  PIO write (long payload)  Payload size  1024  Unaligned portion of payload  DMA write (multiple of 64 bytes) 4.Wait for transfer completion 5.Signal AM arrival PIO write  Message-ready flag = type of AM  Message-exist flag = TRUE 6.Wait for reply/control signal Free up remote slot for reuse

10. IEEE Workshop on HSLN 16 Nov 2004 10 Results Experimental testbeds  SCI, Quadrics, InfiniBand, and MPI conduits (via testbed at UF) Nodes: Dual 2.4 GHz Intel Xeons, 1GB DDR PC2100 (DDR266) RAM, Intel SE7501BR2 server motherboard with E7501 chipset SCI: 667 MB/s (300 MB/s sustained) Dolphin SCI D337 (2D/3D) NICs, using PCI 64/66, 4x2 torus Quadrics (Elan): 528 MB/s (340 MB/s sustained) Elan3, using PCI-X in two nodes with QM-S16 16-port switch InfiniBand (VAPI): 4x (10Gb/s, 800 MB/s sustained) Infiniserv HCAs, using PCI-X 100, InfiniIO 2000 8-port switch from Infinicon RedHat 9.0 with gcc compiler V 3.3.2, MPI uses MPICH 1.2.5, Berkeley UPC runtime system 2.0  Myrinet (GM) conduit (via testbed at MTU) Nodes*: Dual 2.0 GHz Intel Xeons, 2GB DDR PC2100 (DDR266) RAM Myrinet*: 250 MB/s Myrinet 2000, using PCI-X, on 8 nodes connected with 16- port M3F-SW16 switch RedHat 7.3 with Intel C compiler V 7.1, Berkeley UPC runtime system 2.0  ES80 AlphaServer (Marvel) Four 1GHz EV7 Alpha processors, 8GB RD1600 RAM, proprietary inter- processor connection network Tru64 5.1B Unix, HP UPC V2.1 compiler * via testbed made available courtesy of Michigan Tech

11. IEEE Workshop on HSLN 16 Nov 2004 11 Results Core-level experiments  SCI raw (SISCI API) scipp (PIO benchmark, ping-pong) dma_bench (DMA benchmark, one-way)  GASNet conduits (GASNet test suite) testam (AM round-trip latency)

12. IEEE Workshop on HSLN 16 Nov 2004 12 Results Extended-level experiments  GASNet conduits (GASNet test suite) testsmall (put/get round-trip latency) testlarge (put/get bandwidth) Spike in the Myrinet conduit was consistent over many runs and does seem to be an anomally

13. IEEE Workshop on HSLN 16 Nov 2004 13 Updates UPC-level experiments (where # of threads = # of processors)  IS (Class A) from NAS benchmark IS (Integer Sort), lots of fine-grain communication, low amount of computation  DES Differential Attack Simulator S-DES (8-bit key) cipher (integer-based), bandwidth-intensive application IS BenchmarkDES Benchmark

14. IEEE Workshop on HSLN 16 Nov 2004 14 Conclusions Experimental version of our conduit is available as part of Berkeley UPC V2.0+ release Despite being limited by existing SCI driver from vendor, conduit able to achieve performance fairly comparable to other HPN conduits of GASnet Enhancements to resolve driver limitations being investigated in close collaboration with Dolphin  Support access of all virtual memory on remote node  Minimize transfer setup overhead

15. IEEE Workshop on HSLN 16 Nov 2004 15 Questions and Answers

16. IEEE Workshop on HSLN 16 Nov 2004 16 Appendix - GASNet Latency on Conduits

17. IEEE Workshop on HSLN 16 Nov 2004 17 Appendix - GASNet Throughput on Conduits