1. Introduction to the XC30 For Users of Cray’s XT5 and XK7 Aaron Vose 1

2. The hardware is similar, with changes: CPU: Move to from AMD Istanbul/Interlagos to Intel SNB/IVB AMD’s CMT vs. Intel’s HyperThreading Interconnect: XT5’s SeaStar or XE/XK7’s Gemini -> XC30’s Ares The software and environment is very similar as well XT5/XK7 to XC30 Changes Overview 2

3. Fill me in XT5/XK7 AMD CPU Overview 3

4. Fill me in XC30 Intel CPU Overview 4

5. AMD and Intel CPU Quick Comparison 5 AMD Istanbul AMD Interlagos Intel SandyBridge Intel IvyBridge SystemKraken (XT5)Titan (XK7)Eos (XC30)XC30 Socket/Node2121 or 2 FPU/Socket6888+ ALU/Socket61688+ Cores/Socket61688+ Threads/Core111 or 2 DP FP/FPU4 /clock8 /clock 16 /clock

6. Intel’s HyperThreading vs. AMD’s Modules 6 AMD CPU module contains 2 integer cores sharing an FPU: Allocate cores sharing FPU by default with 16 threads/CPU Can get a dedicated FPU per thread by running 8 per CPU Intel CPU can run 2 threads per core with “HyperThreading”: aprun -j 2 -> Two ranks per core (“DualStream”) aprun -j 1 -> One rank per core (“SingleStream”) This is the default

7. XC30’s Ares Interconnect 7 Switch from 3D Torus to Dragonfly topology Adaptive routing avoids network hotspots Node placement not as important Greatly increased global bandwidth Fill me in

8. Software Changes: ACML -> MKL 8 AMD Core Math Library is gone; replaced with: Intel’s Math Kernel Library Update code: call vrda_exp(VL,RF(1,1),RF(1,1)) (ACML) call vdexp(VL,RF(1,1),RF(1,1)) (MKL) To use MKL: module load intel Link with: -L$(MKLROOT)/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core

9. Software Changes: Intel Compiler 9 Intel Compiler: module swap PrgEnv-* PrgEnv-intel ftn / cc / CC commands wrap ifort / icc / icpc Useful Flags: -openmp (Enables OpenMP) -xAVX (Enables AVX) -mkl (Enables MKL)

10. Hardware: Move to Intel SandyBridge / IvyBridge Intel HyperThreading Interconnect: Ares -- Dragonfly Topology Software and environment: Intel MKL Intel Compiler XT5/XK7 to XC30 Changes: Summary 10