1. Frank Mueller North Carolina State University
ARC Cluster
Frank Mueller North Carolina State University

2. PIs & Funding Frank Mueller Vincent Freeh Helen Gu Xuxian Jiang
NSF funding level: $550k
NCSU: $60k (ETF) + $60+k (CSC)
NVIDIA: donations ~$30k
PIs/co-PIs:
Frank Mueller
Vincent Freeh
Helen Gu
Xuxian Jiang
Xiaosong Ma
Contributors:
Nagiza Samatova
George Rouskas

3. ARC Cluster: In the News
“NC State is Home to the Most Powerful Academic HPC in North Carolina” (CSC News, Feb 2011)
“Crash-Test Dummy For High-Performance Computing” (NCSU, The Abstract, Apr 2011)
“Supercomputer Stunt Double” (insideHPC, Apr 2011)

4. Purpose
Create a mid-size computational infrastructure to support research in areas such as:

5. Researchers Already Active
Many people:
Groups from within NCSU:
CSC, ECE, Mech+AeroSpace, Physics, Chem/Bio Engineering, Materials, Operations Research
ORNL
VT, U Arizona
Tsinghua University, Beijing, China
etc.

6. ARC - A Root Cluster System Overview Mid Tier Front Tier Back Tier
Head/Login Nodes
IB Switch Stack
GEther Switch Stack
I/O Nodes
Storage Array
Compute/Spare Nodes
PFS Switch Stack
Interconnect
ARC - A Root Cluster
SSD+
SATA

7. Hardware 108 Compute Nodes
2-way SMPs with AMD Opteron/ Intel Sandy/Ivy/Broadwell procs
8 cores/socket, 16 cores/node
16-64 GB DRAM per node
1728 compute cores available

8. Interconnects Gigabit Ethernet interactive jobs, ssh, service
Home directories
40Gbit/s Infiniband (OFEDstack)
MPI Communication
Open MPI, MVAPICH
IP over IB

9. NVIDIA GPUs C/M2070 C2050 GTX480 K20c GTX780 GTX680 K40c GTX Titan X

10. Solid State Drives Most compute nodes equipped with
OCZ RevoDrive 120GB PCIe Crucial_CT275MX3 275GB SATA3 SamsungPM1725 Series 1.6TB NVMe

11. File Systems Available Today:
NFS home directories over Gigabit Ethernet
Local per-node scratch on spinning disks (ext3)
Local per-node 120GB SSD (ext2)
Parallel File System
PVFS2
Separate dedicated nodes are available for parallel filesystems
4 clients, one of them doubles as MDS

12. Power Monitoring Watts Up Pro Serial and USB available.
Connected in groups of:
Mostly 4 nodes (sometimes just 3)
2x 1 node
1 w/ GPU
1 w/o GPU

13. Software Stack Additional packages and libraries upon request but…
Not free?  you need to pay
License required?  you need to sign it
Installation required?  you need to
Test it
Provide install script
check ARC website  constantly changing

14. Base System OpenHPC 1.2.1 (over CentOS 7.2) Batch system: Slurm
All compilers and tools are available on the compute nodes
Gcc, gfortran, java…

15. MPI mpicc MVAPICH Open MPI Infiniband support
Already in your default PATH
mpicc
Open MPI
Operates over Infiniband
Activate: module switch mvapich2 openmpi

16. OpenMP The "#pragma omp" directive in C programs works.
gcc -fopenmp -o fn fn.c

17. CUDA SDK Ensure you are using a node with a GPU
Several types available to fine tune for your applications needs:
Well-performing single or double precision devices.
Active: module load cuda

18. PGI Compiler (Experimental)
pgcc, pgCC
pgf77, pgf99, pghpf
OpenACC support
module load pgi64

19. Virtualization via LXD
Uses Linux containers
Goal: To allow a user to configure their own environment
User gets full root access within container
Much smaller footprint than VM
Docker support inside LXD
Requires more space
May run full VM inside:
VirtualBox…
Full VM space required

20. Job Submission cannot SSH to a compute node
must use srun to submit jobs
Either as batch
or interactively
Presently there are “hard” limits for job times and sizes. In general, please be considerate of other users and do not abuse the system.
There are special queues for nodes with certain CPUs/GPUs

21. PBS Basics On the login node: to submit a job: srun …
to list jobs: squeue
to list details of your job: scontrol…
to delete/cancel/stop your job: scancel…
to check node status: sinfo

22. qsub Basics
srun -n 32 --pty /bin/bash # get 32 cores (2 nodes) in interactive mode
srun -n 16 -N 1 --pty /bin/bash # get 1 interactive node with 16 cores
srun -n 32 -N 2 -w c[30,31] --pty /bin/bash #run on nodes 30+31
srun -n 64 -N 4 -w c[30-33] --pty /bin/bash #run on nodes 30-33
srun -n 64 -N 4 -p opteron --pty /bin/bash #run on any 4 opteron nodes
srun -n 64 -N 4 -p gtx480 --pty /bin/bash #any 4 nodes w/ GTX 480 GPUs

23. Listing your nodes
Once your job begins, $SLURM_NODELIST has list of nodes allocated to you
MPI is already integrated with Slurm. Simply using prun … will automatically use all requested processes directly from Slurm.
For example, a CUDA programmer that wants to use 4 GPU nodes:
~]$ srun –N 4 –ppy /bin/bash –p gtx480
~]$ echo $SMURL_NODELIST
c[ ]
---SSHing between these nodes FROM WITHIN the Slurm session is allowed---

24. Hardware in Action
4 racks in server room

25. Temperature Monitoring
It is the user’s responsibility to maintain room temperatures below 80 degrees while utilizing the cluster.
ARC website has links to online browser- based temperature monitors.
And the building staff have pagers that will alarm 24/7 when temperatures exceed the limit.

26. Connecting to ARC ARC access is restricted to on-campus IPs only.
If you ever are unable to log in (connection gets dropped immediately before authentication) then this is likely the cause.
Non-NCSU users may request remote access by providing a remote machine that their connections must originate from.

27. Summary Your ARC Cluster@Home: What can I do with it?
Primary purpose: Advance Computer Science Research (HPC and beyond)
Want to run a job over the entire machine?
Want to replace parts of the software stack?
Secondary purpose: Service to sciences, engineering & beyond
Vision: Have domain scientists work w/ Computer Scientists on code
Equipment donations welcome 
Ideas how to improve ARC?  let us know
Qs?  send to mailing list (once you have an account)
request an account: dfiala<at>ncsu.edu
Research topic, abstract, and compute requirements/time
Must include your unity ID
NCSU Students: Advisor sends as means of their approval
Non-NCSU: same + preferred username + hostname(your remote login location.

28. Slides provided by David Fiala Edited by Frank Mueller Current as of Aug 21, 2017