1. Using Kure and Killdevil
The Title Slide: Add the name of the presentation, the appropriate division or presenter and date of the presentation.
Mark Reed
Sandeep Sarangi
ITS Research Computing

2. Outline Compute Clusters Logging In File Spaces
Killdevil
Kure
Logging In
File Spaces
User Environment and Applications, Compiling
Job Management
Basic content slide: Add slide title and slide text in the appropriate places. To create a new slide, go to “Insert > New Slide” from the main menu.

3. Links UNC Research Computing Getting started Killdevil page
Getting started Killdevil page
Killdevil FAQ
Getting started Kure page

4. What is a compute cluster? What exactly is Killdevil? Kure?
The Title Slide: Add the name of the presentation, the appropriate division or presenter and date of the presentation. 4

5. What is a compute cluster?
Some Typical Components
Compute Nodes
Interconnect
Shared File System
Software
Operating System (OS)
Job Scheduler/Manager
Mass Storage

6. Compute Cluster Advantages
fast interconnect, tightly coupled
aggregated compute resources
can run parallel jobs to access more compute power and more memory
large (scratch) file spaces
installed software base
scheduling and job management
high availability
data backup

7. Multi-Core Computing
The trend in High Performance Computing is towards multi-core or many core computing.
More cores at slower clock speeds for less heat
Dual and quad core processors are now common.
Soon 64+ core processors will be common

8. Kure A HPC/HTC research compute cluster in RC
Named after the beach in North Carolina
It’s pronounced like the Nobel prize winning physicist and chemist, Madame Curie

9. Kure Compute Cluster Heterogeneous Research Cluster
Hewlett Packard Blades
200+Compute Nodes, mostly
Xeon GHz
Nehalem Microarchitecture
Dual socket, quad core
48 GB memory
over 1800 cores
some higher memory nodes
Infiniband 4x QDR
priority usage for patrons
Buy in is cheap
Storage
/netscr – 197 TB
Isilon space

10. Kure Cont.
The original configuration of Kure was mostly homogeneous but it became increasingly heterogeneous as patrons added to it.
Most (non-patron) compute nodes are 48 GB but there are additional high memory nodes
3 nodes each with 192 GB of memory
2 nodes each with 96 GB of memory
patron nodes with 72 GB of memory

11. Multi-Purpose Killdevil Cluster
High Performance Computing
Large parallel jobs, high speed interconnect
High Throughput Computing (HTC)
high volume serial jobs
Large memory jobs
special nodes for extreme memory
GPGPU computing
computing on Nvidia processors

12. Killdevil Nodes Three types of nodes: compute nodes large memory nodes
GPGPU nodes

13. Killdevil Cluster – Compute Nodes
Intel Xeon processors, Model X5670
Dual socket hex core (12 cores per node)
2.93 GHz processors for each core
48 or 96 GB memory per node

14. Killdevil Cluster – Compute Nodes
Intel Xeon processors, Model E5-2670
Dual socket oct core (16 cores per node)
2.60 GHz processors for each core
64 GB memory per node

15. Killdevil Cluster – Compute Nodes
68 nodes with 64 GB memory per node
604nodes with 48 GB memory per node
68 nodes with 96 GB memory
total of 740 nodes with 9152 cores
plus GPU and large memory nodes
So 774 nodes with 9600 cores total

16. Killdevil Extreme Memory Nodes
2 nodes each with 1 TB of memory
extremely large shared memory node!
Intel Xeon Model X7550
32 cores per node
2.0 GHz processors
Use the bigmem queue

17. Killdevil GPGPU Computing
General Purpose computing on Graphics Processing Units (GPGPU)
32 compute nodes are paired with 64 GPU’s in a 2:1 ratio
this is configurable and may vary
compute nodes are Intel Xeon X5650, 2.67 GHz, 12 cores, 48 GB memory nodes
GPUs are Nvidia Tesla (M2070), each with 448 compute cores
Use the gpu queue

18. Infiniband Connections
Connection comes in single (SDR), double (DDR), and quad data rates (QDR). Now also FDR and EDR.
Killdevil is QDR.
Single data rate is 2.5 Gbit/s in each direction per link.
Links can be aggregated - 1x, 4x, 12x.
Killdevil is 4x.
Links use 8B/10B encoding —10 bits carry 8 bits of data — useful data transmission rate is four-fifths the raw rate. Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s respectively.
Data rate for Killdevil is 32 Gb/s or 4 GB/s (4x QDR).

19. Login to Killdevil/Kure
Use ssh to connect:
ssh killdevil.unc.edu
ssh kure.unc.edu
SSH Secure Shell with Windows
see
For use with X-Windows Display:
ssh –X killdevil.unc.edu or ssh –X kure.unc.edu
ssh –Y killdevil.unc.edu or ssh –Y kure.unc.edu
Off-campus users (i.e. domains outside of unc.edu) must use VPN connection

20. File Spaces

21. Killdevil File Spaces Home directories Scratch Space Mass Storage
/nas02/home/<a>/<b>/<onyen>
a = first letter of onyen, b = second letter of onyen
hard limit of 15 GB
Scratch Space
NOT backed up
purged regularly (21 days or less)
run jobs with large output in these spaces
/netscr – 197 TB (tuned for small files)
/lustre – 126 TB (tuned for large files)
Mass Storage
~/ms

22. Kure File Spaces Home directories Scratch Space Mass Storage
/nas02/home/<a>/<b>/<onyen>
a = first letter of onyen, b = second letter of onyen
hard limit of 15 GB
Scratch Space
NOT backed up
purged regularly (21 days or less)
run jobs with large output in these spaces
/netscr – 197 TB (tuned for small files)
Mass Storage
~/ms

23. File System Notes
Note that the same home directory is mounted on Killdevil and Kure
Check your home file space usage with the quota command
quota –s (this uses more readable units)
Lustre file space in Killdevil is attached via Infiniband and may be faster
Best practice for jobs with large output is to run them in scratch space, tar and compress results, and store them in mass storage.

24. Mass Storage “To infinity … and beyond” - Buzz Lightyear
long term archival storage
access via ~/ms
looks like ordinary disk file system – data is actually stored on tape
“limitless” capacity
data is backed up
For storage only, not a work directory (i.e. don’t run jobs from here)
if you have many small files, use tar or zip to create a single file for better performance
Sign up for this service on onyen.unc.edu
“To infinity … and beyond” Buzz Lightyear

25. User Environment and Applications, Compiling Code
Modules

26. Modules
The user environment is managed by modules. They provide a convenient way to access software applications
Modules modify the user environment by modifying and adding environment variables such as PATH or LD_LIBRARY_PATH
Typically you set these once and leave them
Note there are two module settings, one for your current environment and one to take affect on your next login (e.g. batch jobs running on compute nodes)

27. Common Module Commands
module avail
module avail apps
module help
Change Current Shell
module list
module add
module rm
Login version
module initlist
module initadd
module initrm
More on modules see

28. Compiling on Killdevil/Kure
Serial Programming
Suites for C, C++, Fortran90, Fortran77, etc
Intel Compilers
icc, icpc, ifort
GNU
gcc, g++, gfortran
Portland Group (PGI)
pgcc, pgCC, pgf90, pgf77
Generally speaking the Intel or PGI compilers will give slightly better performance

29. Parallel Jobs with MPI
There are three implementations of the MPI standard installed on both systems:
mvapich
mvapich2
openmpi
Performance is similar for all three, all three run on the IB fabric. Mvapich is the default. Openmpi and mvapich2 have more the MPI-2 features implemented.

30. Compiling MPI programs
Use the MPI wrappers to compile your program
mpicc, mpiCC, mpif90, mpif77
the wrappers will find the appropriate include files and libraries and then invoke the actual compiler
for example, mpicc will invoke either gcc, pgcc or icc depending upon which module you have loaded

31. Compiling on Killdevil/Kure
Parallel Programming
MPI (see previous page)
OpenMP
Compiler flag:
-openmp for Intel
-fopenmp for GNU
-mp for PGI
Must set OMP_NUM_THREADS in submission script

32. Job Scheduling and Management
The Title Slide: Add the name of the presentation, the appropriate division or presenter and date of the presentation.
LSF 32

33. What does a Job Scheduler and batch system do?
Manage Resources
allocate user tasks to resource
monitor tasks
process control
manage input and output
report status, availability, etc
enforce usage policies

34. Job Scheduling Systems
Allocates compute nodes to job submissions based on user priority, requested resources, execution time, etc.
Many types of schedulers
Load Sharing Facility (LSF) – Used by Killdevil/Kure
IBM LoadLeveler
Portable Batch System (PBS)
Sun Grid Engine (SGE)
Simple Linux Utility for Resource Management (SLURM)

35. LSF
All Research Computing clusters use LSF to do job scheduling and management
LSF (Load Sharing Facility) is a (licensed) product from Platform Computing (now owned by IBM)
Fairly distribute compute nodes among users
enforce usage policies for established queues
most common queues: int, now, week, month
RC uses Fair Share scheduling, not first come, first served (FCFS)
LSF commands typically start with the letter b (as in batch), e.g. bsub, bqueues, bjobs, bhosts, …
see man pages for much more info!

36. Simplified view of LSF
job dispatched to run on available host which satisfies job requirements
Jobs Queued
job_J
job_F
myjob
job_7
Login Node
job routed to queue
bsub –q week myjob
user logged in to login node submits job

37. Running Programs on Killdevil
Upon ssh to Killdevil/Kure, you are on the Login node.
Programs SHOULD NOT be run on Login node.
Submit programs to one of the many, many compute nodes.
Submit jobs using Load Sharing Facility (LSF) via the bsub command.

38. Common batch commands bsub - submit jobs
bqueues – view info on defined queues
bqueues –l week
bkill – stop/cancel submitted job
bjobs – view submitted jobs
bjobs –u all
bhist – job history
bhist –l <jobID>

39. Common batch commands bhosts – status and resources of hosts (nodes)
bpeek – display output of running job
Use man pages to get much more info!
man bjobs

40. Submitting Jobs: bsub Command
Submit Jobs - bsub
Run large jobs out of scratch space, smaller jobs can run out of your home space
bsub [-bsub_opts] executable [-exec_opts]
Common bsub options:
–o <filename>
–o out.%J
-q <queue name>
-q week
-R “resource specification”
-R “span[ptile=8]”
-n <number of processes>
used for parallel, MPI jobs

41. Two methods to submit jobs:
bsub example: submit the executable job, myexe, to the week queue and redirect output to the file out.<jobID> (default is to mail output)
Method 1: Command Line
bsub –q week –o out.%J myexe
Method 2: Create a file (details to follow) called, for example, myexe.bsub, and then submit that file. Note the redirect symbol, <
bsub < myexe.bsub

42. Method 2 cont.
The file you submitted will contain all the bsub options you want in it, so for this example myexe.bsub will look like this
#BSUB –q week
#BSUB –o out.%J
myexe
This is actually a shell script so the top line could be the normal #!/bin/csh, etc and you can run any commands you would like.
if this doesn’t mean anything to you then nevermind :)

43. Batch Command Line Method
Parallel Job example
Batch Command Line Method
bsub –q week –o out.%J -n 64 mpirun myParExe
Batch File Method
bsub < myexe.bsub
where myexe.bsub will look like this
#BSUB –q week
#BSUB –o out.%J
#BSUB –n 64
mpirun myParallelExe

44. Minor Killdevil caveats
Memory limits: Killdevil has a default memory limit of 4 GB for a job- if you need more than the default, use the “-M” LSF option:
bsub –q week –o out.%J –M 9 myExe
PI groups: On Killdevil when you submit a job make sure you use the correct PI group (only applicable if you belong to more than one PI group) by using the “-G” LSF option:
bsub –q week –G itsrc_grp myExe

45. Minor Killdevil caveats (cont’d)
Using the correct PI group is important for bookkeeping we do in regard to cluster usage by the PI groups
To check the PI groups to which you belong:
bugroup | grep <onyen>

46. Interactive Jobs
To run long shell scripts on Kure, use int (interactive) queue
bsub –q int –Ip /bin/bash
This bsub command provides a prompt on compute node
Can run program or shell script interactively from compute node
on Killdevil use hour or day as needed
bsub –q hour –Ip /bin/bash

47. Specialty Scripts
There are specialty scripts provided on Kure for the user convenience.
Batch scripts
bmatlab, bsas, bstata
X-window scripts
xmatlab, xsas, xstata
Interactive scripts
imatlab, istata
Killdevil only provides the *matlab scripts listed above

48. MPI/OpenMP Training
Courses are taught throughout year by Research Computing
See schedule for next course
MPI
OpenMP

49. Further Help with Killdevil/Kure
More details can be found on the Getting Started help documents:
Killdevil
Kure
For assistance with Killdevil/Kure, please contact the ITS Research Computing group
Phone: HELP
Submit help ticket at
For immediate assistance on a particular command, see the manual pages
man <command>