Rhea Analysis & Post-processing Cluster Robert D. French NCCS User Assistance

2 Rhea Quick Overview 200 Dell PowerEdge C6220 Nodes – 196 Compute / 4 Login – RHEL 6.4 – 2 x 8-Core Intel Xeon 2.0 GHz Hyperthreading is enabled, so “top” shows 32 CPUs – 64GB of RAM – New 56Gb/s IB Fabric Mounts Atlas – Does not mount Widow Replaces Lens No Preemptive Queue

3 Allocation & Billing Rhea is prioritized as an extra resource for INCITE and ALCC users through the end of the year. – DD Projects may request access 1 node hour charged per node per hour – Ex: 10 nodes for 2 hours = 20 node hours Each project will be awarded 1,000 hours per Month – Separate from Titan / Eos usage – Request more if you run low

4 Rhea Queue Policy Job SizeJob LengthJob LimitsRestricted by 1 – 16 Nodes0 – 12 Hours 12 – 36 Hours 36 – 96 Hours 3 eligible / unlimited running 2 active 1 active User System 17 – 32 Nodes0 – 12 Hours 12 – 36 Hours 2 active 1 active System 33 – 128 Nodes0 – 3 Hours1 activeSystem Should minimize large jobs swamping the system Small runs should complete quickly Request a Reservation for more nodes / longer wall-times

5 Software Stack Most Lens software will already be installed Here are some highlights: – Visualization: ParaView, VisIt, VMD – Compilers: GCC, Intel, and PGI – Scientific Languages: MATLAB, Octave, R, SciPy – Data Management: Globus, BBCP, NetCDF, HDF5, Adios – Debugging: DDT, Vampir, Valgrind Full list of installed software available on our website If you can’t find what you need, just ask!

6 Transitioning to Rhea Titan Lens Widow Now: Titan and Lens mount Widow

7 Transitioning to Rhea Titan Rhea Lens Widow Atlas Soon (mid-to-late November): Titan will mount both Atlas and Widow Move data to Atlas and take advantage of Rhea

8 Transitioning to Rhea Titan Rhea Atlas Near Future: Lens will be decommissioned Rhea will be the center’s viz & analysis cluster Widow

9 Questions?

Spider II Directory Layout Changes Chris Fuson

11 OLCF Center-wide File Systems Spider – Center-wide scratch space – Temporary; not backed-up – Available from compute nodes – Fast access to job-related temporary files and for staging large files to and from archival storage – Contains multiple Lustre file systems

12 Spider I v/s Spider II Spider I Widow [1-3] 240 GB/s 10 PB 3 MDS 192 OSS 1,344 OST Current Center-wide Scratch Decommissioned Early January, 2014 Atlas [1-2] 1 TB/s 30 PB 2 MDS 288 OSS 2,016 OST Available on Additional OLCF Systems Soon Spider II

13 Spider II Change Overview 1.New directory structure – Organized by project – Each project given a directory on one of the atlas filesystems – WORKDIR now within project areas » You may have multiple WORKDIRs » * Requires Change 2.Quota increases – Increased file system size allows for increased quotas 3.All areas purged – To help ensure space available for all projects Before using Spider II, please note the following:

14 Purpose: Batch job I/O Path: -$MEMBERWORK/ 10 TB quota 14 day purge Permissions: -User allowed to change permissions to share within project -No automatic permission changes Spider II Directory Structure ProjectID Member Work World Work Project Work

15 Purpose: Data sharing within project Path: -$PROJWORK/ 100 TB quota 90 day purge Permissions: -Read, Write, Execute access for project members Spider II Directory Structure ProjectID Member Work World Work Project Work

16 Purpose: Data sharing with users who are not members of project Path: -$WORLDWORK/ 10 TB quota 14 day purge Permissions: -Read, Execute for world -Read, Write, Execute for project Spider II Directory Structure ProjectID Member Work World Work Project Work

17 Spider II Directory Structure New directory structure –Organized by project

18 Before Using Atlas Modify scripts to point to new directory structure /tmp/work/$USER $WORKDIR /tmp/proj/ $MEMBERWORK/ Migrate data You will need to transfer needed data onto Spider II (atlas) $PROJWORK/

19 Questions? More information: – –

20 Other Items Dec 17 th - Titan to return to 100% 2013 User Survey – Available on olcf.ornl.gov

21 Thanks for your time.