1. Viglen Rocks Cluster Training University of Liverpool
David Power
HPC Systems Engineer

2. Viglen Rocks Cluster Training
Rocks Overview
Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk Quotas
Using Rocks 2

3. Rocks Overview Rocks Overview Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk Quotas
Using Rocks 3

4. Rocks Overview
Rocks is an open-source Linux cluster distribution that enables end users to easily build computational clusters, grid endpoints and visualization tiled-display walls
It was started by National Partnership for Advanced Computational Infrastructure and the SDSC (San Diego Supercomputer Centre) in 2000
It is based on CentOS with a modified anaconda installer that simplifies mass installation onto many computers.
Installations are customised with additional software packages called rolls

5. Adding Software to Compute Nodes
Rocks Overview
Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk Quotas
Using Rocks 5

6. Building RPMs in Rocks Use rpmbuild if a .spec file is provided
Use the Rocks built in mechanism to create RPMs from directories.
Build your software from source and install on the frontend:
configure
make
make install
Or, just untar a binary bundle

7. Building RPMs in Rocks Creating RPM packages in Rocks:
rocks create package <path> <package-name>
rocks create package /opt/myapp myapp

8. Contribute the RPM
Your distribution looks for packages from Rolls and in the contrib area.
Copy your RPMs in to the contrib directory.
cp myapp x86_64.rpm /export/rocks/install/contrib/5.X/x86_64/RPMS

9. Extend XML
Compute Node installation profile can be extended with additional applications by creating an extend-compute.xml file and editing this:
cd /export/rocks/install/site-profiles/5.1/nodes/
cp skeleton.xml extend-compute.xml
vi extend-compute.xml

10. Add a package to the Compute Profile
original
modified (with myapp)
<kickstart>
<description>
Skeleton XML Node
</description>
<changelog>
</changelog>
<!—
<package>myapp</package>
<package></package>
-->
<post>
</post>
</kickstart>

11. Rebuild Distribution RPM package is present in the contrib directory
XML node file is extended (and updated)
Now we need to rebuild the distribution (apply changes above)
This must be done in /export/rocks/install
$ cd /export/rocks/install/
$ rocks create distro

12. Reinstall the nodes PXE Boot Network boot is first in BIOS boot order
Set Rocks Boot action to install
Reboot the host
$ rocks set host boot
<host>
action=<boot-action>
compute-0-0
action=install
(or action=os to cancel reinstall)

13. Reinstall the nodes Files updated in /tftpboot
/tftpboot/pxelinux/pxelinux.cfg/XXYYZZAABBCC
default rocks
prompt 0
label rocks
localboot 0

14. Reinstall the nodes Initiate a reboot on the nodes to re-install.
$ rocks run host
<host>
<command>
compute-0-0
/sbin/init 6

15. Check the PXE boot action
List the current PXE boot action on all the nodes.
$ rocks list host boot
HOST ACTION
frontend: os
compute-0-0: os
compute-0-1: os
compute-1-0: os
compute-1-1: install
compute-2-0: os
compute-2-1: os

16. <post> section
The <post> section of the extend-compute.xml can be used to do any post installation configuration
The file is XML based and certain characters are not permitted:
Special Character XML syntax
> >
< <
“ "
& &
<post>
...
/path/to/daemon > /path/to/outputfile
</post>

17. <post> section
You can put in standard shell commands in to the <post> section
<post>
...
chkconfig --level 2345 rocks-grub off
</post>
You can also put in python commands
<eval shell=“python”>
python shell code
</eval>

18. Append files in the <post> section
...
<file name=“/etc/motd” mode=“append”>
Updated MOTD message
</file>
</post>

19. Create files in the <post> section
...
<file name=“/etc/rc.d/rocksconfig.d/post-99- dofinalconf” mode=“create” perms=“0755”>
#!/bin/bash
cd /share/apps/myapp
cp file /opt/myapp
rm -f /etc/rc.d/rocksconfig.d/post-99-dofinalconf
</file>
</post>

20. Problems with adding new packages
Verify the syntax of the xml extend-compute file
$ xmllint –noout /path/to/extend-compute.xml
Are kickstart files being generated correctly?
$ /export/rocks/sbin/kickstart.cfg
--client=compute-0-0

21. Sync Files across Cluster (411)
411 is used to keep files consistent across the cluster
Used for passwd/shadow/group etc
Can be used for other files also
/var/411/Files.mk
FILES_NOCOMMENT = /etc/passwd \
/etc/group \
/etc/shadow
$ rocks sync users

22. Torque/Maui Rocks Overview Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk Quotas
Using Rocks 22

23. Software Included
Torque
Maui
mpiexec
pbstools
pbspython ftp://ftp.sara.nl/pub/outgoing/

24. Installed daemons Frontend Compute maui pbs_server
pbs_mom (not running)
mpiexec (mostly for the man-page)
Compute
pbs_mom
mpiexec

25. Scheduling Features Maui provides a rich set of scheduling features
Maui can schedule on
CPUs
Walltime
Memory
Disk size
Network Topology

26. Needed Job Info
For scheduling to be useful one needs info about the jobs
At least number of cpus and walltime
Memory requirement also useful
#PBS –lwalltime=HH:MM:SS
#PBS –lnodes=10:ppn=8
#PBS –lpmem=1gb

27. Memory handling on Linux
Torque/Maui supports two memory specification types, (p)mem and (p)vmem on linux.
pmem is not enforced
Used only as information to the sceduler
pvmem is enforced
Terminating procs that cross the limit
Limits vmem size setting ulimit –v on the processes

28. Torque Hacking Torque is installed in /opt/torque
qmgr is the torque mgt. command
Note: backup your working config
$ qmgr –c “print server” > /tmp/pbsconfig.txt

29. Torque Hacking Roll back to escape from a messed up system:
$ qterm
$ pbs_server –t create
$ qmgr < /tmp/pbsconfig.txt
This will bring you back to where you started
Note: this will wipe the whole queue setup and all current queued and running jobs will be lost!

30. Maui Hacking Most things can be achieved by modifying:
/etc/maui/maui.cfg
Maui needs a restart after changing the config file:
$ service maui restart

31. Should I edit Torque or Maui?
If you can achieve the same thing by changing either torque or maui, use maui.
Restarting maui is rather lightweight operation, and seldom causes problems for live systems.
Restarting pbs_server can make the system oscillatory for a few minutes
pbs_server needs to contact all pbs_moms to get back in state.

32. Prioritising Short Jobs
Often it is useful to give shorter jobs higher priority (maui configuration).
Use the XFACTOR feature in maui rather than torque queues with different priorities
XFACTORWEIGHT 1000

33. Prioritising Short Jobs
XFACTOR is defined as
XFACTOR=(walltime+queuetime)/walltime
XFACTOR will increase faster for shorter walltimes thus giving higher priorities for short jobs.
Depends on users giving reasonable walltime limits.

34. Prioritising large jobs (maui)
In a cluster with a diverse mix of jobs it is useful to prioritize the large jobs and make the smaller ones fill the gaps.
CPUWEIGHT 1000
MEMWEIGHT 100
This should be combined with fairshare to avoid starving users falling outside this prioritisation.

35. Fairshare (maui) Also known as Can be done on several levels
“keeping all users equally unhappy”
Can be done on several levels
users, groups …
Set a threshold
USERCFG[DEFAULT] FSTARGET=10
FSWEIGHT 100
Users having more than 10% will get reduced priority and vice versa.

36. Adjusting your policy
You can play with the weights to fine tune your scheduling policies
XFACTORWEIGHT 100
FSWEIGHT 1000
RESWEIGHT 10
CPUWEIGHT 1000
MEMWEIGHT 100
Analyse the prioritisation with
$ diagnose -p

37. Job Node Distribution Default is MINRESOURCE Spread or pack?
Run on the nodes which gives at least unused resources.
Spread or pack?
NODEALLOCATIONPOLICY PRIORITY
Select the most busy nodes
NODECFG[DEFAULT] PRIORITYF=JOBCOUNT
Select the least busy nodes
NODECFG[DEFAULT] PRIORITYF=-1.0*JOBCOUNT

38. Node Access Policy Default access policy is SHARED
Can choose to limit this to SINGLEJOB or SINGLEUSER, i.e.
NODEACCESSPOLICY SINGLEUSER
Single user access prevents users from stepping on each others toes while allowing good utilisation for serial jobs.

39. Throttling Policies
Sometimes one needs to limit the user from taking over the system…
MAXPROC
MAXNODE
MAXPS
MAXJOB
MAXIJOB (I indicating idle, scheduled but not running)
All can be set for all individuals users and groups
USERCFG[DEFAULT]
USERCFG[USERA] MAXPROC=16

40. Debugging and Analysing
Lots of tools:
pbsnodes node status
qstat –f all details of a job
diagnose –n -- node status from maui
diagnose –p -- job priority calculation
showres –n -- job reservation per node
showstart estimated job start time
checkjob check job status
checknode check node status

41. Example: Express Queue
Goal: Supporting development and job script testing, but prevents misuse
Basic philosophy:
Create a separate queue
Give it the highest priority
Throttle is so it is barely usable

42. Example: Express Queue
Create the queue with qmgr
create queue express
set queue express queue_type = Execution
set queue express resources_max.walltime = 08:00:00
set queue express resources_default.nodes = 1:ppn=8
set queue express resources_default.walltime = 08:00:00
set queue express enabled = True
set queue express started = True

43. Example: Express Queue
Increase the priority and limit the usage (maui.cfg)
CLASSWEIGHT 1000
CLASSCFG[express] PRIORITY=1000 MAXIJOB=1 \
MAXJOBPERUSER=1 QLIST=express QDEF=express \
QOSCFG[express] FLAGS=IGNUSER
This will allow users to test job scripts and run interactive jobs with a good turnaround.

44. Check jobs running on the nodes
Check for jobs running on the cluster (showq):
nodes]# showq
ACTIVE JOBS
JOBNAME USERNAME STATE PROC REMAINING STARTTIME
asmith Running :13:35:10
klangfeld Running :22:01:12
2 Active Jobs of 948 Processors Active (19.41%)
16 of 79 Nodes Active (20.25%)

45. Check host jobs are running on
Whats nodes are the jobs running on: (checkjob JOBID) :
nodes]# checkjob 1473
checking job 1473
...
Req[0] TaskCount: 120 Partition: DEFAULT
Network: [NONE] Memory >= 0 Disk >= 0 Swap >= 0
Opsys: [NONE] Arch: [NONE] Features: [NONE]
Allocated Nodes:
[compute-2-31:8][compute-2-30:8][compute-2-29:8][compute- 2-28:8]
[compute-2-27:8][compute-2-26:8][compute-2-25:8][compute- 2-24:8]
[compute-2-23:8][compute-2-35:8][compute-2-34:8][compute- 2-33:8]
[compute-2-32:8][compute-2-21:8][compute-2-20:8]

46. Take nodes OFFLINE Take nodes offline (no jobs will be allocated) :
nodes]# pbsnodes -o compute-2-0
nodes]# pbsnodes compute-2-0
compute-2-0
state = offline
np = 12
ntype = cluster
status = opsys=linux,uname=Linux compute-2-0.local el5 #1 SMP Wed Jan 21 10:41:14 EST x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=248891,totmem= kb,availmem= kb,physmem= kb,ncpus=12,loadave=0.00,ne tload= ,state=free,jobs=,varattr=,rectime=

47. Bring nodes ONLINE Take nodes offline (no jobs will be allocated) :
nodes]# pbsnodes -c compute-2-0
nodes]# pbsnodes compute-2-0
compute-2-0
state = free
np = 12
ntype = cluster
status = opsys=linux,uname=Linux compute-2-0.local el5 #1 SMP Wed Jan 21 10:41:14 EST x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=248981,totmem= kb,availmem= kb,physmem= kb,ncpus=12,loadave=0.00,ne tload= ,state=free,jobs=,varattr=,rectime=

48. Create Queues Create additional queues using qmgr
qmgr -c "create queue infiniband“
qmgr -c "set queue infiniband queue_type = Execution“
qmgr -c "set queue infiniband enabled = True“
qmgr -c "set queue infiniband started = True“
qmgr -c "set queue infiniband resources_default.neednodes = ib“
qmgr -c "create queue primare“
qmgr -c "set queue primare queue_type = Execution“
qmgr -c "set queue primare enabled = True“
qmgr -c "set queue primare started = True“
qmgr –c “set queue default resources_default.neednodes = ethernet”

49. Create Queues Assign resources to nodes:
In file: /opt/torque/server_priv/nodes
...
compute-1-7 np=8 ethernet
compute-1-8 np=8 ethernet
compute-1-9 np=8 ethernet
compute-2-0 np=12 ib
compute-2-10 np=12 ib
compute-2-11 np=12 ib
compute-2-12 np=12 ib

50. Create Queues Verify nodes report resources correctly:
~]# pbsnodes compute-1-0
compute-1-0
state = free
np = 8
properties = ethernet
~]# pbsnodes compute-2-0
compute-2-0
np = 12
properties = ib

51. Queue Priority Set the priority levels for the queues:
In File: /opt/maui/maui.cfg
...
CLASSWEIGHT
CLASSCFG[default] PRIORITY=1
CLASSCFG[infiniband] PRIORITY=1
CLASSCFG[primare] PRIORITY=10000
Save and restart maui
service maui restart

52. Check Job Priorities (As root) use the diagnose command:
~]# diagnose -p
diagnosing job priority information (partition: ALL)
Job PRIORITY* Cred(Class) Serv(QTime)
Weights ( 1) 1( 1)
(10000) 0.0( 0.0)
( 1.0) 15.5( 0.2)
( 1.0) 7.7( 0.1)
Percent Contribution (100.0) 0.0( 0.0)
*indicates system priority set on job
Red text indicates a job on high priority queue (classcgf)

53. Submitting to the queues
Submit to Infiniband Queue
#PBS -q infiniband
Submit to Primare Queue
#PBS -q primare
Submit to Primare Queue (on IB nodes)
#PBS -l nodes=NN:ppn=PP:ib
Submit to Primare Queue (on Ethernet nodes)
#PBS -l nodes=NN:ppn=PP:ethernet

54. Restricting User Access to Queue
Allow only users from the primare group use primare Q
qmgr -c "set queue primare acl_group_enable=true"
qmgr -c "set queue primare acl_groups=primare"

55. Summary Limit the number of queues.
You need good information about walltime.

56. IPMI Management Rocks Overview Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk Quotas
Using Rocks 56

57. IPMI Overview
IPMI is a standard which defines a set of common interfaces to monitor system health and manage the system.
IPMI operates independent of the OS
IPMI operates even if the monitored system is powered off, but connected to a power source
IPMI BMCs can be communicated to using IP addressing or queried locally.

58. Configuring the IPMI Module
RPMs installed:
OpenIPMI
OpenIPMI-tools (inc ipmitool)
OpenIPMI-libs
Kernel Modules required:
ipmi_msghandler
ipmi_devinf
ipmi_si

59. Configuring the IPMI Module
Set the IP address (static):
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr
ipmitool lan set 1 netmask
ipmitool lan set 1 defgw ipaddr
Set the IP address (dhcp):
ipmitool lan set 1 ipsrc dhcp

60. Configuring the IPMI Network
IPMI modules part of the Rocks Network
Edit the /etc/hosts.local file
compute-ipmi-0-0
compute-ipmi-0-1
Run dbreport hosts or rocks report hosts
Sync the /etc/hosts file using 411 (/var/411/Files.mk)
$ rocks sync users

61. Configuring the IPMI Network
IPMI Modules on a separate subnet
Step 1: Add a network
$ rocks add network <network-name>
<network>
<subnet>
$ rocks add network ipmi

62. Configuring the IPMI Network
List the networks on your Rocks cluster
$ rocks list network
NETWORK SUBNET NETMASK
private:
public:
ipmi:

63. Configuring the IPMI Network
Step 2: Add the interfaces through Rocks
Host first must be installed
Then secondary NICs can be added
After all hosts are configured just re-install

64. Configuring the IPMI Network
Add host interface:
$ rocks add host interface <host> <iface>
ip=<address> subnet=<name> gateway=<address>
name=<hostname>
$ rocks add host interface compute-0-0 ipmi
ip= subnet=ipmi gateway=ipmi
gateway=1 name=compute-ipmi-0-0

65. Configuring the IPMI Network
List host interface:
$ rocks list host interface compute-0-0
SUBNET IFACE MAC IP NETMASK GATEWAY MODULE NAME
private eth0 00:15:17:79:d3:c e1000e compute-0-0
Ipmi ipmi ipmi-0-0

66. (Alternative) Configuring the IPMI Network
Add hosts to /etc/hosts.local:
compute-1-0-ipmi
compute-1-1-ipmi
compute-1-2-ipmi
compute-1-3-ipmi
Regenerate the /etc/hosts file (which will append all entries from /etc/hosts.local
$ dbreport hosts > /etc/hosts
...
# import from /etc/hosts.local

67. Check the IPMI serivice is running
Use the tentakel command:
tentakel -g rack2 chkconfig --list '| grep ipmi'
### compute-2-35(stat: 0, dur(s): 0.16):
ipmi :off 1:off 2:on 3:on 4:on 5:on 6:off
### compute-2-23(stat: 0, dur(s): 0.17):
If any nodes have the service set to off, turn on using:
tentakel -g rack2 chkconfig --level 2345 ipmi on

68. Using IPMItool Chassis Status ipmitool chassis status
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event : ac-failed
Chassis Intrusion : active
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false

69. Using IPMItool Chassis Status over the network
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 chassis status
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event : ac-failed
Chassis Intrusion : active
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false

70. Using IPMItool Reset the IPMI Module (sometimes causes network issues)
ipmitool mc reset cold
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 mc reset cold
Then reload network modules on nodes:
rmmod e1000e ; modprobe e1000e

71. Using IPMItool Controlling the Power Initiate a clean shutdown
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power soft
Power Off
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power off
Power On
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power on
Power Cycle
ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power cycle

72. Implementing Disk/User Quotas
Rocks Overview
Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk/User Quotas
Using Rocks 72

73. Disk Quotas (general)
In order to use disk quotas, they must first be enabled:
Modify /etc/fstab
Remount the filesystem(s)
Run quotacheck
Assign Quotas

74. Disk Quotas (general) Modify /etc/fstab
Edit the line with your home directory
LABEL=/state/partition
On this line change the mount options in include grpquota and usrquota
defaults -> grpquota,usrquota,defaults
Reboot the machine

75. Disk Quotas (ext3) When the system has come back up, run quotacheck
This examines quota-enabled file systems, building a table of the current disk usage for each one.
quotacheck –guvma
-g -- group quotas
-u -- user quotas
-v -- report the operation as it progresses
-m -- do not remount filesystems as RO
-a -- check all mounted non-NFS filesystems

76. Disk Quotas (ext3) Turn File system quotas on Setup a user quota
quotaon –guva
Setup a user quota
edquota –u <username>
Modify the number of days for the hard and soft limits
edquota –t
Propagate these settings to all other users
edquota –p <user_with_updated_quota> -u users..

77. Disk Quotas (ext3) Editing the quota limits (edquota)
Disk quotas for user viglen (uid 501):
Filesystem blocks soft hard inodes soft hard
/dev/sda
Blocks: The amount of space in 1K blocks the user is currently using.
Inodes: The number of files the user is currently using.
Soft Limit: The maximum blocks/inodes a quota user may have on a partition. The role of a soft limit changes if grace periods are used. When this occurs, the user is only warned that their soft limit has been exceeded. When the grace period expires, the user is barred from using additional disk space or files. When set to zero, limits are disabled.
Hard Limit: The maximum blocks/inodes a quota user may have on a partition when a grace period is set. Users may exceed a soft limit, but they can never exceed their hard limit.

78. Disk Quotas (ext3) Reporting on quota usage:
repquota /state/partition1
Useful TIP: Display user quotas at their login (/etc/profile)
quota –u $USER

79. Disk Quotas (XFS) Confirm quotas are enabled xfs_quota –x –c state
*[Steps below assume /etc/fstab has been updated and the system has been rebooted]*
Confirm quotas are enabled
xfs_quota –x –c state
Check the current space usage
xfs_quota –x –c quot

80. Disk Quotas (XFS)
Set user limits (where username is the user account name)
Limits below implement a 6GB hard limit (cant write more than 6GB) and a 5GB soft limit (warnings issued after 5GB)
xfs_quota –x /state/partition1
xfs_quota> limit bsoft=5g bhard=6g username
xfs_quota> quota -h username
Disk quotas for User username(id)
Filesystem Blocks Quota Limit Warn/Time Mounted on /dev/sdb G 5G 6G 00 [------] /state/partition1

81. Disk Quotas (XFS) Disable user quota, set the limits back to 0
xfs_quota –x /state/partition1
xfs_quota> limit bsoft=0 bhard=0 username
xfs_quota> quota -h username
Disk quotas for User username(id)
Filesystem Blocks Quota Limit Warn/Time Mounted on /dev/sdb G [------] /state/partition1

82. Disk Quotas (XFS) Report on XFS Quota Usage
scripts]# xfs_quota -x
xfs_quota> report
User quota on /state/partition1 (/dev/sdb1)
Blocks
User ID Used Soft Hard Warn/Grace
root [0 days]
maui [ ]
icr [ ]
biouser [ ]
viglen [ ]
klangfeld [ ]
pmills [ ]
troc [ ]
ngardiner [ ]

83. Implementing Disk/User Quotas
Rocks Overview
Managing Rocks
Under the hood with Torque/Maui
IPMI Management
Implementing Disk/User Quotas
Using Rocks 83

84. System Access Access the Rocks Cluster via ssh (port 22)
From Linux systems this is already built in (ssh from the CLI)
For graphical forwarding use ssh with X-forwarding
ssh –X headnode
To copy file use scp
scp file
scp –r directory
From Windows systems use putty
For graphical forwarding in Windows use Xming

85. System Access From Windows systems use putty
For graphical forwarding in Windows use Xming
To copy files use WinSCP

86. System Environment
To setup you environment to compile/run applications, u se the modules environment
~]$ which pgcc
/usr/bin/which: no pgcc in $PATH
~]$ pgcc -V
-bash: pgcc: command not found
~]$ module available
/usr/share/Modules/modulefiles
pgi/32/ pgi/64/ rocks-mpich2/gnu/1.1.1-p1
pgi/32/10.9-mpich pgi/64/10.9-mpich
~]$ module load pgi/64/10.9
/share/apps/pgi/10.9/linux86-64/10.9/bin/pgcc
pgcc bit target on x86-64 Linux -tp istanbul-64

87. Job Control Check Jobs running on the queue [viglen@ulgqcd ~]$ showq
ACTIVE JOBS
JOBNAME USERNAME STATE PROC REMAINING STARTTIME
0 Active Jobs of 496 Processors Active (0.00%)
0 of 31 Nodes Active (0.00%)
IDLE JOBS
JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME
0 Idle Jobs
BLOCKED JOBS

88. Sample Submission Script
#!/bin/bash #PBS -l nodes=4 #PBS –l walltime=2:00 #PBS -o mytest.out #PBS -e mytest.err cd $PBS_O_WORKDIR NPROCS=`wc -l $PBS_NODEFILE | awk '{ print $1 }'` /opt/mpich/bin/mpirun -np $NPROCS -machinefile $PBS_NODEFILE hello

89. Job Control Submit a job to the queue
qsub]$ qsub q_sleep.sh
110.ulgqcd.ph.liv.ac.uk
qsub]$ showq
ACTIVE JOBS
JOBNAME USERNAME STATE PROC REMAINING STARTTIME
viglen Running :00:00 Wed Nov 3 23:20:45
1 Active Job of 496 Processors Active (0.20%)
1 of 31 Nodes Active (3.23%)

90. Job Control Delete a job from the queue
qsub]$ qsub q_sleep.sh
110.ulgqcd.ph.liv.ac.uk
qsub]$ qdel 110

91. Job Control Take a look at the output before jobs finishes
qsub]$ qpeek 190
memtester version (64-bit)
Copyright (C) 2007 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).
pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 90000MB ( bytes)
got MB ( bytes), trying mlock ...locked.
Loop 1/1:
Stuck Address : ok
Random Value : ok
Compare XOR : Compare SUB : Compare MUL : Compare DIV : Compare OR : Compare AND : Sequential Increment: ok
Solid Bits : testing 49 Block Sequential : setting 11

92. Torque/PBS Submit Options
#PBS –I # Submit an interactive job #PBS –q queue_name # Submit to a certain queue #PBS –l nodes=compute-0-0 # Submit to a specific host #PBS –l nodes=4:ppn=4 # 4 Host with 4 Processes per node #PBS –j oe # Combine stdout and stderr #PBS –o stdout_file # Specify the Output file #PBS –e stderr_file # Specify the Error file #PBS –V # Import you environment vars

93. Torque/PBS Variables
$PBS_O_HOST # Hostname where job launched $PBS_O_QUEUE # Originating Queue $PBS_O_WORKDIR # Execution Queue $PBS_JOBID # Job Identifier $PBS_JOBNAME # Job name $PBS_NODEFILE # Nodefile (for parallel jobs) $PBS_O_HOME # Current Home Directory $PBS_PATH # PATH

94. Thank You! Web: http://www.viglen.co.uk/hpc