Presentation on theme: "Submitting Jobs to the Sun Grid Engine"— Presentation transcript:
1 Submitting Jobs to the Sun Grid Engine CiCS DeptThe University of SheffieldSee /usr/mpi/pgi h_vmem and mem the same time always wallclockAdvanced reservationJob dependenciesOctober 2011
2 Topics Covered Introducing the grid and batch concepts. Job submission scripts.How to submit Batch Jobs.How to monitor the progress of your jobs.Starting Interactive Jobs.Cancelling submitted jobs.SGE current version is 5.3 .Definitions: GRID: A collection of Computing Resources. The term is borrowed from Power Systems. GRID is there to meet users demands for Computing Power. It is a term used collectively for all the computing resources that are available to the user ( i.e. all the Machines he is allowed to use)Ideally, users should be able to view ‘the grid’ as a single computational resource into which they can tap. Jobs are submitted to the ‘Grid’ to be run without worrying about or even being aware of which specific node/ ‘machine’/’cpu’ gets allocated to running his/her job.
3 Current WR Grid Resources At Sheffield UniversityIceberg31 of Sun X AMD Opteron processors with 32 GBytes shared memory in each unit) (Total Cores: 248 )96 of Sun X2200 boxes ( 4 AMD Opteron processors in the 2 * dual_processor configuration with 16 GBytes shared memory in each box ) ( Total Cores: 384 )Total of =632 Opteron processors for general use.At Leeds UniversityEverestTotal of 444 AMD Opteron processors catering for jobs using up to 32 GBytes of memory per job.At York UniversityNevada24 machines ( 4 AMD Opteron processors in 2*dual_processor configuration) ( Total cores=96 ) processors, 2 GBytes of memory per processorFimbrata3 machines with 32 GBtes of shared memory ( 8 AMD Opteron Processors each) (Total cores = 24 )
4 iceberg Resources in 2010Two head nodes ( for logging on, system management and light user tasks etc.)96 of AMD Opteron based Sun X2200 Servers:Each server has 4 cores (=in 2*dual processor configuration) and 16 GB of RAM (i.e 4 GB per core ), Clock rate of each core is 2.6GHzTherefore total cores = 96*4 = 38420 of AMD Opteron based Sun V40 servers (parallel nodes):Each server has 8 cores ( in 2*quadcore configuration) and 32 GB of RAM ( i.e. 4GB per core ). Clock rate of each core is 1.9GHz .These parallel nodes are connected to each other with Infiniband based fast interconnects for use with MPI programming, each connection providing 16 Gbits of connection speedTherefore total cores = 20*8=16010 more Sun V40 servers for dedicated research projects.Total number of cores generally available = =544 ( not including the 10 dedicated research project servers )45 Terabytes of usable file store.
5 Iceberg resources after the 2011 upgrade 76 extra nodes proving (12*76=912) extra cores.Each node contains two Intel X5650 CPU’s with hyper-threading disabled but turbo-boost enabled.Each CPU has 6 cores.Therefore, each node has 12 cores and they share RAM memory of 24 GBytes ( i.e. 2 GBytes per core )A few of the nodes has 4 GBytes per core RAM making a total of 48 GBytes per node. These nodes will be dedicated to running large memory ( i.e upto 48 GByte ) jobs.8 NVideo Tesla M2070 GPU’s (i.e. 8*448=3584 GPU cores)An extra brand-new parallel file system ( lustre ) providing 80 Terabytes of extra filespace ( /fastdata ) as well as the currently available 45 Terabytes (in /home & /data )
6 Iceberg Cluster There are two head-nodes for the cluster login login qsh,qsub,qrshqsh,qsub,qrshWorker nodeWorker nodeWorker nodeWorker nodeWorker nodeWorker nodeThere are (new) worker machines in the clusterAll workers share the same filestoreWorker nodeWorker nodeWorker nodeWorker nodeWorker nodeWorker node
7 Using the iceberg cluster Iceberg headnodes are the gateways to the cluster of worker nodes and the only ones where direct logging in is allowed.File system looks identical from all worker nodes and the headnode ( exception being the HPC software directories that are only visible from the worker nodes)All cpu intensive computations must be performed on the worker nodes. This is achieved by the qsh or qrsh command for the interactive jobs and qsub command for the batch jobs.Once you log into iceberg, taking advantage of the power of a worker node for interactive work is done simply by typing qsh and working in the new shell window that is opened. This what appears to be a trivial task would in fact have queried all the worker nodes for you and started a session on the least loaded worker in the cluster.
8 Objectives of a Resource Management Scheme -Fair Sharing of resources amongst the usersManaged by means of SGE Resource Management and Policy Administration Components.-Optimal Use of ResourcesManaged by careful definition of job queues and in real time by SGE queue, job and share management components.- Utilisation Policy can be –functional –share_based or -deadline_based with manual override if needed. We use share_based policy at WRG whereby past usage is taken into account.
9 How the SGE System Operates Users submit an interactive (qsh, qrsh) or a batch job (qsub) to the Sun Grid Engine.For an interactive job: (qsh, qrsh, qlogin)If there are resources immediately available, job gets startedOtherwise the user is informed about the lack of resources and job gets abandoned.For a batch job: ( qsub)If there are resources immediately available the job gets startedOtherwise the job is kept in a queue until resources to execute it becomes available.Jobs are always passed onto the available executing hostsRecords of each jobs progress through the system are kept and reported when requested.
10 Managing Jobs 1:Sun Grid Engine Overview Resource management system, job scheduler, batch system…Can schedule Serial and Parallel jobsSerial jobs run in individual host queuesParallel jobs must include a parallel environment request (-pe <pe_name> N)
12 Running interactive jobs on the cluster User asks to run an interactive job (qsh, qrsh )SGE checks to see if there are resources available to start the job immediately (i.e a free worker )If so, the interactive session is started under the control/monitoring of SGE on that worker.If resources are not available the request is simply rejected and the user notified. This is because by its very nature users can not wait for an interactive session to start.3. User terminates the job by typing exit or logout or the job is terminated when the queue limits are reached (i.e. currently after 8 hours of wall-clock time usage).titan00 is the master host for Sun Grid Engine. This is the machine that holds the waiting job queues, runs the master_daemon.Also must define during installation: -Administrative Host, -Submit Host,-Execution Host, -Quese
13 Summary table of useful SGE commands DescriptionUser/Systemqsub, qresub,qmonSubmit batch jobsUSERqsh,qrshSubmit Interactive Jobsqstat , qhost, qdel, qmonStatus of queues and jobs in queues , list of execute nodes, remove jobs from queuesqacct, qmon, qalter, qdel, qmodMonitor/manage accounts, queues, jobs etcSYSTEM ADMIN
14 Working with SGE as a user qsub ( submit a batch job )qsh or qrsh ( start an interactive job )qstat ( query jobs’ status )qdel ( delete a job )
15 Exercise 1: Submit a job via qsub Create a script file (named example.sh) by using a text editor such as gedit ,vi or emacs and inputing the following lines:#!/bin/sh#echo “This code is running on” /bin/hostname/bin/dateNow Submit this script to SGE using the qsub command:qsub example.sh
16 More on Submitting Batch Jobs qsub command details In its simplest form any script file can be submitted to the SGE by simply typing qsub scriptfile . In this way the scriptfile is queued to be executed by the SGE under default conditions and using default amount of resources.Such use is not always desirable as the default conditions provided may not be appropriate for that job . Also, providing a good estimate of the amount of resources needed helps SGE to schedule the tasks more efficiently.There are two distinct mechanisms for specifying the environment & resources;1) Via parameters to the qsub command2) Via special SGE comments (#$ ) in the script file that is submitted.The meaning of the parameters are the same for both methods and they control such things as;- cpu time required- number of processors needed ( for multi-processor jobs),- output file names,- notification of job activity.
17 Method 1 Using qsub command-line parameters Format:qsub [qsub_params] script_file [-- script_arguments]Examples:qsub myjobqsub –cwd $HOME/myfolder1qsub –l h_rt=00:05:00 myjob -- test1 -largeNote that the last example passes parameters to the script file following the -- token.
18 Method 2 special comments in script files A script file is a file containing a set of Unix commands written in a scripting language ‘usually Bourne/Bash or C-Shell’. When the job runs these script files are executed as if their contents were typed at the keyboard.In a script file any line beginning with # will normally be treated as a comment line and ignored.However the SGE treats the comment lines in the submitted script, which start with the special sequence #$ ,in a special way. SGE expects to find declarations of the qsub options in these comment lines. At the time of job submission SGE determines the job resources from these comment lines.If there are any conflicts between the actual qsub command-line parameters and the special comment (#$) sge options the command line parameters always override the #$ sge options specified in the script.
19 An example script containing SGE options #!/bin/sh#A simple job script for sun grid engine.##$ -l h_rt=01:00:00#$ -m be#$ -Mbenchtest < inputfile > myresults
20 More examples of #$ options in a scriptfile #!/bin/csh# Force the shell to be the C-shell# On iceberg the default shell is the bash-shell#$ -S /bin/csh# Request 2 GBytes of virtual memory#$ -l h_vmem=2G# Specify myresults as the output file#$ -o myresults# Compile the programpgf90 test.for –o mytestprog# Run the program and read the data that program# would have read from the keyboard from file mydatamytestprog < mydata
21 Qsub options on iceberg (resources related) -l h_rt=hh:mm:ssThe wall clock time.Example: -l h_rt=01:00:00 ( for one hour job )-l h_vmem=memoryor-l mem=memorySets the limit of virtual memory required (for parallel jobs per processor). Memory can be nnK, nnM or nnGExample: -l mem=3G ( 3 Gigabytes )-l mem= 512M ( 512 Mbytes )-pe openmp n-pe mvapich2-ib n-pe openmpi-ib n-pe ompigige nSpecifies the parallel environment to be used for parallel jobs. n is the number of cpu’s needed to run the parallel job. In general, to run an n-way parallel job you must specify n number of processors via this option.-pe [env] n-mRather than a fixed number of processors for any of the above parallel environments, a range of processors can also be specified to request minimum of (n) and maximum of (m) processors. Example: –pe openmp 4-8-helpPrints a list of options
22 Qsub options ( environment related … ) -cwd directoryExecute the job from the specified directory. By default the job runs on the same directory it was submitted from. However, using this option you can specify a different directory.-P projectnameRun the job under a special project account-S shellUse the specified shell to interpret the script rather than the default Bash shell. Example: -S /bin/csh-v variable[=value]Passes an environment variable to the executing jobs shell. Example –v OMP_NUM_THREADS=4-VExport all environment variables to the job.
23 Qsub options ( notifications and testing related ) -M _addressaddress for job notifications.Example:-M-m b e a sSend (s) when the job begins , ends, aborted or suspendedExample: –m be-nowStart running the job now or if can’t be run exit with an error code.-verifydo not submit the job but check and report on submission.
24 Qsub options (output files and job names related ) When a job is submitted it is given a unique job_number . Also by default the name of the script filebecomes the jobname . When the job starts running the standard output and error output is sent tofiles that are named as jobname.ojob_number and jobname.ejob_number respectively.For example: myscript.o myscript.e45612The following parameters modify this behaviour.-o pathnameName of the file that will contain the main output-e pathnameName of the file that will contain error output messages-j y ( highly recommended )(Join) Send both main ouput and error output messages into the same file.-N jobnameRather than the scriptfile use the supplied name as the jobname
25 Example: relating to job output files Passed to qsub as arguments during submission:qsub –N walldesign –j y walljobOR insert in the submit script file walljob:#$!/bin/bash#$ -N walldesign#$ -j y/home/data/my_appand submit the job qsub walljobUsing either of these methods, when the job runs the both normal and error output will be contained in a file named walldesign.onnnnnwhere nnnnn is the unique job number SGE designated to your job at the time of submission.
26 More on starting interactive iobs qsh and qrsh commands qsh : starts an Xterm session on a worker node. Use this command if you have XWindows capabilities.qrsh : starts a remote command shell and optionally executes a shell-scripts on a worker node. If you do not have Xwindows capability, i.e. you are not using “Exceed, Xming Cygwin ” so on, this is the way for starting interactive jobs on iceberg. It is suitable when you log in using putty or ssh in line-mode.In Sheffield all interactive jobs are put into the short queues that limit the clock time to 8 hours of wall clock time.BEWARE: As soon as the time limit is exceeded the job will terminate without any warning.
27 qrsh command qrsh [parameters] If no parameters are given it behaves exactly like qsh. This is the normal method of using qrsh.If there are parameters a remote shell is started up on one of the workers and the parameters are passed to shell for execution. For example, if a script file name is presented as a parameter, commands in the script file are executed and the job terminates when the end of the script file is reached.Example : qrsh myscript
28 qsh and Qsh qsh –display display_specifier qsh starts up an X-terminal within which the interactive job is started. It is possible to pass any Xterm parameters via the -- construct. Example : qsh -- –title myjob1Type man xterm for a list of parameters.Qsh : this is a Sheffield only qsh with nicer X-term parameters if you are coming from a Windows machine.
29 Monitoring your jobs A submitted job will either be; still waiting in the queue,be executing,finished execution and left the SGE scheduling system.In order to monitor the progress of your job while in states (1) and (2) use the qstat or Qstat commands that will inform you if the job is still waiting or started executing. The command qstat gives info about all the jobs but Qstat gives info about your jobs alone.While executing (state 2) ;use qstat –j job_number to monitor the jobs status including time and memory consumptions. Contains too much information !Better still use qstat –j job_number | grep mem that will give time and memory consumed information.Also use tail –f job_output_filename to see the latest output from the jobFinished executing ( state 3) ;qacct is the only command that may be able to tell you about the past jobs by referring to a data-base of past usage. Output file names will contain the job number so;qacct -j job_numbershould give some information.
30 If you are interested in only your job use Qstat Monitoring your jobIf you are interested in only your job use QstatIf you want to see all the jobs use qstatjob-ID prior name user state submit/start at queue slots ja-task-IDINTERACTIV cs1ds r /21/ :03:INTERACTIV cs1ds r /21/ :37:job-ID prior name user state submit/start at queue slots ja-task-IDgrd80x120x cpp06kw r /15/ :35:M11_NiCl.3 chp09bwh r /15/ :44:pythonjob co1afh r /16/ :34:parallelNe elp05ws r /16/ :34:parallelNe elp05ws r /16/ :31:coaxialjet zzp09hw r /17/ :03:periodichi mep09ww r /18/ :39:gav me1ccl r /18/ :00:ga me1ccl r /18/ :00:DDNaca0 mep09ww r /18/ :41:ga me1ccl r /18/ :00:ga me1ccl r /18/ :00:job elp07dc r /19/ :25:fluent12jo fc1jz qw /19/ :32:01Oct_ATLAS php09ajf qw 10/19/ :41:01
31 qstat commandqstat command will list all the jobs in the system that are either waiting to be run or running. This can be a very long list !qstat –f full listing ( even longer)qstat –u username or Qstat ( recommend this ! )qstat –f –u username ( detailed information )Status of the job is indicated by letters in qstat listings as:qw waiting t transferingr running s,S suspendedR restarted T treshold
32 Deleting Jobs with the qdel command qdel command will remove from the queue the specified jobs that are waiting to be run or abort jobs that are already running.Individual Jobqdel 15112List of JobsqdelAll Jobs running or queueing under a given usernameqdel –u <username>
33 Reasons for Job Failures SGE cannot find the binary file specified in the job scriptOne of the Linux environment resource limits is exceeded (see command ulimit –a )Required input files are missing from the startup directoryYou have exceeded your quota and job fails when trying to write to a file ( use quota command to check usage )Environment variable is not set (LM_LICENSE_FILE etc)Hardware failure
34 Job ArraysBy using a single qsub command, it is possible to submit a series of jobs that use the same job template. These jobs are described as array jobs.For example: qsub myanalysis –t 1-10 will submit the script named myanalysis as 10 separate jobs. Of course it is pointless to run the same job 10 times. The only justification for doing so is that all these jobs are doing different tasks. This is where a special environment variable named SGE_TASK_ID becomes essential.In the above example in each job the variable SGE_TASK_ID will contain a unique number between 1 and 10 to differentiate these jobs from each other.Thus we can use this variable’s value to control each job in a different manner.
35 Example Array jobs and the $SGE_TASK_ID variable #$ -S /bin/tcsh#$ -l h_cpu=01:00:00#$ -t 2-16:2#$ -cwdmyprog > results.$SGE_TASK_ID < mydata.$SGE_TASK_IDThis will run 8 jobs. The jobs are considered to be independent of each other and hence may run in parallel depending on the availability of resources. Note that tasks will be numbered 2, 4 ,6 ,8 … (steps of 2)For example job 8 will read its data from file mydata.8 and write its output to file results.8It is possible to make these jobs dependent on each other so as to impose an order of execution by means of the –hold_jid parameter.
36 An example OpenMP job script OpenMP programming takes advantage of the multiple CPU’s that reside in a single computer to distribute work amongst CPUs that share the same memory.Currently we have maximum of 8 CPU’s per computer and therefore only upto 8 processors can be requested for an iceberg openmp job.After the next upgrade this figure will increase to minimum 24.#$ -pe openmp 4#$ -l h_rt=01:30:00OMP_NUM_THREADS=4./myprog
37 An example MPI job script MPI programs are harder to code but can take advantage of interconnected multiple computers by passing messages between them ( MPI= Message Passing Interface )23 workers on the iceberg pool are connected together with fast Infiniband communications cabling to provide upto 10 Gbits/sec data transfer rate between them.The rest of the workers can communicate with each other via the normal 1 Gbits/sec ethernet cables.#$ -pe mvapich2-ib 4# limit run to 1 hours actual clock time#$ -l h_rt=1:00:00mpirun_rsh -rsh -np $NSLOTS -hostfile $TMPDIR/machines ./executable
38 The progress of your batch job The user submits a batch job as described aboveeg. qsub myscript_fileThe job is placed in the queue and given a unique job number <nnnn>The user is informed immediately of the job number <nnnn>The user can check the progress of the job by using the qstat command. Status of the job is shown as qw (waiting), t (transfering) or r (running)User can abort a job by using the qdel command at this stage.When the job runs the standard output and error messages are placed in files named <my_scriptfile>.o<nnnn> and <my_scriptfile>.e<nnnn> respectively
39 HintsOnce you prepared your job script you can test it by simply running it, if possible for a very small problem. Note also that the qsub parameters which are defined using the #$ sequence will be treated as comments during this run.Q: Should I define the qsub parameters in the script file or as parameters at the time of issuing qsub ?A: The choice is yours, I prefer to define any parameter, which is not likely to alter between runs, within the script file to save myself having to remember it at each submission.
40 SGE related Environment Variables Apart from the specific environment variables passed via the –v or –V options, during the execution of a batch job the following environment variables are also available to help build unique or customized filenames messages etc.$HOME : Your own login directory$USER : your iceberg username$JOB_NAME : Name of the job$HOSTNAME : Name of the cluster node that is being used$SGE_TASK_ID : Task number (important for task arrays)$NSLOTS : Number of processors used ( important for parallel ‘openmp or mpi’ jobs )
41 Submitting Batch Jobs via the qmon command If you are using an X terminal ( such as provided by Exceed ) then a GUI interface named qmon can also be used to make job submission easier.This command also allows an easier way of setting the job parameters.
42 Job submission panel of QMON Click on Job Submission IconClick to browse for the job scripttest2
43 Job queuesUnlike the traditional batch queue systems, users do not need to select the queue they are submitting to. Instead SGE uses the resource needs as specified by the user to determine the best queue for the job.In Sheffield and Leeds the underlying queues are setup according to memory size and cpu time requirements and also numbers of multiple cpu’s needed (for mpi & openmp jobs )qstat –F displays full queue information, Also qmon (Task-Queue_Control) will allow information to be distilled about the queue limits.
44 Job queue configuration Normally you will not need to know the details of each queue, as the Grid Engine will make the decisions for you in selecting a suitable queue for your job. If you feel the need to find out how the job queues are configured, perhaps to aid you in specifying the appropriate resources, you may do so by using the qconf system administrator command.qconf –sql will give a list of all the queuesqconf –sq queue_name will list details of a specific queue’s configuration
45 Monitoring the progress of your jobs The commands qstat and the XWindows based qmon can be used to check on the progress of your jobs through the system.We recommend that you use the qmon command if your terminal has X capability as this makes it easier to view your jobs progress and also cancel or abort it, if it becomes necessary to do so.
46 Checking the progress of jobs with QMON Click on Job Control IconClick on Running Jobs tab
47 TutorialsOn iceberg copy the contents of the tutorial directory to your user area into a directory named sge:cp –r /usr/local/courses/sge sgecd sgeIn this directory the file readme.txt contains all the instructions necessary to perform the exercises.
48 Further documentation and help man ( manual pages) : man sge , man qsh man qsub so on..HTML pagesThe End