1. Introduction to Job Description Language (JDL) Alessandro Costa INAF Catania Corso di Calcolo Parallelo Grid Computing Catania - ITALY 25-29 September 2006

2. 2Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Introduction The Workload Management System (WMS) is the gLite 3.0 component that allows users to submit jobs, and performs all tasks required to execute them, without exposing the user to the complexity of the Grid. The JDL attributes described in this document are the ones supported when the submission to the WMS is performed through the legacy Network Server interface, i.e. using the python command line interface or the C++/Java API of the gLite WMS-UI subsystem. It basically represents a subset of the whole set of attributes supported by the WMS when accessed via the new web services based interface (WMProxy).

3. 3Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania JDL Overview The Job Description Language (JDL) The Job Description Language (JDL) is a high-level, user-oriented language based on Condor classified advertisements (classads) for describing jobs and aggregates of jobs such as Direct Acyclic Graphs (DAG). Being the JDL an extensible language the user is allowed to use whatever attribute for the description of a request without incurring in errors from the JDL parser. However, only a certain set of attributes, that we will refer as “supported attributes” from now on, is taken into account by the Workload Management System (WMS) components in order to schedule and submit a job or the jobs of a complex request. The JDL is used to specify the desired job characteristics and constraints, which are used by the match-making process to select the best resource to execute the job.

4. 4Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania JDL format A job description is a file (called JDL file) consisting of lines having the format: Attribute = expression; and terminated by a semicolon. Expressions can span several lines, but only the last one must be terminated by a semicolon. Comments must have a sharp character (#) or a double slash (//) at the beginning of each line. Comments spanning multiple lines can be specified enclosing the text between “/*” and “*/”.

5. 5Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania JDL example Type="Job"; JobType="Normal"; Executable = “ls"; StdError = "stderr.log"; StdOutput = "stdout.log"; Arguments = “-lrt"; InputSandbox = "start_hostname.sh"; OutputSandbox = {"stderr.log", "stdout.log"}; RetryCount = 7; VirtualOrganisation="gilda";

6. 6Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Type and JobsType The Type attribute is a string representing the type of the request described by the JDL: Eg:Type = “Job”; or Type = “Dag”; The JobType attribute is a string representing the type of the job described by the JDL: Eg:JobType = “Normal”; If “JobType” is not specified in the JDL the WMS will set it to “Normal”

7. 7Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Type The WMS currently supports the following types for Requests : Jobsimple job DAGa Direct Acyclic Graph of dependent jobs Collection a set of independent jobs Through WMPROXY

8. 8Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania JobsType Normala simple batch job Interactivea job whose standard streams are forwarded to the submitting client MPICHa parallel application using MPICH-P4 implementation of MPI Partitionableset of independent sub-jobs, each one taking care of a step or of a sub-set of steps, and which can be executed in parallel Checkpointablea job able to save its state, so that the job execution can be suspended and resumed later, starting from the same point where it was first stopped Through WMPROXY Parametrica job whose JDL contains parametric attributes (e.g. Arguments, StdInput etc.) whose values can be made vary in order to obtain submission of several instances of similar jobs only differing for the value of the parameterized attributes

9. 9Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Types of requests The JDL allows description of the following types of requests (ns submission): JOB a simple job (default) DAG a Direct Acyclic Graph of dependent jobs

10. 10Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Executable The Executable attribute is a string representing the executable/command name. The user can specify an executable that lies already on the remote CE-WN and in this case the absolute path, possibly including environment variables referring to this file should be specified Eg: Executable = “/usr/local/java/j2sdk1.4.0_01/bin/java”; The other possibility is to provide a local executable name, which will be copied from the UI node to the WMS node and then staged to the Computing Element WN where the job is run. In this case only the file name has to be specified as executable. Eg: Executable = “cms_sim.exe”;

11. 11Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Arguments The Arguments attribute is a string containing all the job command line arguments. Eg: $ ls –lrtCommand: Executable = “/bin/ls”; Arguments =“-lrt”; If you want to specify a quoted string inside the Arguments then you have to escape quotes with the \ character. Eg: $ grep –i “my name” *.txt Command: Executable = “/bin/grep”; Arguments = “-i \”my name\” *.txt”;

12. 12Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania StdInput – StdOutput - StdError StdInput, StdOutput, StdError = The StdInput attribute is a string representing the standard input of the job. $ job < “standard input file” The StdOutput attribute is a string representing the file name where the standard output of the job is saved on the WN. $ job > “standard output file” The StdError attribute is a string representing the file name where the standard error of the job is saved on the WN. $ job 2> “standard error file”

13. 13Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania StdInput – StdOutput - StdError These attributes cannot are included in the JDL for Interactive jobs. Eg: StdInput = “/var/tpm/jobInput”; StdInput = “myjobInput”; Wild cards are not allowed. The value specified for StdError can be the same as the one for StdOutput: this means that the two standard streams of the job are saved in the same file. To have this file staged back on the submitting machine the user has to list the file name also in the OutputSandbox attribute expression and use e.g the glite-job-output command

14. 14Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania InputSandbox If some files have to be copied from the UI to the execution node, they can be listed in the InputSandbox attribute. The InputSandbox attribute is a string or a list of strings identifying the list of files on the UI local disk needed by the job for running. These files are transferred from the UI to the WMS node and then downloaded on the CE WN where the job is run by the WMS JobWrapper script. Wildcards and environment variables are admitted in the specification of this attribute (both are resolved on the UI node by the WMS client software). Eg:Executable = “cms_sim.exe”; InputSandbox = {“cms_sim.exe", "/home/edguser/HandsOn/mytest.exe", "myscript.sh", "/home/edguser/HandsOn/file*", "/home/edguser/data/*", "$GLITE_TMP/mytest.conf" };

15. 15Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania OutputSandbox The OutputSandbox attribute is a string or a list of strings identifying the list of files generated by the job on the WN, which the user wants to retrieve on the UI. Wildcards are not admitted in the specification of this attribute. File names can be provided as simple file names, or relative paths with respect to the current working directory on the execution WN. Eg: OutputSandbox = { "myjobOutput", "myjobError", "run1/event1", "run1/event2", "/tmp/myjob.log" };

16. 16Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania ExpiryTime A job for which no compatible CEs have been found during the matchmaking phase is hold in the WMS Task Queue for a certain time (currently it is 1 day from job submission) so that it can be subjected again to matchmaking from time to time until a compatible CE is found. If no match is found after 1 day the job is aborted. The ExpiryTime attribute is an integer representing the date and time (in seconds since epoch) until the job has to be considered valid by the WMS. Eg:ExpiryTime = $ date –date ‘1970-01-01 UTC 1112339655 seconds’ Transforms seconds since epoch in date $ date +%s Returns date in seconds since epoch

17. 17Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Environment The Environment attribute is a list of string representing environment settings that have to be performed on the execution machine and are needed by the job to run properly. The JobWrapper on the Worker Node performs these settings just before the job is started. Each item of the list is an equality “VAR_NAME=VAR_VALUE”. Eg Environment = {“JOB_LOG_FILE=/tmp/myjob.log”, “ORACLE_SID=edg_rdbms_1”, “JAVABIN=/usr/local/java”};

18. 18Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania VirtualOrganisation The VirtualOrganisation attribute is a string representing the name of the VO the submitting user is currently working for. The value for this attribute has to match with the VO specified within the credentials issued by VOMS or with the possible options provided by the WMProxy client commands. Eg: VirtualOrganisation = “gilda”;

19. 19Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania MyProxyServer The MyProxyServer attribute specifies the hostname of a MyProxy server where the user has registered her/his long- term proxy certificate. A MyProxy server can be used to store a long-lived user certificate which can be used by the WMS to renew the lifetime of a standard user certificate proxy (usually valid only for 12 hours). Eg:MyProxyServer = “grid001.ct.infn.it”; Catania, Tutorial TriGrid VL, 27-28.07.2006

20. 20Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania RetryCount, ShallowRetryCount The RetryCount attribute is an integer representing the maximum number of deep job re-submissions to be done in case of failure due to some grid component The ShallowRetryCount attribute is an integer representing the maximum number of shallow job re-submissions to be done in case of failure due to some grid component Job resubmission is defined “deep” when the user’s job has started running on the WN and then the job itself or the WMS JobWrapper has failed. Job resubmission is instead defined “shallow” when the WMS JobWrapper has failed before starting the actual user’s job. RetryCount / ShallowRetryCount has to be a number equal or greater than 0 and the actual number of submission retries for a job is represented by the minimum value between RetryCount / ShallowRetryCount itself and the value of the MaxRetryCount / MaxShallowRetryCount parameter in the WM configuration file (default for MaxRetryCount / MaxShallowRetryCount is 10). Eg: RetryCount = 2; ShallowRetryCount=3;

21. 21Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania NodeNumber MPICH Jobs The NodeNumber attribute is an integer greater than 1 specifying the number of nodes needed for a MPI job. This attribute is only allowed if the job type is MPICH The RB uses this attribute during the matchmaking for selecting those CE having a number of CPUs equal or greater than the one specified in NodeNumber. Eg: NodeNumber = 5;

22. 22Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Requirements Requirements = –Expression that uses C-like operators. –It represents job requirements on resources. –The Requirements expression can contain attributes that describe the CE in the IS To have a job scheduled to run on a given CE, this Requirements expression must evaluate to true on the given CE. The evaluation of this expression is performed by the RB during the match making phase. All these attributes are reported in the Glue Schema for the CE.

23. 23Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Requirements Requirements = Eg: Requirements = other.GlueCEInfoLRMSType == "PBS" && other.GlueCEInfoTotalCPUs > 2 && Member("IDL1.7",other.GlueHostApplicationSoftwareRunTimeEnviro nment); The above expression requires a CE whose local resource manager is PBS, having at least 2 CPUs and the IDL software version 1.7 already installed. The classAd Member function returns true if the tag "IDL1.7" is a member of the other.GlueHostApplicationSoftwareRunTimeEnvironme nt list.

24. 24Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Rank The Rank attribute is a ClassAd Floating-Point expression that states how to rank CEs that have already met the Requirements expression. Rank expresses a preference. A higher numeric value equals a better rank. The WMS will submit the job to the CE with the highest rank. The Rank expression can contain attributes that describe the CE in the IS which are prefixed with “other.”. All these attributes are reported in the Glue Schema for the CE. The evaluation of the rank expression is performed by the RB during the match making phase. Eg: Rank = other.GlueCEPolicyMaxRunningJobs – other.GlueCEStateRunningJobs; With the above Rank, the preferred CEs are the ones having the greatest number of free slots available for running jobs.

25. 25Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Rank The Rank attribute is mandatory in the JDL as it is a mandatory parameter for the matchmaking library. If this attribute is not included in the JDL the glite-job- submit command sets it to: Rank = -other.GlueCEStateEstimatedResponseTime; that expresses a preference for those CEs having a shorter estimated time for traversing the local batch system queue. This automatic setting is however not insured if submission is performed using client tools other than glite-job-submit.

26. 26Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (1/9) Submitted job has been submitted by the user but not yet processed by the Network Server

27. 27Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (2/9) Waiting The job has been accepted by the Network Server but not yet processed by the Workload Manager

28. 28Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (3/9) Ready The job has been assigned to a Computing Element but not yet transferred to it

29. 29Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (4/9) Scheduled The job is waiting in the Computing Element’s queue

30. 30Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (5/9) Running The job is running

31. 31Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (6/9) Done The job has finished

32. 32Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (7/9) Aborted The job has been aborted by the WMS (e.g. because it was too long, or the proxy certificated expired, etc.)

33. 33Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (8/9) Cancelled The job has been canceled by the user

34. 34Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania Jobs State Machine (9/9) Cleared The Output Sandbox has been transferred to the User Interface

35. 35Corso Avanzato di Calcolo Parallelo e Grid Computing – 28-29 Sep - Catania References GLITE 3.0 user guide manuals series https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf Job Description Language (JDL ) di Sergio Lo Buono http://agenda.trigrid.it//askArchive.php?base=agenda&categ=a069 &id=a069s0t18/transparenciesStep-by-step JDL JDL Attributes Specification (submission via WMS Network Server) https://edms.cern.ch/file/555796/1/EGEE-JRA1-TEC-555796-JDL- Attributes-v0-8.pdf