1. Condor Tutorial Prabhaker Mateti Wright State University

2. Mateti, Condor 2 Acknowledgements Many of these slides are adapted from tutorials by Miron Livny, and his associates University of Wisconsin-Madison http://www.cs.wisc.edu/condor

3. Mateti, Condor 3 Clusters with Part Time Nodes Cycle Stealing: Running of jobs on a workstations that don't belong to the owner. Definition of Idleness: E.g., No keyboard and no mouse activity Tools/Libraries –Condor –PVM –MPI

4. Mateti, Condor 4 Performance v. Throughput High Performance - Very large amounts of processing capacity over short time periods –FLOPS - Floating Point Operations Per Second High Throughput - Large amounts of processing capacity sustained over very long time periods –FLOPY - Floating Point Operations Per Year FLOPY = 365x24x60x60*FLOPS?

5. Mateti, Condor 5 Cooperation Workstations are “personal” Others use slows you down –Immediate-Eviction –Pause-and-Migrate Willing to share –Letting you cycle-steal Willing to trust

6. Mateti, Condor 6 Granularity of Migration Process migration –Process = Collection of objects –at least one active object Object migration –Passive objects –Active objects

7. Mateti, Condor 7 Migration of Jobs: Technical Issues Checkpointing: Preserving the state of the process so it can be resumed. One architecture to another Your “environment” – keyboard, mouse, display, files, …

8. Mateti, Condor 8 Condor A system for high throughput computing by making use of idle computing resources Lots of jobs over a long period of time, not a short burst of high performance Manages both machines and jobs Has been stable, and delivered thousands of CPU hours

9. Mateti, Condor 9 Condor Techniques Migratory programs –Checkpointing –Remote IO Resource matching

10. Mateti, Condor 10 Condor: Assumptions Large numbers of workstations are idle most of the time Owners of such machines would not mind their use by others while idle Owners want their work to be given high priority

11. Mateti, Condor 11 Roles Owner offers his machine for use by others User requests to run his jobs Administrator manages the pool of available machines Multiple roles possible

12. Mateti, Condor 12 Classified Advertisements: Example MyType = "Machine" TargetType = "Job" Name = "froth.cs.wisc.e du" StartdIpAddr= "<128.105.73.44 :33846>" Arch = "INTEL" OpSys = "SOLARIS26" VirtualMemory = 225312 Disk = 35957 KFlops = 21058 Mips = 103 LoadAvg = 0.011719 KeyboardIdle = 12 Cpus = 1 Memory = 128 Requirements = LoadAvg 15 * 60 Rank = 0

13. Mateti, Condor 13 Condor User Requests Describes the program, and its needs Example condor_submit File Universe = standard Executable = /home/wsu03/condor/my_job.condor Input = my_job.stdin Output = my_job.stdout Error = my_job.stderr Log = my_job.log Arguments = -arg1 -arg2 InitialDir = /home/wsu03/condor/run_1 Queue

14. Mateti, Condor 14 ClassAds: Example for Jobs Requirements = Arch == “INTEL” && OpSys == “LINUX” && Memory > 20 Rank = (Memory > 32) * ( (Memory * 100) + (IsDedicated * 10000) + Mips )

15. Mateti, Condor 15 Condor Pool of Machines “Pool” can be a single machine, or a group of machines volunteered by their owners Determined by a “central manager” - the matchmaker and centralized information repository Each machine runs various daemons to provide different services, either to the users who submit jobs, the machine owners, or the pool itself

16. Mateti, Condor 16 Condor System Structure

17. Mateti, Condor 17 Condor Agents Condor Resource Agent –condor_startd daemon –allows a machine to execute Condor jobs –enforces owner policy Condor User Agent –condor_schedd daemon –allows a machine to submit jobs to a pool

18. Mateti, Condor 18 Condor: Robustness Checkpointing allows guaranteed forward progress of your jobs, even jobs that run for weeks before completion If an execute machine crashes, you only loose work done since the last checkpoint Condor maintains a persistent job queue - if the submit machine crashes, Condor will recover

19. Mateti, Condor 19 What’s Condor Good For? Managing a large number of jobs –You specify the jobs in a file and submit them to Condor, which runs them all and can send you email when they complete –Mechanisms to help you manage huge numbers of jobs (1000’s), all the data, etc. –Condor can handle inter-job dependencies (DAGMan)

20. Mateti, Condor 20 Throughput Checkpointing allows your job to run on opportunistic resources, not dedicated Checkpointing permits migration - if a machine is no longer available, migrate With remote system calls, you don’t even need an account on a machine where your job executes

21. Mateti, Condor 21 Can your program work with Condor? What kind of I/O does it do? Does it use TCP/IP? (network sockets) Can the job be resumed? Multiple processes? –fork(), pvm_addhost(), etc.

22. Mateti, Condor 22 Typical IO Interactive TTY “Batch” TTY (just reads from STDIN and writes to STDOUT or STDERR, but you can redirect to/from files) X Windows NFS, AFS, or another network file system Local file system TCP/IP

23. Mateti, Condor 23 Condor Universes Different universes support different functionalities Vanilla Standard Scheduler PVM

24. Mateti, Condor 24 Condor Universes: IO support No support for interactive TTY X11NFSLocalFilesTCP Vanillaxx-x Standard-xx- Schedulerxxxx PVMxxxx

25. Mateti, Condor 25 Condor Universes PVM (Parallel Virtual Machine) –Multiple processes in Condor Scheduler –The job is run on the submit machine, not on a remote execute machine –Job is automatically restarted if the condor_schedd is shutdown –Used to schedule jobs

26. Mateti, Condor 26 Submitting Jobs to Condor Choosing a “Universe” for your job Preparing your job –Making it “batch-ready” –Re-linking if checkpointing and remote system calls are desired (condor_compile) Creating a submit description file condor_submit your request to the User Agent (condor_schedd)

27. Mateti, Condor 27 Making your job “batch-ready” Must be able to run in the background: no interactive input, windows, GUI, etc. Can still use STDIN, STDOUT, and STDERR but files are used for these instead of the actual devices If your job expects input from the keyboard, you have to put the input you want into a file

28. Mateti, Condor 28 Preparing Your Job (cont’d) If you are going to use the standard universe with checkpointing and remote system calls, you must re-link your job with Condor’s libraries condor_compile gcc -o myjob myjob.c

29. Mateti, Condor 29 Submit Description File Tells Condor about your job: –Which executable, universe, input, output and error files to use, command-line arguments, environment variables, any special requirements or preferences (more on this later) Can describe many jobs at once (a “cluster”) each with different input, arguments, output, etc.

30. Mateti, Condor 30 Example condor_submit File Universe = standard Executable = /home/wsu03/condor/my_job.condor Input = my_job.stdin Output = my_job.stdout Error = my_job.stderr Log = my_job.log Arguments = -arg1 -arg2 InitialDir = /home/wsu03/condor/run_1 Queue

31. Mateti, Condor 31 Example Submit Description File Submits a single job to the standard universe, specifies files for STDIN, STDOUT and STDERR, creates a UserLog, defines command line arguments, and specifies the directory the job should be run in As if you did % cd /home/wright/condor/run_1 % /home/wsu03/condor/my_job.condor -arg1 -arg2 \ > my_job.stdout 2> my_job.stderr \ < my_job.stdin

32. Mateti, Condor 32 “Clusters” and “Processes” A submit file describes one or more jobs The collection of jobs is called a “cluster” Each job is called a “process” or “proc” A Condor “Job ID” is the cluster number, a period, and the proc number (e.g., 23.5) Proc numbers always start at 0

33. Mateti, Condor 33 A Cluster Submit Description File Universe = standard Executable = /home/wsu03/condor/my_job.condor Input = my_job.stdin Output = my_job.stdout Error = my_job.stderr Log = my_job.log Arguments = -arg1 -arg2 InitialDir = /home/wsu03/condor/run_$(Process) Queue 500

34. Mateti, Condor 34 A Cluster Submit Description File “Queue 500” = submit 500 jobs at once The initial directory for each job is specified with the $(Process) macro $(Process) will be expanded to the process number for each job in the cluster; “run_0”, “run_1”, … “run_499” directories All the input/output files will be in different directories

35. Mateti, Condor 35 condor_submit condor_submit the-submit-file-name condor_submit parses the file and creates a “ClassAd” that describes your job(s) Creates the files you specified for STDOUT and STDERR Sends your job’s ClassAd(s) and executable to the condor_schedd, which stores the job in its queue

36. Mateti, Condor 36 Monitoring Your Jobs Using condor_q Using a “User Log” file Using condor_status Using condor_rm Getting email from Condor Using condor_history after completion

37. Mateti, Condor 37 Using condor_q Displays the status of your jobs, how much compute time it has accumulated, etc. Many different options: –A single job, a single cluster, all jobs that match a certain constraint, or all jobs –Can view remote job queues, either individual queues, or “-global”

38. Mateti, Condor 38 Using a “User Log” file Specify in your submit file: –Log = filename Entries logged for: –When it was submitted –when it started executing –if it is checkpointed or vacated –if there are any problems, etc.

39. Mateti, Condor 39 Using condor_status the “-run” option to see –Machines running jobs –The user who submitted each job –The machine they submitted from Can also view the status of various submitters with “-submitter ”

40. Mateti, Condor 40 Using condor_rm Removes a job from the Condor queue You can only remove jobs that you own Root can condor_rm someone else’s jobs You can give specific job ID’s (cluster or cluster.proc), or you can remove all of your jobs with the “-a” option.

41. Mateti, Condor 41 Getting Email from Condor By default, Condor will send you email when your jobs completes If you don’t want this email, put this in your submit file: notification = never If you want email every time something happens to your job (checkpoint, exit, etc), use this: notification = always

42. Mateti, Condor 42 If you only want email if your job exits with an error, use this: notification = error By default, the email is sent to your account on the host you submitted from. If you want the email to go to a different address, use this: notify_user = email@address.here Getting Email from Condor

43. Mateti, Condor 43 Using condor_history Once your job completes, it will no longer show up in condor_q Now, you must use condor_history to view the job’s ClassAd The status field (“ST”) will have either a “C” for “completed”, or an “X” if the job was removed with condor_rm

44. Mateti, Condor 44 Classified Advertisements A ClassAd is a set of named expressions –Each named expression is an attribute Expressions are similar to those in C … –Constants, attribute references, operators

45. Mateti, Condor 45 Classified Advertisements: Example MyType = "Machine" TargetType = "Job" Name = "froth.cs.wisc.e du" StartdIpAddr= "<128.105.73.44 :33846>" Arch = "INTEL" OpSys = "SOLARIS26" VirtualMemory = 225312 Disk = 35957 KFlops = 21058 Mips = 103 LoadAvg = 0.011719 KeyboardIdle = 12 Cpus = 1 Memory = 128 Requirements = LoadAvg 15 * 60 Rank = 0

46. Mateti, Condor 46 ClassAd Matching ClassAds are always considered in pairs: –Does ClassAd A match ClassAd B (and vice versa)? –This is called “2-way matching” If the same attribute appears in both ClassAds, you can specify which attribute you mean by putting “MY.” or “TARGET.” in front of the attribute name

47. Mateti, Condor 47 ClassAd Matching “Example” ClassAd A MyType = "Apartment“ TargetType = "ApartmentRenter“ SquareArea = 3500 RentOffer = 1000 OnBusLine = True Rank = UnderGrad==False + TARGET.RentOffer Requirements = MY.RentOffer - TARGET.RentOffer < 150 ClassAd B MyType = "ApartmentRenter" TargetType = "Apartment" UnderGrad = False RentOffer = 900 Rank = 1/(TARGET.RentOffer + 100.0) + 50*HeatIncluded Requirements = OnBusLine && SquareArea > 2700

48. Mateti, Condor 48 ClassAds in the Condor System ClassAds allow Condor to be a general system –Constraints and ranks on matches expressed by the entities themselves –Only priority logic integrated into the Match- Maker All principal entities in the Condor system are represented by ClassAds –Machines, Jobs, Submitters

49. Mateti, Condor 49 ClassAds: Example for Machines Friend = Owner == "tannenba“ || Owner == "wright" ResearchGroup = Owner == "jbasney" || Owner == "raman" Trusted = Owner != "rival" && Owner != "riffraff" Requirements = Trusted && ( ResearchGroup || (LoadAvg 15*60) ) Rank = Friend + ResearchGroup*10

50. Mateti, Condor 50 ClassAd Machine Example Machine will never start a job submitted by “rival” or “riffraff” If someone from ResearchGroup (“jbasney” or “raman”) submits a job, it will always run If anyone else submits a job, it will only run here if the keyboard has been idle for more than 15 minutes and the load average is less than 0.3

51. Mateti, Condor 51 Machine Rank Example Described If the machine is running a job submitted by owner “foo”, it will give this a Rank of 0, since foo is neither a friend nor in the same research group If “wright” or “tannenba” submits a job, it will be ranked at 1 (since Friend will evaluate to 1 and ResearchGroup is 0) If “raman” or “jbasney” submit a job, it will have a rank of 10 While a machine is running a job, it will be preempted for a higher ranked job

52. Mateti, Condor 52 ClassAds: Example for Jobs Requirements = Arch == “INTEL” && OpSys == “LINUX” && Memory > 20 Rank = (Memory > 32) * ( (Memory * 100) + (IsDedicated * 10000) + Mips )

53. Mateti, Condor 53 Job Example Described The job must run on an Intel CPU, running Linux, with at least 20 megs of RAM All machines with 32 megs of RAM or less are Ranked at 0 Machines with more than 32 megs of RAM are ranked according to how much RAM they have, if the machine is dedicated (which counts a lot to this job!), and how fast the machine is, as measured in MIPS

54. Mateti, Condor 54 ClassAd Attributes in your Pool Condor defines a number of attributes by default, which are listed in the User Manual (“About Requirements and Rank”) To see if machines in your pool have other attributes defined, use: –condor_status -long A custom-defined attribute might not be defined on all machines in your pool, so you’ll probably want to use “meta- operators”

55. Mateti, Condor 55 ClassAd “Meta-Operators” Meta operators allow you to compare against “UNDEFINED” as if it were a real value: –=?= is “meta-equal-to” –=!= is “meta-not-equal-to” –Color != “Red” (non-meta) would evaluate to UNDEFINED if Color is not defined –Color =!= “Red” would evaluate to True if Color is not defined, since UNDEFINED is not “Red”

56. Mateti, Condor 56 Priorities In Condor User Priorities –Priorities between users in the pool to ensure fairness –The lower the value, the better the priority Job Priorities –Priorities that users give to their own jobs to determine the order in which they will run –The higher the value, the better the priority –Only matters within a given user’s jobs

57. Mateti, Condor 57 User Priorities in Condor Each active user in the pool has a user priority Viewed or changed with condor_userprio The lower the number, the better A given user’s share of available machines is inversely related to the ratio between user priorities. –Example: Fred’s priority is 10, Joe’s is 20. Fred will be allocated twice as many machines as Joe.

58. Mateti, Condor 58 User Priorities in Condor, cont. Condor continuously adjusts user priorities over time –machines allocated > priority, priority worsens –machines allocated < priority, priority improves Priority Preemption –Higher priority users will grab machines away from lower priority users (thanks to Checkpointing…) –Starvation is prevented –Priority “thrashing” is prevented

59. Mateti, Condor 59 Job Priorities in Condor Can be set at submit-time in your description file with: prio = Can be viewed with condor_q Can be changed at any time with condor_prio The higher the number, the more likely the job will run (only among the jobs of an individual user)

60. Mateti, Condor 60 Managing a Large Cluster of Jobs Condor can manage huge numbers of jobs Special features of the submit description file make this easier Condor can also manage inter-job dependencies with condor_dagman –For example: job A should run first, then, run jobs B and C, when those finish, submit D, etc… –We’ll discuss DAGMan later

61. Mateti, Condor 61 Submitting a Large Cluster Each process runs in its own directory: InitialDir = dir.$(process) Can either have multiple Queue entries, or put a number after Queue to tell Condor how many to submit: Queue 1000 A cluster is more efficient: Your jobs will run faster, and they’ll use less space Can only have one executable per cluster: Different executables must be different clusters!

62. Mateti, Condor 62 Inter-Job Dependencies with DAGMan DAGMan handles a set of jobs that must be run in a certain order Also provides “pre” and “post” operations, so you can have a program or script run before each job is submitted and after it completes Robust: handles errors and submit- machine crashes

63. Mateti, Condor 63 Using DAGMan You define a DAG description file, which is similar in function to the submit file you give to condor_submit DAGMan restrictions: –Each job in the DAG must be in its own cluster (for now) –All jobs in the DAG must have a User Log and must share the same file

64. Mateti, Condor 64 DAGMan Description File # is a comment First section names the jobs in your DAG and associates a submit description file with each job Second (optional) section defines PRE and POST scripts to run Final section defines the job dependencies

65. Mateti, Condor 65 Example DAGMan File Job A A.submit Job B B.submit Job C C.submit Job D D.submit Script PRE D d_input_checker Script POST A a_output_processor A.out PARENT A CHILD B C PARENT B C CHILD D

66. Mateti, Condor 66 Setting up a DAG for Condor Create all the submit description files for the individual jobs Prepare any executables you plan to use Can have a mix of Vanilla and Standard jobs Setup any PRE/POST commands or scripts you wish to use

67. Mateti, Condor 67 Submitting a DAG to Condor condor_submit_dag DAG-description-file This will check your input file for errors and submit a copy of condor_dagman as a scheduler universe job with all the necessary command-line arguments

68. Mateti, Condor 68 Removing a DAG On shutdown, DAGMan will remove any jobs that are currently in the queue that are associated with its DAG Once all jobs are gone, DAGMan itself will exit, and the scheduler universe job will be removed from the queue

69. Mateti, Condor 69 Typical Problems Special requirements expressions for vanilla jobs You didn’t submit it from a directory that is shared Condor isn’t running as root You don’t have your file permissions setup correctly

70. Mateti, Condor 70 Special Requirements Expressions for Vanilla Jobs When you submit a vanilla job, Condor automatically appends two extra Requirements: –UID_DOMAIN == –FILESYSTEM_DOMAIN == Since there are no remote system calls with Vanilla jobs, they depend on a shared file system and a common UID space to run as you and access your files

71. Mateti, Condor 71 Special Requirements Expressions for Vanilla Jobs By default, each machine in your pool is in its own UID_DOMAIN and FILESYSTEM_DOMAIN, so your pool administrator has to configure your pool specially if there really is a common UID space and a network file system If you don’t have an account on the remote system, Vanilla jobs won’t work

72. Mateti, Condor 72 Shared Files for Vanilla Jobs May be not all directories are shared Initialdir = /tmp will probably cause trouble for Vanilla jobs! You must be sure to set Initialdir to a shared directory (or cd into it to run condor_submit) for Vanilla jobs

73. Mateti, Condor 73 Why Don’t My Jobs Run? Try condor_q -analyze Try specifying a User Log for your job Look at condor_userprio: maybe you have a low priority and higher priority users are being served Problems with file permissions or network file systems Look at the SchedLog

74. Mateti, Condor 74 Using condor_q -analyze Analyzes your job’s ClassAd, get all the ClassAds of the machines in the pool, and tell you what’s going on: Will report errors in your Requirements expression (impossible to match, etc.) Will tell you about user priorities in the pool (other people have better priority)

75. Mateti, Condor 75 Looking at condor_userprio You can look at condor_userprio yourself If your priority value is a really high number (because you’ve been running a lot of Condor jobs), other users will have priority to run jobs in your pool

76. Mateti, Condor 76 File Permissions in Condor If Condor isn’t running as root, the condor_shadow process runs as the user the condor_schedd is running as (usually “condor”) You must grant this user write access to your output files, and read access to your input files (both STDOUT, STDIN from your submit file, as well as files your job explicitly opens)

77. Mateti, Condor 77 File Permissions in Condor Often, there will be a “condor” group and you can make your files owned and write- able by this group For vanilla jobs, even if the UID_DOMAIN setting is correct, and they match for your submit and execute machines, if Condor isn’t running as root, your job will be started as user Condor, not as you!

78. Mateti, Condor 78 Problems with NFS in Condor For NFS, sometimes the administrators will setup read-only mounts, or have UIDs remapped for certain partitions (the classic example is root = nobody, but modern NFS can do arbitrary remappings)

79. Mateti, Condor 79 Problems with NFS in Condor If your pool uses NFS automounting, the directory that Condor thinks is your InitialDir might not exist on a remote machine With automounting, you always need to specify InitialDir explicitly –InitialDir = /home/me/...

80. Mateti, Condor 80 Problems with AFS in Condor If your pool uses AFS, the condor_shadow, even if it’s running with your UID, will not have your AFS token. You must grant an unauthenticated AFS user the appropriate access to your files Some sites provide a better alternative that world-writable files –Host ACLs –Network-specific ACLs

81. Mateti, Condor 81 Looking at the SchedLog Looking at the log file of the condor_schedd, the “SchedLog” file can possibly give you a clue if there are problems. Find it with: condor_config_val schedd_log You might need your pool administrator to turn on a higher “debugging level” to see more verbose output

82. Mateti, Condor 82 Other User Features Submit-Only installation Heterogeneous Submit PVM jobs

83. Mateti, Condor 83 Submit-Only Installation Can install just a condor_master and condor_schedd on your machine Can submit jobs into a remote pool Special option to condor_install

84. Mateti, Condor 84 Heterogeneous Submit The job you submit doesn’t have to be the same platform as the machine you submit from –Maybe you have access to a pool that is full of Alphas, but you have a Sparc on your desk, and moving all your data is a pain You can take an Alpha binary, copy it to your Sparc, and submit it with a requirements expression that says you need to run on ALPHA/OSF1

85. Mateti, Condor 85 PVM Jobs in Condor Condor can run parallel applications –PVM applications now –Future work includes support for MPI Master-Worker Paradigm What does Condor-PVM do? How to compile and submit Condor-PVM jobs

86. Mateti, Condor 86 Master-Worker Paradigm Condor-PVM is designed to run PVM applications based on the master-worker paradigm. Master –has a pool of work, sends pieces of work to the workers, manages the work and the workers Worker –gets a piece of work, does the computation, sends the result back

87. Mateti, Condor 87 What does Condor-PVM do? Condor acts as the PVM resource manager. All pvm_addhost requests get re-mapped to Condor. –Condor dynamically constructs PVM virtual machines out of non-dedicated desktop machines. When a machine leaves the pool, the user gets notified via the normal PVM notification mechanisms.

88. Mateti, Condor 88 Submission of Condor-PVM jobs Binary Compatible –Compile and link with PVM library just as normal PVM applications. No need to link with Condor. In the submit description file, set: universe = PVM machine_count =..

89. Mateti, Condor 89 Resource Agent Configuration Expressions STARTSTART WANT SUSPEND SUSPENDSUSPEND VACATEVACATE WANT VACATE KILLKILL True False

90. Mateti, Condor 90 Resource Agent Configuration Default Setup WANT_VACATE : True WANT_SUSPEND : True START : Keyboard_Idle && CPU_Idle SUSPEND : Keyboard_Busy || CPU_Busy CONTINUE : Keyboard and CPU idle again VACATE : If Suspended > 10 minutes KILL : If spent > 10 minutes in VACATE state

91. Mateti, Condor 91 condor_master Watches/restarts other daemons Sends Email if suspicious problems arise Runs condor_preen Provides administrator remote control

92. Mateti, Condor 92 Condor Administrator Commands condor_off [ hostname … ] condor_on condor_restart condor_reconfig condor_vacate Can be used by the Owner also

93. Mateti, Condor 93 Host-based Access Control HOST_ALLOW and HOST_DENY to grant machines (subnets, domains) different access levels: READ access WRITE access ADMINISTRATOR access OWNER access

94. Mateti, Condor 94 Host-based Access Control Ex. HOSTDENY_READ = *.com HOSTALLOW_WRITE = *.cs.wright.edu HOSTDENY_WRITE = ppp*.wright.edu, 172.44.* HOSTALLOW_ADMINISTRATOR = osis111.cs.wright.edu HOSTALLOW_OWNER = $(FULL_HOSTNAME), $(HOSTALLOW_ADMINISTRATOR)

95. Mateti, Condor 95 Configuration File Hierarchy condor_config –Pool-wide default –Condor pool administrator’s requirements condor_config.local –Overrides for a specific machine –Reflects Owner’s requirements condor_config.root –System Administrator requirements

96. Mateti, Condor 96 Obtaining Condor Condor accounts available! E-mail miron@cs.wisc.edu Condor executables can be downloaded from http://www.cs.wisc.edu/condor http://www.cs.wisc.edu/condor Complete Users and Administrators manual http://www.cs.wisc.edu/condor/manual http://www.cs.wisc.edu/condor/manual