1. Accounting, Group Quotas, and User Priorities

2. Why are you here? To learn about
How Condor chooses the next job to run
How you can change which job will run
How you can prioritize jobs by project instead of by user
How you can group users into different projects
How you can assign usage minimums to groups of users

3. What job runs next? A condor “queue” is not FIFO!
Determined by a balancing the wants & needs of three entities
The user (schedd)
The pool administrator (negotiator)
The machine owner (startd)
All comes together in the negotiation cycle

4. negotiator collector startd startd schedd schedd startd
1. Startds send machine ads
2. Schedds send submittor ads

5. Machine Ads condor_status –l romano.cs.wisc.edu
MachineMyType = "Machine"
TargetType = "Job"
Name =
Machine = "romano.cs.wisc.edu“
Requirements = LoadAvg < 0.5
Rank = 0.0
Disk =
LoadAvg =
KeyboardIdle =
Memory = 512
Cpus = 1
Mips =

6. Submittor Ads condor_status -sub matthew@wisc.edu -l
MyType = "Submitter"
TargetType = ""
Machine = "rosalind.cs.wisc.edu"
ScheddIpAddr = "< :42190>"
Name =
RunningJobs = 1
IdleJobs = 1
HeldJobs = 0
MaxJobsRunning = 500
StartSchedulerUniverse = TRUE
MonitorSelfImageSize =
MonitorSelfResidentSetSize = 2432

7. negotiator collector Let’s look inside the negotiator during a
startd
Let’s look inside the negotiator during a
negotiation cycle…
startd
schedd
schedd
startd
1. Startds send machine ads
2. Schedds send submittor ads

8. Inside the Negotiator…

9. negotiator
accountant

10. Negotiation Cycle Get all startd and submittor ads
Get user priorities for all submitters, or accounting principles. (via Name attribute in submitter ad)
Sort submitter ads
Talk to schedds in accounting principle order
Schedds send requests one at a time, sorted by job priority

11. For Each Job:  Job Ad’s RANK  NEGOTIATOR_POST_JOB_RANK
Find all machine ads that match
Sort machine ads that match by:
 NEGOTIATOR_PRE_JOB_RANK
 Job Ad’s RANK
 NEGOTIATOR_POST_JOB_RANK
Is the machine ad candidate already running a job? Priority preemption if PREEMPTION_REQUIREMENTS evaluates to True.
Give the schedd the match, or tell it no match found. Schedd responds w/ next request (maybe skipping to the current AutoCluster).

12. Some observations
Job priority (condor_prio) will not allow one user to run ahead of another user.
Job priority is specific per user per schedd.

13. Examples Job says : Rank = Memory
Config file does not define: NEGOTIATOR_PRE_JOB_RANK
User will then ALWAYS get the highest memory machine, even if already being used by a lower priority user.

14. Examples Job says : Rank = Memory Config file says:
NEGOTIATOR_PRE_JOB_RANK =
RemoteUser =?= UNDEFINED
User will then get the highest IDLE memory machine, and will only preempt a user if there are no idle machines match.

15. Accounting Groups I don’t care about WHO submitted the job.
How do I change the accounting principle?

16. Account Groups, cont In job submit file executable = foo
universe = vanilla
+AccountingGroup = “Project44”
queue

17. A given group should have priority on 50 nodes of my 500 machine cluster.
How?
Answer A: Startd Rank
Answer B: Group Quotas

18. Group Quotas – Config Params
GROUP_NAMES - list the recognized group names.Example:       GROUP_NAMES = group-cms, group-infn GROUP_QUOTA_<groupname> - the number of machines 'owned' by
this group.    Example:       GROUP_QUOTA_group-cms = 10       GROUP_QUOTA_group-cms = 5 GROUP_AUTOREGROUP - set this to either True or False.  Defaults to      false.  If true, then users who submitted to a specific group      will also negotiate a second time with the "none" group, allowing      group jobs to be matched w/ idle machines even if the group is      overquota.

19. Negotiation w/ Group Quotas
Matchmaker first negotiates for groups, sorted by how far they are under quota.
Negotiation within a group follows the exact same algorithm as before.
THEN, negotiate for all users that are not in a group as before.

20. Lots o “tools” to get what you want
USER: Job Requirements, Job Rank
ADMIN: user priorities, accounting groups, accounting group quotas, preemption_requirements, negotiator_pre|post_job_rank.
OWNER: Machine Requirements, Machine Rank.

21. Challenge Question
I want LOW, MED, and HIGH strict priority job types. But at each priority level, I want fair share.
HOW??

22. Questions?