Accounting, Group Quotas, and User Priorities

Why are you here? To learn about How Condor chooses the next job to run How you can change which job will run How you can prioritize jobs by project instead of by user How you can group users into different projects How you can assign usage minimums to groups of users

What job runs next? A condor “queue” is not FIFO! Determined by a balancing the wants & needs of three entities The user (schedd) The pool administrator (negotiator) The machine owner (startd) All comes together in the negotiation cycle

negotiator collector startd startd schedd schedd startd 1. Startds send machine ads 2. Schedds send submittor ads

Machine Ads condor_status –l romano.cs.wisc.edu MachineMyType = "Machine" TargetType = "Job" Name = "vm6@romano.cs.wisc.edu" Machine = "romano.cs.wisc.edu“ Requirements = LoadAvg < 0.5 Rank = 0.0 Disk = 2019048 LoadAvg = 0.000000 KeyboardIdle = 1018497 Memory = 512 Cpus = 1 Mips = 124122

Submittor Ads condor_status -sub matthew@wisc.edu -l MyType = "Submitter" TargetType = "" Machine = "rosalind.cs.wisc.edu" ScheddIpAddr = "<128.105.166.39:42190>" Name = “matthew@cs.wisc.edu” RunningJobs = 1 IdleJobs = 1 HeldJobs = 0 MaxJobsRunning = 500 StartSchedulerUniverse = TRUE MonitorSelfImageSize = 9164.000000 MonitorSelfResidentSetSize = 2432

negotiator collector Let’s look inside the negotiator during a startd Let’s look inside the negotiator during a negotiation cycle… startd schedd schedd startd 1. Startds send machine ads 2. Schedds send submittor ads

Inside the Negotiator…

negotiator accountant

Negotiation Cycle Get all startd and submittor ads Get user priorities for all submitters, or accounting principles. (via Name attribute in submitter ad) Sort submitter ads Talk to schedds in accounting principle order Schedds send requests one at a time, sorted by job priority

For Each Job:  Job Ad’s RANK  NEGOTIATOR_POST_JOB_RANK Find all machine ads that match Sort machine ads that match by:  NEGOTIATOR_PRE_JOB_RANK  Job Ad’s RANK  NEGOTIATOR_POST_JOB_RANK Is the machine ad candidate already running a job? Priority preemption if PREEMPTION_REQUIREMENTS evaluates to True. Give the schedd the match, or tell it no match found. Schedd responds w/ next request (maybe skipping to the current AutoCluster).

Some observations Job priority (condor_prio) will not allow one user to run ahead of another user. Job priority is specific per user per schedd.

Examples Job says : Rank = Memory Config file does not define: NEGOTIATOR_PRE_JOB_RANK User will then ALWAYS get the highest memory machine, even if already being used by a lower priority user.

Examples Job says : Rank = Memory Config file says: NEGOTIATOR_PRE_JOB_RANK = RemoteUser =?= UNDEFINED User will then get the highest IDLE memory machine, and will only preempt a user if there are no idle machines match.

Accounting Groups I don’t care about WHO submitted the job. How do I change the accounting principle?

Account Groups, cont In job submit file executable = foo universe = vanilla +AccountingGroup = “Project44” queue

A given group should have priority on 50 nodes of my 500 machine cluster. How? Answer A: Startd Rank Answer B: Group Quotas

Group Quotas – Config Params GROUP_NAMES - list the recognized group names.Example:       GROUP_NAMES = group-cms, group-infn GROUP_QUOTA_<groupname> - the number of machines 'owned' by this group.    Example:       GROUP_QUOTA_group-cms = 10       GROUP_QUOTA_group-cms = 5 GROUP_AUTOREGROUP - set this to either True or False.  Defaults to      false.  If true, then users who submitted to a specific group      will also negotiate a second time with the "none" group, allowing      group jobs to be matched w/ idle machines even if the group is      overquota.

Negotiation w/ Group Quotas Matchmaker first negotiates for groups, sorted by how far they are under quota. Negotiation within a group follows the exact same algorithm as before. THEN, negotiate for all users that are not in a group as before.

Lots o “tools” to get what you want USER: Job Requirements, Job Rank ADMIN: user priorities, accounting groups, accounting group quotas, preemption_requirements, negotiator_pre|post_job_rank. OWNER: Machine Requirements, Machine Rank.

Challenge Question I want LOW, MED, and HIGH strict priority job types. But at each priority level, I want fair share. HOW??

Questions?