Peter Couvares Associate Researcher, Condor Team Computer Sciences Department University of Wisconsin-Madison The Bologna Batch System: Flexible Policy with Condor

The Bologna Batch System › Custom batch scheduling system for local users at INFN in Bologna, Italy.  “Istituto Nazionale di Fisica Nucleare”  Dr. Paolo Mazzanti initiated the idea. › Implement on a small subset of machines within the larger nationwide INFN Condor pool:  INFN Condor Pool: ~300 CPUs  INFN-Bologna Condor: ~100 CPUs  Bologna Batch System: ~50 CPUs

The Bologna Batch System › Paolo wanted two things:  Resources to remain in the larger INFN Condor pool  But local jobs prioritized and managed specially & differently than elsewhere at INFN.  Local control over policies of local resources -- without abandoning commitment to a shared resource pool.

Where We Started › Basic Condor Policy  Opportunistic resources Jobs only run when machines are otherwise idle Jobs can be preempted for machine owners or higher-priority users  Fair-share across INFN pool Highest priority user in the pool gets first crack at a given resource The more you use, the worse your priority becomes › Some problems:  Long-running vanilla jobs (with no checkpointing) were frequently preempted before running to completion  Users dislike waiting for a resource if they only want to run a short job  High-priority users from other INFN sites running on local resources while lower-priority local users wait.

BBS Policy Requirements › Prioritize local work  Share resources, but run outside jobs as backfill › Treat local servers as “dedicated” resources for local jobs, but “opportunistic” resources for other jobs.  Run outside Condor jobs only if the server is idle.  Run local batch jobs regardless of other system load or console activity.  Preempt outside Condor jobs to allow local batch jobs to run, but don’t preempt local jobs for outside work.

BBS Policy Requirements › Ensure resource availability for both short and long-running jobs  Prioritize short batch jobs so that they are never kept waiting by long batch jobs.  Prevent long batch jobs from being preempted or starved by short jobs. › Never waste resources  No idle CPUs when jobs are waiting to run!  No preemption of vanilla jobs! Preemption ideal if you can checkpoint, but here we can’t…

A Contradiction! › No way to guarantee resource availability for short or long jobs without “reserving” some CPUs for each… ›...But no way to avoid idle CPUs without allowing them to start any kind of job:  If CPUs reserved for short jobs are used for long jobs, they become unavailable to run short jobs.  If CPUs reserved for short jobs are not used for long jobs, they’re being wasted when there are no short jobs to run. › What to do, what to do…

A Solution! › Allow resources to be temporarily overcommitted  We treat one CPU as two…  On a two-CPU machine, define four Condor VMs (virtual machines): two for short jobs and two for long jobs. › Allow jobs to be suspended rather than preempted  Think of as “checkpointing to swap”… › OR allow jobs to be “de-prioritized” temporarily  If memory is adequate, allow “suspended” long jobs to continue running at a poor OS priority and steal cycles whenever “active” short jobs are busy doing I/O.

Everybody wins! › Short jobs start right away on dedicated “short” VMs › Long jobs aren’t preempted by short jobs, but rather suspend temporarily or run at a lower priority. › Outside jobs run only when no Bologna jobs waiting. › All CPUs available to all types of jobs.  No idle CPUs when jobs are waiting.

Okay, how? › Flipside of flexibility is complexity! › It’s pretty cool that Condor allows you to combine dedicated and opportunistic scheduling in one system, but it takes a bit of work to get it all set up…  It took the world-experts in writing Condor policy expressions (that’s us) many weeks of concerted effort to get this all working smoothly. › Luckily for y’all, we’ve already done the hard part, and now you can copy it.

Copy it from where? › Bologna Batch System document  › A detailed walk-through of the specific policies and the necessary Condor configuration to make each one work. › Line by line examples of how we implemented each. › A work in progress.  Just this February, we added material.  Open to contributions – what clever things have you done? › What’s in it? Let’s take a look…

Advanced Policies In the BBS Document › How to mark certain jobs as being of one type or another.  (And how to verify jobs are what they say they are.) › How to mark certain machines as being of one type or another. › How to configure multiple Condor VMs on a single CPU. › How to implement different policies for different VMs on the same machine.  New: how to make one VM aware of what’s running on another VM

Simple for Users › Although policy is complicated, the interface for users is kept simple:  Users call bbs_submit_long or bbs_submit_short, just as they would condor_submit… Short jobs start quickly, but those that run for >1 hour are killed. Long jobs will run to completion...  bbs_submit_* scripts automatically add the appropriate classad attributes to the job to take advantage of the long or short running VMs on Bologna Batch Servers.

What About COD? › Condor’s “Computing On Demand” › Designed to solve a similar problem  short-lived interactive computations that need to run immediately, without preempting longer batch jobs. › Some differences:  COD focused on truly interactive work, not batch computation  BBS focused on local control of a shared resource, and balancing the needs of short and long-running batch jobs  COD tasks not represented as jobs in a queue, accounted for resource usage, etc. – they “just run”. COD is more of a instant “Condor rsh”  BBS preserves normal Condor job management structure – jobs are submitted like always, accounted for in the same way, etc.

Any Questions? › Ask me now... › Feel free to me at › Check the Bologna Batch System document at › Thanks!