6 EDG Resource Broker Gets Lazy… Addition of a DAGMan callouts DAGMan is given a command (script) to run immediately before submission of job to Condor-G (different than a PRE script on a node) The helper command is passed a copy of the job submit file when DAGMan is about to submit that node in the graph This allows changes to be made to the submit file (i.e. changing globussite attribute) at the last minute
7 Eager Example First Pass of EDG Resource Broker RB DAGMan Condor-G Globus Fabric Site Scheduler callout
8 Moving Condor-G to Just-In-Time Delay the binding of the task (job) to the resource until the resource is ready. Need to know when the resource is ready. One way: unimplemented globus 1.1 queue wait time estimate Not really just-in-time, because of lies, lies lies… Another way… Condor-G Glidein Mechanism.
9 How It Works Schedd LSF Collector Condor-GGlobus Resource 600 Condor jobs
10 How It Works Schedd LSF Collector Condor-GGlobus Resource 600 Condor jobs GlideIn jobs
11 How It Works Schedd LSF Collector Condor-GGlobus Resource GridManager 600 Condor jobs GlideIn jobs
12 How It Works Schedd JobManager LSF Collector Condor-GGlobus Resource GridManager 600 Condor jobs GlideIn jobs
13 How It Works Schedd JobManager LSF Startd Collector Condor-GGlobus Resource GridManager 600 Condor jobs GlideIn jobs
14 How It Works Schedd JobManager LSF Startd Collector Condor-GGlobus Resource GridManager 600 Condor jobs GlideIn jobs
15 How It Works Schedd JobManager LSF User Job Startd Collector Condor-GGlobus Resource GridManager 600 Condor jobs GlideIn jobs
16 A Just-in-time Submit executable = find_particle requirements = TARGET.Arch == Intel/Linux || TARGET.Arch == Sparc/Solaris # job describes the power rank = MFlops * Memory
18 Lots of Tradeoffs… Just-in-Time Pro: Dynamic. Resources can come and go. Can take advantage of changing circumstances. Con: Coordination of multiple resources Eager Pro: Easier to coordinate multiple resources Con: Hard to scale… how to know about all the resources in advance? Con: Plan falls apart if assumptions change.
19 Some observations A complete separation of task from resource is difficult. Lots and lots of structured data required. But this separation is required to in order to achieve Just-In-Time planning. Grid Protocols that do not separate task from resource cannot realistically live on the grid. Virtualization can help.
20 Plan for failure Much effort on how to create a plan. How about a plan for when things fail?
21 Job Failure Policy Expressions Condor/Condor-G augemented so users can supply job failure policy expressions in the submit file. Can be used to describe a successful run, or what to do in the face of failure. on_exit_remove = on_exit_hold = periodic_remove = periodic_hold =
22 Job Failure Policy Examples Do not remove from queue (i.e. reschedule) if exits with a signal: on_exit_remove = ExitBySignal == False Place on hold if exits with nonzero status or ran for less than an hour: on_exit_hold = ((ExitBySignal==False) && (ExitSignal != 0)) || ((ServerStartTime – JobStartDate) < 3600) Place on hold if job has spent more than 50% of its time suspended: periodic_hold = CumulativeSuspensionTime > (RemoteWallClockTime / 2.0)