1. What Good is a Scheduling Competition? - Insights from the IPC Terry Zimmerman Carnegie Mellon University, Robotics Institute 5000 Forbes Avenue, Pittsburgh, PA wizim@cs.cmu.edu

2. The International Planning Competition: An exemplar for a scheduling competition? Close (and growing closer) relationship between planning & scheduling − How would a scheduling competition distinguish itself? (a variety of ‘scheduling domains’ have been featured in IPC events) − Same question relative to the various other competitions (e.g. Benedetti, Pecora, Policella 2007) 5 competitions held to date (1998, 2000, 2002, 2004, 2006) −Young enough: initial competition setup/design issues in recent memory −Old enough: Good examples of what worked / didn’t work −Learning curve over initial years is of interest It’s the only computational competition I have experience with…

3. Stated general goals of the IPC analyzing and advancing the planning state-of- the-artanalyzing and advancing the planning state-of- the-art providing new benchmarks and a representation formalism to aid planner comparison and evaluationproviding new benchmarks and a representation formalism to aid planner comparison and evaluation emphasizing new research issues and directionsemphasizing new research issues and directions promoting applicability of planning technologypromoting applicability of planning technology (disseminating as much performance data as possible to the community)(disseminating as much performance data as possible to the community)

4. Major Focus: –Non-temporal ‘classical’ planning only –No explicit resource modeling or metric values Domain Language extensions & Domains PDDL introduced, 6 domains Competitors: 5 planners entered Overview of the International Planning Competitions First IPC: 1998 Pittsburgh, PA Blackbox, STAN, HSP, IPP, SGP Blackbox, STAN, HSP, IPP, SGP -all but HSP are Graphplan based -all but HSP are Graphplan based Results: No clear-cut winner. ‘Big’ plans: 30- 40 steps, Max solution sizes >100 steps

5. Focus: –largely ‘classical’ planning, limited metric values –2 tracks: 1) Fully automated 2) Hand tailored Minor refinement of PDDL, 5 domains Competitors: 17 planners entered Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS Results: Fully automated> Top performers vary by domains – FF, STAN, MIPs, HSP2, GRT scale over the 5 domains Hand-tailored> TALplanner dominates (scaled to 500 blocks, ~1.5s), SHOP often gets shorter length plans Overview of the International Planning Competitions Second IPC: 2000 Breckenridge, CO

6. Focus: –extension to temporal planning –extension to numeric constraints & fluents –2 tracks: 1) Fully automated 2) Hand tailored Extended PDDL to support temporal & numeric features Competitors: 14 planners entered Overview of the International Planning Competitions Third IPC: 2002 Toulouse, Fr.

7. PDDL extended for both tracks: Deterministic –Derived predicates, Limited exogenous events Deterministic –Derived predicates, Limited exogenous events Probabilistic –Created PPDDL: effects of actions may have discrete Probabilistic –Created PPDDL: effects of actions may have discrete outcome probs & probabilistic initial state literals outcome probs & probabilistic initial state literals Domains: 7 for deterministic track (2 replays from IPC-3), 8 for prob. track Focus: –development of benchmark domains close to applications and diverse in structure –optimal planners separated from sub-optimal –Introduced uncertainty (probabilistic action effects) Limitation: fully observable domains, discrete distr. –2 tracks: 1) Deterministic 2) Probabilistic Overview of the International Planning Competitions Fourth IPC: 2004 Whistler, B.C Competitors: 19 deterministic planners: Competitors: 19 deterministic planners: Optimal – BFHSP, CPT, HSP*-a, Optiplan, SemSyn, SATPLAN-04, TP4-04 Optimal – BFHSP, CPT, HSP*-a, Optiplan, SemSyn, SATPLAN-04, TP4-04 Sub-optimal - CRIKEY, FAP, Fast Downward, Fast Diag. Downward, LPG-TD, Macro-FF, Sub-optimal - CRIKEY, FAP, Fast Downward, Fast Diag. Downward, LPG-TD, Macro-FF, Marvin, Optop, P-MEP, Roadmapper, SGPlan, Tilsapa, YAHSP, Marvin, Optop, P-MEP, Roadmapper, SGPlan, Tilsapa, YAHSP, FF, MIPS, & LPG from IPC-3 also run where capable. FF, MIPS, & LPG from IPC-3 also run where capable. 10 probabilistic planners: mGPT, Purdue-Humans, Classy, FF-rePlan, 10 probabilistic planners: mGPT, Purdue-Humans, Classy, FF-rePlan, NMRDPP, ProbaPOP, FCPlanner, CERT NMRDPP, ProbaPOP, FCPlanner, CERT

8. PDDL extended for both tracks: Deterministic –Derived predicates, Limited exogenous events Deterministic –Derived predicates, Limited exogenous events Probabilistic –Created PPDDL: effects of actions may have discrete Probabilistic –Created PPDDL: effects of actions may have discrete outcome probs & probabilistic initial state literals outcome probs & probabilistic initial state literals Domains: 7 for deterministic track (2 replays from IPC-3), 8 for prob. track 8 for prob. track Focus: 2 major tracks- Deterministic: fully deterministic & observable (previously called "classical" planning). Subtracks- Optimal Satisficing (sub-optimal) Deterministic: fully deterministic & observable (previously called "classical" planning). Subtracks- Optimal Satisficing (sub-optimal) Non-deterministic: 2 subtracks: 1) Conformant planning: nondeterministic problems for which planners must produce a contingency-safe and linear solution. 2) Probabilistic planning: Focus on real-time decision making not complete policies. Non-deterministic: 2 subtracks: 1) Conformant planning: nondeterministic problems for which planners must produce a contingency-safe and linear solution. 2) Probabilistic planning: Focus on real-time decision making not complete policies. Overview of the International Planning Competitions Fifth IPC: 2006 English Lakes, U.K

9. IPC Goals “analyzing and advancing the planning state-of-the-art” Visibility for the competition’s focal problems/ algorithms/tracks will likely increase across the diverse & broad scheduling community Relative maturity of planning / scheduling −Planning advances have been driven more by perceived need to expand model expressiveness −Scheduling advances have come more from imminent and immediate applications

10. IPC Goals “providing new benchmarks and a representation formalism to aid planner comparison and evaluation” There are many existing scheduling benchmarks –But tend to be ‘classical’ in that they don’t include breadth of constraints found in practical apps. There is no existing broadly recognized scheduling domain description language (i.e. no ‘SDDL’) Like PDDL creation of an SDDL may facilitate comparisons of scheduling paradigms across diverse problems Like PDDL creation of an SDDL may facilitate comparisons of scheduling paradigms across diverse problems Differences: modeling of resources as 1 st class objects Differences: modeling of resources as 1 st class objects Shed light on relative strengths of planning/scheduling approaches to similar problems: Translate the several IPC ‘scheduling’ domains into SDDL….

11. IPC Goals “emphasizing new research issues and directions” tasks with uncertain durations and/or outcomes tasks with uncertain durations and/or outcomes scheduling/ rescheduling to keep pace with execution scheduling/ rescheduling to keep pace with execution distributed or multi-agent scheduling distributed or multi-agent scheduling trade-offs in schedule robustness vs. quality/utility trade-offs in schedule robustness vs. quality/utility

12. IPC Goals “promoting applicability of planning technology” (arguably) Demonstrating and promoting applicability is a larger concern at this time for planning than scheduling

13. IPC Goals “disseminating as much performance data as possible to the community” IPC experience: few competing systems are effective, let alone dominate, across many domains and tracks --Even more likely to be the case for scheduling systems, at least in early competitions. Must motivate competition to generate this data Performance visibility improves with a successful competition

14. The tally: So expect recruitment for Scheduling Competition committees to get underway shortly (!)