1. Learning Control Knowledge for Planning Yi-Cheng Huang

2. Outline I.Brief overview of planning II.Planning with Control knowledge III.Learning control knowledge IV.Conclusion

3. I. Overview of Planning Planning - a very general framework for many applications:  Robot control;  Airline scheduling;  Hubble space telescope control. Planning – find a sequence of actions that leads from an initial state to a goal state.

4. Planning Is Difficult – Abundance of Negative Complexity Results Domain-independent planning: PSPACE- complete or worse (Chapman 1987; Bylander 1991; Backstrom 1993). Domain-dependent planning: NP-complete or worse (Chenoweth 1991; Gupta and Nau 1992). Approximate planning: NP-complete or worse (Selman 1994).

5. Recent State-of-the-art Planners Constraint-based Planners – Graphplan, Blackbox. Heuristic Search Planners – HSP, FF. Both kinds of planners can solve problems in seconds or minutes that traditional planners take hours or days.

6. Graphplan (Blum & Furst, 1995) Facts... FactsActions... Search on planning graph to find plan Time iTime i+1

7. Blackbox (Kautz & Selman, 1999) Satisfiability Tester ( Chaff,WalkSat, Satz, RelSat,...) plan problem

8. Heuristic Search Based Planning (Bonet & Geffner, ‘97) Use various heuristic functions to approximate the distance from the current state to the goal state based on the planning graph. Use Best-First Search or A* search to find plans.

9. II. Planning With Control General focus on planning: avoid search as much as possible. Many real-world applications are tailored and simplified by domain specific knowledge. TLPlan is an efficient planner when using control knowledge to guild a forward-chaining search planner (Bacchus & Kabanza 2000).

10. TLPlan Temporal Logic Control Formula

11. A Simple Control Rule Example Goal Do NOT move an object at the goal location (goal (at (obj loc)) at (obj loc)) Temporal logic operator: “always”“next”

12. Question: Whether the same level of control can be effectively incorporated into constraint-based planner?

13. I.Rules involves only static information. II.Rules depends on the current state. III.Rules depends on the current state and require dynamic user-defined predicates. Control Rules Categories

14. Category I Control Rules (only depends on goal; toy example) Do NOT unload an package from an airplane if the current location is not in the package’s goal Goal L a a a

15. Pruning the Planning Graph Category I Rules Facts Actions...

16. Effect of Graph Pruning

17. Category II Control Rules L Do NOT move an airplane if there is an object in the airplane that needs to be unloaded at that location. a

18. Control by Adding Constraints Temporal Logic Control Rules Planning FormulaConstraints Clauses

19. Rules Without Compact Encoding NYC SFO ORL Do NOT move a vehicle unless (a) there is an object that needs to be picked up (b) there is an object in the vehicle that needs to be unloaded Goal DC a a b b

20. Complex Encoding for Category III Rules Need to define extra predicates: need_to_move_by_airplane; need_to_unload_by_airplane Introduce extra literals and clauses. O(mn) ground literals; O(mn+km^2) clauses at each time step. m: #cities, n: #objects, k: #airports No easy encoding for category III rules. However, it appears category I & II rules do most of work.

21. Blackbox with Control Knowledge (Logistics domain with hand-coded rules) Note: Logarithmic time scale

22. Comparison of Blackbox and TLPlan (Run Time)

23. Comparison of Blackbox and TLPlan (parallel plan length; “plan quality”)

24. Summary Adding Control Knowledge We have shown how to add declarative control knowledge to a constraint-based planners by using temporal logic statements. Adding such knowledge gives significant speedups (up to two orders of magnitude ). Pure heuristic search with control can be still faster but with much lower plan quality.

25. III. Can we learn domain knowledge from example plans?

26. Motivation Control Rules used in TLPlan and Blackbox are hand-coded. Idea: learn control rules on a sequence of small problems solved by planner.

27. Learning System Framework Plan Justification / Type Inference Blackbox Planner Problem ILP Learning Module / Verification Control Rules

28. Target Concepts for Actions Action Select Rule: indicate conditions under which the action can be performed immediately. Action Reject Rule: indicate conditions under which it must not be performed.

29. Basic Assumption on Learning Control Plan found by planner on simple problems are optimal or near-optimal. Actions appear in an optimal plan must be selected. Actions that can be executed but do not appear in the plan must be rejected.

30.  Real action: action appears in the plan.  Virtual action: action that its preconditions are hold but does not appear in the plan. Definition

31. An Toy Planning Example Goal Initial BOSSFONYC Initial ab ab

32. Real & Virtual Actions for UnloadAirplane Time 1: LoadAirplane (P a BOS) Time 2: FlyAirplane (P SFO NYC) UnloadAirplane (P a BOS) Time 3: LoadAirplane (P b NYC) UnloadAirplane (P a NYC) Time 4: FlyAirplane (P NYC SFO) UnloadAirplane (P a NYC) UnloadAirplane (P b NYC) Time 5: UnloadAirplane (P a SFO) UnloadAirplane (P b SFO) Real Virtual

33. Heuristics for Extracting Examples

34. Rule Induction Literal:  Xi = Xj, ex., loc1 = loc2  P(X1,…, Xn), ex., at (pkg, loc)  goal (P(X1,…, Xn)), ex., goal (at (pkg, loc))  negation of the above Based on Quinlan’s FOIL (Quinlan 1990; 1996).

35. Reject Rule: UnloadAirplane timeplnpkgapt +2PaBOS +3PaNYC +4Pa +4Pa -5PaSFO -5Pa UnloadAirplane (pln pkg apt)

36. Reject Rule: UnloadAirplane UnloadAirplane (pln pkg apt) goal(at (pkg loc)) timeplnpkgaptloc +2PaBOSSFO +3PaNYCSFO +4PaNYCSFO +4PaNYCSFO -5Pa -5Pa

37. Reject Rule: UnloadAirplane UnloadAirplane (pln pkg apt) goal(at (pkg loc)) ^ (apt != loc) timeplnpkgaptloc +2PaBOSSFO +3PaNYCSFO +4PaNYCSFO +4PaNYCSFO -5Pa -5Pa

38. Learning Time

39. Logistics Domain

40. Learned Logistics Control Rules If an object’s goal location is at different city, do NOT unload the object from airplanes. Unload an object from a truck if the current location is an airport and it is not in the same city as the package’s goal location.

41. Briefcase Domain

42. Grid Domain

43. Gripper Domain

44. Mystery Domain

45. Tireworld Domain

46. Summary of Learning for Planning Introduced inductive logic programming methodology into constraint-based planning framework to obtain “trainable planner”. Demonstrated clear practical speedups on range of benchmark problems.

47. IV. Single-agent vs. Multi-agent planning  Observations: heuristic planners degrade rapidly in multi-agent settings. They tend to assign all work to a single agent.  We studied this phenomenon by exploring different work-load distributions.

48. Force the Planners There is no easy way to modify the heuristic search planners to find better quality plans. Limit the number of feature actions an agent can perform to force the planners to find plans with the same level of participation of all agents.

49. Sokoban Domain

50. Restricted Sokoban Domain

51. Complexity Analysis on Restricted Domain C.B.PH.P. SokobanPSPACE-Complete (Culberson, 1997) V RocketNP-Complete (reduce from vertex feedback) V GridPolynomial SolvableV ElevatorPolynomial SolvableV

52. Conclusions (a) Demonstrated how performance of state-of-the- art general purpose planning systems can be boosted by incorporating control knowledge. Knowledge encoded in purely declarative form using temporal logic formulas. Obtained up to 2 orders of magnitude speedup on series of benchmarks.

53. Conclusions (b) Demonstrated feasibility of a “trainable” planning system: system learns domain / control knowledge from many small example plans. Based on concepts from inductive logic programming. Learned knowledge in temporal logic form. First demonstration of practical speedups using learning in a planning system on realistic benchmarks. Approach avoids learning “accidental truths” that can hurt system performance (problem in earlier systems)

54. Conclusions (c) Uncovered link between performance of planners and inherent complexity of planning task. Heuristic search planners work well on problems solvable in poly time with specialized algorithms. Constraint-based planner dominate on NP- complete planning tasks.

55. Conclusion Comparison of constraint-based planner and heuristic search planner shows that they complement each other on different domains. Hand-coded control knowledge can be effectively applied in constraint-based planners.

56. Conclusion (cont.) Our learning system is simple and modular; learning time is short. Learned rules are on par with hand-coded ones and shown to improve the performance for over two orders of magnitude. Learned rules are in logic form and can be used on other planning systems.

57. Demonstrated a way for effectively learning domain knowledge from small general plans. Learned control knowledge boosts performance on larger problems. First clear demonstration of boosting plan system performance through learning. Declarative, logic-based approach is general and fits wide range of planning applications.

58. The End