1. Scalability of Planning  Before, planning algorithms could synthesize about 6 – 10 action plans in minutes  Significant scale-up in the last 6-7 years  Now, we can synthesize 100 action plans in seconds. Realistic encodings of Munich airport! The primary revolution in planning in the recent years has been domain-independent heuristics to scale up plan synthesis Problem is Search Control!!! …and now for a ring-side retrospective

4. Relevance, Rechabililty & Heuristics Progression takes “applicability” of actions into account –Specifically, it guarantees that every state in its search queue is reachable..but has no idea whether the states are relevant (constitute progress towards top-level goals) SO, heuristics for progression need to help it estimate the “relevance” of the states in the search queue Regression takes “relevance” of actions into account –Specifically, it makes sure that every state in its search queue is relevant.. But has not idea whether the states (more accurately, state sets) in its search queue are reachable SO, heuristics for regression need to help it estimate the “reachability” of the states in the search queue Reachability: Given a problem [I,G], a (partial) state S is called reachable if there is a sequence [a 1,a 2,…,a k ] of actions which when executed from state I will lead to a state where S holds Relevance: Given a problem [I,G], a state S is called relevant if there is a sequence [a1,a2,…,ak] of actions which when executed from S will lead to a state satisfying (Relevance is Reachability from goal state) Since relevance is nothing but reachability from goal state, reachability analysis can form the basis for good heuristics

5. Reachability through progression p pq pr ps pqr pq pqs psq ps pst A1 A2 A3 A2 A1 A3 A1 A3 A4 [ECP, 1997]

6. Planning Graph Basics –Envelope of Progression Tree (Relaxed Progression) Linear vs. Exponential Growth –Reachable states correspond to subsets of proposition lists –BUT not all subsets are states Can be used for estimating non- reachability –If a state S is not a subset of k th level prop list, then it is definitely not reachable in k steps p pq pr ps pqr pq pqs p psq ps pst pqrspqrs pqrstpqrst A1 A2 A3 A2 A1 A3 A1 A3 A4 A1 A2 A3 A1 A2 A3 A4 [ECP, 1997]

7. Planning Graph Basics –Envelope of Progression Tree (Relaxed Progression) Linear vs. Exponential Growth –Reachable states correspond to subsets of proposition lists –BUT not all subsets are states Can be used for estimating non- reachability –If a state S is not a subset of k th level prop list, then it is definitely not reachable in k steps p pq pr ps pqr pq pqs p psq ps pst pqrspqrs pqrstpqrst A1 A2 A3 A2 A1 A3 A1 A3 A4 A1 A2 A3 A1 A2 A3 A4 [ECP, 1997]

8. Don’t look at curved lines for now… Have(cake) ~eaten(cake) ~Have(cake) eaten(cake) Eat No-op Have(cake) eaten(cake) bake ~Have(cake) eaten(cake) Have(cake) ~eaten(cake) Eat No-op Have(cake) ~eaten(cake) Graph has leveled off, when the prop list has not changed from the previous iteration The note that the graph has leveled off now since the last two Prop lists are the same (we could actually have stopped at the Previous level since we already have all possible literals by step 2)

9. Blocks world State variables: Ontable(x) On(x,y) Clear(x) hand-empty holding(x) Stack(x,y) Prec: holding(x), clear(y) eff: on(x,y), ~cl(y), ~holding(x), hand-empty Unstack(x,y) Prec: on(x,y),hand-empty,cl(x) eff: holding(x),~clear(x),clear(y),~hand-empty Pickup(x) Prec: hand-empty,clear(x),ontable(x) eff: holding(x),~ontable(x),~hand-empty,~Clear(x) Putdown(x) Prec: holding(x) eff: Ontable(x), hand-empty,clear(x),~holding(x) Initial state: Complete specification of T/F values to state variables --By convention, variables with F values are omitted Goal state: A partial specification of the desired state variable/value combinations --desired values can be both positive and negative Init: Ontable(A),Ontable(B), Clear(A), Clear(B), hand-empty Goal: ~clear(B), hand-empty All the actions here have only positive preconditions; but this is not necessary

10. onT-A onT-B cl-A cl-B he Pick-A Pick-B onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he

11. onT-A onT-B cl-A cl-B he Pick-A Pick-B onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he St-A-B St-B-A Ptdn-A Ptdn-B Pick-A onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he on-A-B on-B-A Pick-B

12. Estimating the cost of achieving individual literals (subgoals) Idea: Unfold a data structure called “planning graph” as follows: 1. Start with the initial state. This is called the zeroth level proposition list 2. In the next level, called first level action list, put all the actions whose preconditions are true in the initial state -- Have links between actions and their preconditions 3. In the next level, called first level propostion list, put: Note: A literal appears at most once in a proposition list. 3.1. All the effects of all the actions in the previous level. Links the effects to the respective actions. (If multiple actions give a particular effect, have multiple links to that effect from all those actions) 3.2. All the conditions in the previous proposition list (in this case zeroth proposition list). Put persistence links between the corresponding literals in the previous proposition list and the current proposition list. *4. Repeat steps 2 and 3 until there is no difference between two consecutive proposition lists. At that point the graph is said to have “leveled off” The next 2 slides show this expansion upto two levels

13. Using the planning graph to estimate the cost of single literals: 1. We can say that the cost of a single literal is the index of the first proposition level in which it appears. --If the literal does not appear in any of the levels in the currently expanded planning graph, then the cost of that literal is: -- l+1 if the graph has been expanded to l levels, but has not yet leveled off -- Infinity, if the graph has been expanded (basically, the literal cannot be achieved from the current initial state) Examples: h({~he}) = 1 h ({On(A,B)}) = 2 h({he})= 0 How about sets of literals?  see next slide

14. Estimating reachability of sets We can estimate cost of a set of literals in three ways: Make independence assumption H(p,q,r)= h(p)+h(q)+h(r) if we define the cost of a set of literals in terms of the level where they appear together h-lev({p,q,r})= The index of the first level of the PG where p,q,r appear together so, h({~he,h-A}) = 1 Compute the length of a “relaxed plan” to supporting all the literals in the set S, and use it as the heuristic (**) h relax

15. Subgoal interactions Suppose we have a set of subgoals G 1,….G n Suppose the length of the shortest plan for achieving the subgoals in isolation is l 1,….l n We want to know what is the length of the shortest plan for achieving the n subgoals together, l 1…n If subgoals are independent: l 1..n = l 1 +l 2 +…+l n If subgoals have +ve interactions alone: l 1..n < l 1 +l 2 +…+l n If subgoals have -ve interactions alone: l 1..n > l 1 +l 2 +…+l n If you made “independence” assumption, and added up the individual costs of subgoals, then your resultant heuristic will be  perfect if the goals are actually independent  inadmissible (over-estimating) if the goals have +ve interactions  un-informed (hugely under-estimating) if the goals have –ve interactions

16. “Relaxed plan” Suppose you want to find a relaxed plan for supporting literals g1…gm on a k-length PG. You do it this way: –Start at kth level. Pick an action for supporting each gi (the actions don’t have to be distinct—one can support more than one goal). Let the actions chosen be {a1…aj} –Take the union of preconditions of a1…aj. Let these be the set p1…pv. –Repeat the steps 1 and 2 for p1…pv—continue until you reach init prop list. The plan is called “relaxed” because you are assuming that sets of actions can be done together without negative interactions. No backtracking needed! Optimal relaxed plan is still NP-hard

17. onT-A onT-B cl-A cl-B he Pick-A Pick-B onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he St-A-B St-B-A Ptdn-A Ptdn-B Pick-A onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he on-A-B on-B-A Pick-B Relaxed plan for our blocks example

18. h-ind; h-lev; h-relax H-lev is lower than or equal to h-relax H-ind is larger than or equal to H-lev H-lev is admissible H-relax is not admissible unless you find optimal relaxed plan –Which is NP-Hard..

19. Planning Graphs for heuristics  Construct planning graph(s) at each search node  Extract relaxed plan to achieve goal for heuristic p5 q5 r5 p6 o pq o pr o 56 p5p5 pqr56pqr56 o pq o pr o 56 pqrst567pqrst567 o ps o qt o 67 q5q5 qtr56qtr56 o qt o qr o 56 qtrsp567qtrsp567 o qs o tp o 67 r5r5 rqp56rqp56 o rq o rp o 56 rqpst567rqpst567 o rs o qt o 67 p6p6 pqr67pqr67 o pq o pr o 67 pqrst678pqrst678 o ps o qt o 78 1 3 4 1 3 o 12 o 34 2 1 3 4 5 o 12 o 34 o 23 o 45 2 3 4 5 3 5 o 34 o 56 3 4 5 o 34 o 45 o 56 66 7 o 67 1 5 1 5 o 12 o 56 2 1 3 5 o 12 o 23 o 56 2 66 7 o 67 G oGoG G oGoG G oGoG G oGoG G oGoG 1 3 3 5 1 5 h( )=5

21. --4/19 lecture ended here-- Slides beyond this were not discussed in the class

22. Progression Regression How do we use reachability heuristics for regression?

23. Use of PG in Progression vs Regression Progression –Need to compute a PG for each child state As many PGs as there are leaf nodes! Lot higher cost for heuristic computation –Can try exploiting overlap between different PGs –However, the states in progression are consistent.. So, handling negative interactions is not that important Overall, the PG gives a better guidance even without mutexes Regression –Need to compute PG only once for the given initial state. Much lower cost in computing the heuristic –However states in regression are “partial states” and can thus be inconsistent So, taking negative interactions into account using mutex is important –Costlier PG construction Overall, PG’s guidance is not as good unless higher order mutexes are also taken into account Historically, the heuristic was first used with progression planners. Then they used it with regression planners. Then they found progression planners do better. Then they found that combining them is even better. Remember the Altimeter metaphor..

24. What if actions have non-uniform costs?

25. Challenges in Cost Propagation

26. Cost of a set of literals? We can compute a relaxed plan to support those literals –It is clear now that optimal relaxed plan will be NP-hard –Greedy approaches could be used Support the goals using the actions that have the lowest propagated cost

27. PGs for reducing actions If you just use the action instances at the final action level of a leveled PG, then you are guaranteed to preserve completeness –Reason: Any action that can be done in a state that is even possibly reachable from init state is in that last level –Cuts down branching factor significantly –Sometimes, you take more risky gambles: If you are considering the goals {p,q,r,s}, just look at the actions that appear in the level preceding the first level where {p,q,r,s} appear for the first time without Mutex.

29. Negative Interactions To better account for -ve interactions, we need to start looking into feasibility of subsets of literals actually being true together in a proposition level. Specifically,in each proposition level, we want to mark not just which individual literals are feasible, –but also which pairs, which triples, which quadruples, and which n-tuples are feasible. (It is quite possible that two literals are independently feasible in level k, but not feasible together in that level) The idea then is to say that the cost of a set of S literals is the index of the first level of the planning graph, where no subset of S is marked infeasible The full scale mark-up is very costly, and makes the cost of planning graph construction equal the cost of enumerating the full progres sion search tree. –Since we only want estimates, it is okay if talk of feasibility of upto k-tuples For the special case of feasibility of k=2 (2-sized subsets), there are some very efficient marking and propagation procedures. –This is the idea of marking and propagating mutual exclusion relations.

30. Don’t look at curved lines for now… Have(cake) ~eaten(cake) ~Have(cake) eaten(cake) Eat No-op Have(cake) eaten(cake) bake ~Have(cake) eaten(cake) Have(cake) ~eaten(cake) Eat No-op Have(cake) ~eaten(cake) Graph has leveled off, when the prop list has not changed from the previous iteration The note that the graph has leveled off now since the last two Prop lists are the same (we could actually have stopped at the Previous level since we already have all possible literals by step 2)

31. Level-off definition? When neither propositions nor mutexes change between levels

32. Rule 1. Two actions a1 and a2 are mutex if (a)both of the actions are non-noop actions or (b) a1 is any action supporting P, and a2 either needs ~P, or gives ~P. (c) some precondition of a1 is marked mutex with some precondition of a2 Rule 2. Two propositions P1 and P2 are marked mutex if all actions supporting P1 are pair-wise mutex with all actions supporting P2. Mutex Propagation Rules Serial graph interferene Competing needs This one is not listed in the text

33. onT-A onT-B cl-A cl-B he Pick-A Pick-B onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he

34. onT-A onT-B cl-A cl-B he Pick-A Pick-B onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he St-A-B St-B-A Ptdn-A Ptdn-B Pick-A onT-A onT-B cl-A cl-B he h-A h-B ~cl-A ~cl-B ~he on-A-B on-B-A Pick-B

35. Level-based heuristics on planning graph with mutex relations h lev ({p 1, …p n })= The index of the first level of the PG where p 1, …p n appear together and no pair of them are marked mutex. (If there is no such level, then h lev is set to l+1 if the PG is expanded to l levels, and to infinity, if it has been expanded until it leveled off) We now modify the h lev heuristic as follows This heuristic is admissible. With this heuristic, we have a much better handle on both +ve and -ve interactions. In our example, this heuristic gives the following reasonable costs: h({~he, cl-A}) = 1 h({~cl-B,he}) = 2 h({he, h-A}) = infinity (because they will be marked mutex even in the final level of the leveled PG) Works very well in practice H({have(cake),eaten(cake)}) = 2

36. Some observations about the structure of the PG 1. If an action a is present in level l, it will be present in all subsequent levels. 2. If a literal p is present in level l, it will be present in all subsequent levels. 3. If two literals p,q are not mutex in level l, they will never be mutex in subsequent levels --Mutex relations relax monotonically as we grow PG 1,2,3 imply that a PG can be represented efficiently in a bi-level structure: One level for propositions and one level for actions. For each proposition/action, we just track the first time instant they got into the PG. For mutex relations we track the first time instant they went away. 4.PG doesn’t have to be grown to level-off to be useful for computing heuristics 5.PG can be used to decide which actions are worth considering in the search

38. Plan Space Planning: Terminology Step: a step in the partial plan—which is bound to a specific action Orderings: s1<s2 s1 must precede s2 Open Conditions: preconditions of the steps (including goal step) Causal Link (s1—p—s2): a commitment that the condition p, needed at s2 will be made true by s1 –Requires s1 to “cause” p Either have an effect p Or have a conditional effect p which is FORCED to happen –By adding a secondary precondition to S1 Unsafe Link: (s1—p—s2; s3) if s3 can come between s1 and s2 and undo p (has an effect that deletes p). Empty Plan: { S:{I,G}; O:{I<G}, OC:{g1@G;g2@G..}, CL:{}; US:{}}

39. Partial plan representation P = (A,O,L,OC,UL) A: set of action steps in the plan S 0,S 1,S 2 …,S inf O: set of action ordering S i < S j,… L : set of causal links OC: set of open conditions (subgoals remain to be satisfied) UL: set of unsafe links where p is deleted by some action S k p SiSi SjSj p SiSi SjSj S0S0 S1S1 S2S2 S3S3 S inf p ~p g1g1 g2g2 g2g2 oc 1 oc 2 G={g 1,g 2 }I={q 1,q 2 } q1q1 Flaw: Open condition OR unsafe link Solution plan: A partial plan with no remaining flaw Every open condition must be satisfied by some action No unsafe links should exist (i.e. the plan is consistent) POP background

40. Algorithm 1. Let P be an initial plan 2. Flaw Selection: Choose a flaw f ( either open condition or unsafe link ) 3. Flaw resolution: If f is an open condition, choose an action S that achieves f If f is an unsafe link, choose promotion or demotion Update P Return NULL if no resolution exist 4. If there is no flaw left, return P else go to 2. S0S0 S1S1 S2S2 S3S3 S in f p ~p g1g1 g2g2 g2g2 oc 1 oc 2 q1q1 Choice points Flaw selection (open condition? unsafe link?) Flaw resolution (how to select (rank) partial plan?) Action selection (backtrack point) Unsafe link selection (backtrack point) S0S0 S inf g1g2g1g2 1. Initial plan: 2. Plan refinement (flaw selection and resolution): POP background

41. Spare Tire Example

43. Plan-space Planning

44. Plan-space planning: Example