1. Continuous Time and Resource Uncertainty CSE 574 Lecture Spring ’03 Stefan B. Sigurdsson

2. (Big Mars Rover Picture)

3. Lecture Overview Context –Classical planning –The Mars Rover domain –Relaxing the assumptions –Q: What’s so different? Innovation Discussion

4. (Shakey Picture) Slide shamelessly lifted from http://www.cs.nott.ac.uk/~bsl/G53DIA/Slides/Deliberative-architectures-I.pdf

5. STRIPS-Like Planning Propositional logic Closed world assumption Finite and static Complete knowledge Discrete time No exogenous effects World Description Attainment – “Win or lose” Conjunctions of positive literals Goal Description Conjunctive precondition STRIPS operators Conj. effect (add/delete) Instantaneous Sequential Deterministic Actions  Plan…

6. (Big Mars Rover Picture)

7. The Mars Rover Domain Robot control, with… –Positioning and navigation –Complex choices (goals and actions) –Rich utility model –Continuous time and concurrency –Uncertain resource consumption –Metric quantities –Very high stakes! But alone in a finite, static universe

8. Resources? Metric Quantities? What are those? Various flavors: –Exclusive (camera arm) –Shared (OS scheduling) –Metric quantity (fuel, power, disk space) Uncertainty

9. Alright, Whatsit Really Mean?

10. Is This Really A Planning Problem? Better suited to OR/DT-type scheduling? –Time, resources, metric quantities, concurrency, complicated goals/rewards… Complex, inter-dependent activities –Select, calibrate, use, reuse, recalibrate sensors –OR-type scheduling can’t handle rich choices Insight: Maybe we can borrow some tricks?

11. Can Planners Scale Up? Large plans –Sequences of ~ 100 actions Where do we start? –POP? –MDP? –Graph/SATplan?

12. Can Planners Scale Up? Large plans –Sequences of ~ 100 actions Where do we start? –POP? (Branch factors are too big) –MDP? –Graph/SATplan?

13. Can Planners Scale Up? Large plans –Sequences of ~ 100 actions Where do we start? –POP? (Branch factors are too big) –MDP? (Complete policy is too large) –Graph/SATplan?

14. Can Planners Scale Up? Large plans –Sequences of ~ 100 actions Where do we start? –POP? (Branch factors are too big) –MDP? (Complete policy is too large) –Graph/SATplan? (Discrete representations)

15. Which Extensions First? Metric quantities –Time –Resources Resource Uncertainty Concurrency  What about non-determinism?  Reasonable for Graphplan?

16. A (Very Incomplete) Research Timeline 1971 STRIPS (Fikes/Nilson) 1989 ADL (Pednault) 1991 PEDESTAL (McDermott) 1992 UCPOP (Penberthy/Weld) 1992 SENSp (Etzioni et al.) CNLP (Peot/Smith) 1993 Buridan (Kushmerick et al.) 1994 C-Buridan (Draper et al.) JIC Scheduling (Drummond et al.) HSTS (Muscettola) Zeno (Penb./Weld) Softbots (Weld/Etzioni) MDP (Williamson/Hanks) 1995 DRIPS (Haddawy et al.) IxTeT (Laborie/Ghallab) 1997 IPP (Koehler et al.) Not implemented  ADL impl. Sensing Conformant Contingent Planning + scheduling Metric time/resources Safe planning Dec. theory goals Uncertain utility Shared resources 1998 PGraphplan (Blum/Langford) Weaver (Blythe) PUCCINI (Golden) CGP (Smith/Weld) SGP (Weld et al.) Pgraphplan (Blum/Langford) 1999 Mahinur (Onder/Pollack) ILP-PLAN (Kautz/Walzer) TGP (Smith/Weld) LPSAT (Wolfman/Weld) 2000 T-MDP (Boyan/Littman) HSTS/RA (Jónsson et al.) Since then? Uncertain/dynamic Sensing Conformant Contingent Resources

17. Domain Assumptions Expressive logic Non-determinism Observation Goal model Plan utility Durative actions Complex concurrence Continuous time Metric quantities Branching factor Resource uncertainty Resource constraints Goal selection Safe planning Exogenous events STRIPS UCPOP CGP CNLP SENSp Buridan Weaver C-Buridan MDP PO-MDP S-MDP T-MDP F-MDP LPSAT Mars Rover Classical Bleeding edge Select contingencies Serialized goals?

18. Brain-teaser: Domain Spec State space S –Cartesian product of continuous and discrete axes (time, position, achievements, energy…) Initial state s i –Probability distribution Domain theory  –Concurrent, non-deterministic, uncertain  What else? (S, s i, , …)

19. Brain-teaser: Kalman Filters Curiously missing from the paper we read (?) 1983 Kalman filters paper: Voyager enters Jupiter orbit through a 30 second window after 11 years in space Hugh Durrant-Whyte’s robots Why not for the Mars Rover?

20. Context Summary Complex, exciting domain Pushes the planning envelope –Expression –Scaling  Where do we start?

21. Lecture Overview Context Innovation –Just-in-case planning –Incremental contingency planning Discussion

22. Just-In-Case Planning Motivated by domain characteristics –Metric quantities –Large branch factors Implications –Not plan, not policy –Expanded plan  What about concurrency?

23. Branch Heuristics Most probable failure point (scheduling) Highest utility branch point (planning)  What is the intrinsic difference?

24. When To Execute A Contingency?

25. Incremental Contingency Planning Algorithm Input: Domain description and master plan Output: Highest-utility branch point Algorithm: –Compute value, estimate resources during master plan –Approximate branch point utilities –Select highest-utility branch point –Solve w/ new initial, goal conditions –Repeat while necessary

26. Branch Utility Approximation … without constructing plan –Construct a plan graph –Back-propagate utility functions through plan graph, instead of regression searching –Compute branch point utilities throughout input plan

27. Back-Propagating Distributions Mausam: “Some parts of the paper are tersely written, which make it a little harder to understand. I got quite confused in the discussion of utility propagation. It would have been nicer had they given some theorems about the soundness of their method.” Well, me too

28. Back-Propagating Distributions A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’

29. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 5 Back-Propagating Distributions

30. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 5 Back-Propagating Distributions 5 15

31. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 5 5 15 5 25 Back-Propagating Distributions

32. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 5 5 15 1 2 5 25 Back-Propagating Distributions

33. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 5 5 15 1 2 r 1 2 t 1 2 5 25 Back-Propagating Distributions

34. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 5 25 Back-Propagating Distributions

35. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 1 5 t Back-Propagating Distributions 5 25

36. 15 A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 5 25 1 15 t + Back-Propagating Distributions 1 5 t 25 6

37. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 1 5 t 1 5 t 6 15 5 25 1 15 t + 25 6

38. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 1 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 r +

39. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 5 t

40. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 +

41. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 15 25

42. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 8 15 25

43. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 8 15 25 1 8

44. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 8 15 25 1 8 (CDE)

45. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 15 5 25 + 5 t 1 5 t 6 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 8 15 25 1 8 (CDE)

46. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 5 t 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 15 25 [(CDE) (ABDE)] [(DCE) (AB) (DABE)]

47. A C D B E (1, 5) (3, 3) (10, 15) (2, 2) p s q r t g g’ 1 1 5 5 5 15 1 2 r r 1 2 t 1 2 Back-Propagating Distributions 5 t 15 5 25 1 15 t + 25 6 1 5 + 1 1 8 + 15 25 1 8 (CDE, ABDE) 6 25 1 6 6 25 26 (DCE, AB, DABE) 5

48. Utility Estimation p s 1 8 (CDE, ABDE) 6 25 1 6 6 25 26 (DCE, AB, DABE) 5

49. Utility Estimation p s 1 8 (CDE, ABDE) 6 25 1 6 6 25 26 (DCE, AB, DABE) 5 1 6 6 25 (DCE, ABDE) MAX operator:

50. Utility Estimation p s 1 8 (CDE, ABDE) 6 25 1 6 6 25 26 (DCE, AB, DABE) 5 1 6 6 25 (DCE, ABDE) MAX operator: (Then combine w/Monte Carlo results)

51. Lecture Overview Context Innovation Discussion –Q: Evaluation? Inference?

52. Evaluation Optimal branch selection? (Greedy…)

53. Incremental Contingencies… Sometimes adding one contingency at a time is non-optimal  Examples?

54. Incremental Contingencies… 1.0 0 Rain Shine 0.5 0 1.0 Work Go climbing Exercise Sometimes adding one contingency at a time is non-optimal

55. Evaluation Optimal branch selection? What else?

56. Inference Where can we take these ideas? What can we add to them?

57. Inference Where can we take these ideas? What can we add to them? Optimal branch selection Optimistic branching Mutexes in plan graph Noisy/costly sensors

58. Review – Alex Yates “I don't quite understand why this planning problem is called contingent planning, since they assume full observability. That conflicts with the previous notion of contingent planning that we've seen. The reason for branches in these plans aren't because there is uncertainty in what state the planner is in, but because it's impossible to enumerate all of the different outcomes. Also, the paper never discusses probabilistic effects for state transitions other than the ones effecting the continuous attributes. I wasn't sure if this meant that all actions were deterministic except in their resource consumption, or if it meant that any attributes which might have probabilistic transitions would need to be treated as resources.”