Classical Situation hellheaven World deterministic State observable

MDP-Style Planning hellheaven World stochastic State observable [Koditschek 87, Barto et al. 89] Policy Universal Plan Navigation function

Stochastic, Partially Observable sign hell?heaven? [Sondik 72] [Littman/Cassandra/Kaelbling 97]

Stochastic, Partially Observable sign hellheaven sign heavenhell

Stochastic, Partially Observable sign heavenhell sign ?? hellheaven start 50%

Robot Planning Frameworks Classical AI/robot planning State/actionsdiscrete & continuous Stateobservable Environmentdeterministic PlansSequences of actions CompletenessYes OptimalityRarely State space size Huge, often continuous, 6 dimensions Computationa l Complexity varies

MDP-Style Planning hellheaven World stochastic State observable [Koditschek 87, Barto et al. 89] Policy Universal Plan Navigation function

Markov Decision Process (discrete) s2s2 s3s3 s4s4 s5s5 s1s r=  10 r=  0  r=0 r=1 r=0 [Bellman 57] [Howard 60] [Sutton/Barto 98]

Value Iteration Value function of policy  Bellman equation for optimal value function Value iteration: recursively estimating value function Greedy policy: [Bellman 57] [Howard 60] [Sutton/Barto 98]

Value Iteration for Motion Planning (assumes knowledge of robot’s location)

Continuous Environments From: A Moore & C.G. Atkeson “The Parti-Game Algorithm for Variable Resolution Reinforcement Learning in Continuous State spaces,” Machine Learning 1995

Approximate Cell Decomposition [Latombe 91] From: A Moore & C.G. Atkeson “The Parti-Game Algorithm for Variable Resolution Reinforcement Learning in Continuous State spaces,” Machine Learning 1995

Parti-Game [Moore 96] From: A Moore & C.G. Atkeson “The Parti-Game Algorithm for Variable Resolution Reinforcement Learning in Continuous State spaces,” Machine Learning 1995

Robot Planning Frameworks Classical AI/robot planning Value Iteration in MDPs Parti-Game State/actionsdiscrete & continuous discretecontinuous Stateobservable Environmentdeterministicstochastic PlansSequences of actions policy CompletenessYes OptimalityRarelyYesNo State space size Huge, often continuous, 6 dimensions millionsn/a Computationa l Complexity variesquadraticn/a

Stochastic, Partially Observable sign ?? start sign heavenhell sign hellheaven 50% sign ?? start

A Quiz  -dim continuous* stochastic 1-dim continuous stochastic actions# statessize belief space?sensors 3: s 1, s 2, s 3 deterministic3perfect 3: s 1, s 2, s 3 stochastic3perfect : s 1, s 2, s 3, s 12, s 13, s 23, s 123 deterministic3 abstract states deterministic3stochastic 2-dim continuous*: p ( S=s 1 ), p ( S=s 2 ) stochastic3none 2-dim continuous*: p ( S=s 1 ), p ( S=s 2 ) *) countable, but for all practical purposes  -dim continuous* deterministic 1-dim continuous stochastic aargh!stochastic  -dim continuous stochastic

Introduction to POMDPs (1 of 3) 80  100 ba  0 ba  40 s2s2 s1s1 action a action b p(s1)p(s1) [Sondik 72, Littman, Kaelbling, Cassandra ‘97] s2s2 s1s1  action aaction b

Introduction to POMDPs (2 of 3) 80  100 ba  0 ba  40 s2s2 s1s1 80% c 20% p(s1)p(s1) s2s2 s1’s1’ s1s1 s2’s2’ p(s 1 ’) p(s1)p(s1) s2s2 s1s1  [Sondik 72, Littman, Kaelbling, Cassandra ‘97]

Introduction to POMDPs (3 of 3) 80  100 ba  0 ba  40 s2s2 s1s1 80% c 20% p(s1)p(s1) s2s2 s1s1  p(s1)p(s1) s2s2 s1s1 s1s1 s2s2 p(s 1 ’|A) B A 50% 30% 70% B A p(s 1 ’|B) [Sondik 72, Littman, Kaelbling, Cassandra ‘97]

Value Iteration in POMDPs Value function of policy  Bellman equation for optimal value function Value iteration: recursively estimating value function Greedy policy: Substitute b for s

Missing Terms: Belief Space Expected reward: Next state density: Bayes filters! (Dirac distribution)

Value Iteration in Belief Space.... next belief state b’ observation o.... belief state b max Q(b’, a) next state s’, reward r’state s Q(b, a) value function

Why is This So Complex? State Space Planning (no state uncertainty) Belief Space Planning (full state uncertainties) ?

Augmented MDPs: [Roy et al, 98/99] conventional state space uncertainty (entropy)

Path Planning with Augmented MDPs information gainConventional plannerProbabilistic Planner [Roy et al, 98/99]

Robot Planning Frameworks Classical AI/robot planning Value Iteration in MDPs Parti-GamePOMDPAugmented MDP State/actionsdiscrete & continuous discretecontinuousdiscrete Stateobservable partially observable Environmentdeterministicstochastic PlansSequences of actions policy CompletenessYes No OptimalityRarelyYesNoYesNo State space size Huge, often continuous, 6 dimensions millionsn/adozensthousands Computationa l Complexity variesquadraticn/aexponentialO(N 4 )