Presentation on theme: "Partially Observable Markov Decision Process (POMDP)"— Presentation transcript:
1Partially Observable Markov Decision Process (POMDP) by Ye FangDepartment of Computer ScienceRice UniversityDouglas Aberdeen, National ICT Australia 2003
2Overview Recap POMDP and the exact solution Heuristic methods Solution Heuristics for Exact methodsGrid methodsFactored belief statesSimulationMethods for continuous state and action spaceSolution
3Learning with a Model The agent knows the model , , Observation/action history:Belief state1/31/31/3Goal1/21/21
4Learning with a Model Update beliefs: Long-term value of a belief stateDefine:
5Complexity of Exact Methods Exponential number of state variables:Updating believe state is expensive.Believe-state monitoring is hard.Exponential number of belief states:PSPACE-Hard for simplified finite-horizon POMDP.NP-Hard to find a policy.
6How to make POMDP feasible? Almost impossible to find a exact solution for POMDP modelWhere does the complexity of exact solution come from?Infinite believe statesUpdating believe states and their value functionsIntroduce heuristic methods for exact methods
7How to make POMDP feasible? Why can Heuristics work?Simplify the representation of value function by assuming the system is an MDP.Replace the believe state b with real world stateThen, we have finite many states
8Heuristic for Exact Methods The intuition behind these heuristics is to assume the system as an MDP by finding an approximate projection from belief state to world state.In exact solution,(Instead of using all possible world state in a belief state b, and their corresponding possible transitions, heuristics are used for exact solution.)Different approximation method needs 1) belief state state in MDP 2) update scheme 3) appropriate value function
9Heuristic for Exact Methods Goal:Find an good approximation of projection from belief state to world state.Find a good policy for each believe state.
10Heuristic for Exact Methods MSL(most likely state)Voting HeuristicsQMDP HeuristicHeuristic using the uncertainty of belief state
11MLS HeuristicWe can assume the system is in the most likely world state(MLS) i at time t. The policy executed at that state is the transition with largest Q-value at state i.Therefore, the state in the model is world state which is the same as MDP model. It ignores the agents confusion about the current world state it in. Then we can use the MDP method learned before.
12MLS HeuristicThis method neglects all possible world states but the MSL state at belief state b.EX: Given optimal action in a world with three states and two actions,u(s0) = a0, u(s1) = a0, u(s2) = a1b = [0.3, 0.3, 0.4]The probability of not being at state s2 is 0.4 and the best action a0
13Voting HeuristicThe voting heuristic assigns a probability distribution over the actions instead of over the states.Given:The action for each belief state:(Probability at state j) * (best action at that state)
14Voting HeuristicEX: Given optimal action in a world with three states and two actions,u(s0) = a0, u(s1) = a0, u(s2) = a1b = [0.3, 0.3, 0.4],V(s0, a0)=5, V(s0, a1)=4,V(s1,a0)=5, V(s1,a1)=4,V(s2,a0)=0, V(s2,a1)= 10.At state s2,expectedR(a0) =3, expectedR(a1) =6.4The most likely best action is a0 (0.6). A1 maybe better given the reward
15Voting HeuristicThis method does not take the reward of an action into account.Introduce QMDP, which emphasize the Q- function of the optimal policy rather than the policy itself.
16QMDP HeuristicQMDP only takes into account the belief state at first step.What if this action does not do much to disambiguate the state, this method cannot improve the action over time.Find the action with largest Q-value.This method takes account the belief state for one step and assume the state is entirely known.
17Shortcomings of the Heuristics What if the belief states is close to uniform?Ex: a robot trying to reach the other end of a futureless desert. By observation, it has almost same belief of it is at everywhere.What if there is a lot of uncertainty in the information state?Consider the uncertainty when taking action
18Formal measurement of Uncertainty Entropy is the measure of a probability distribution that reflects how spiked or spread out the probability mass is, essentially capturing the amount of uncertainty with a single number.f(.) is a discrete probability mass function.
19Two Objectives When choosing actions, we want to: To take actions that will yield the highest rewards.To reduce the entropy of information state.
20Weighted Entropy Control Intuition: relate the entropy to the rewards to give some rough measure of the value of information.
21Weighted Entropy Control When the entropy is near 1, it means the environment is totally unobservable.When the entropy near 0, it means the model is almost a MDP.
22Weighted Entropy Control Define VL to be the lower bound for POMDP value function.The value at each belief state is:The control strategy will be:Vco means completely observable. VL is the lower bound for any state in POMDP.
23Other Heuristics for POMDP Grid methodFactored belief methodSimulation
24Grid methodInstead of compose the world state from belief state, it picks the real world states.How to choose the set of real world states (a interesting region of each belief state)?How to interpolate?
25How to choose grid points? Simulation to find useful pointsAdding points where the value differed a lot though with similar observation.
26How to interpolate? Maintain the convex nature of the value function: f(g,u) is the value grid point g under action u.Example: nearest neighbors, linear interpolations, etc.MSL heuristic can be thought of as a grid method with points at the belief-simplex corners and a simple 1 NN interpolationA value function over a continuous belief space can be approximated by a finite set of grid points G and an interpolation-extrapolation rule that estimates the value of an arbiturary point of the belief space by relying on the points of the grid and their associate values.
27Factored Belief StateIntuition: learn the dependency of state variablesEx: at time t: the state of raining is trueat time t+1: the state of “ground is dry” is not very likely to be true.Two-slice temporal Bayes network showing dependencies between state variables over successive time steps.
28Factored Belief StateWe can use a subset of state variables to construct a Bayes network(BN).Belief-state projection can be searched to find a suitable BN for a specific problem（belief monitoring）. It is a learning of adjusting the belief network parametrized by ϕ.Factored linear value function: weighted linear combinations of polynomial basis functions.Search procedure is provided to determine belief-state projection with bounded errors.
29Simulation and Belief State Concentrate learning effort on the states that are most likely to be encountered.In terms of Q-learning, we can simulate a path in POMDP and perform iteration of the value function on the monitored current belief states.Not good for POMDP with more than hundreds of states.The full DP update is not efficient.
30Simulation and Belief State Learn Q-function that generalize to all belief statesArtificial neural network can also be used to approximate the value function of the full belief states.
31Continuous State and Action Spaces Sampled belief states.Use particle filters to update the belief state.The value function is approximated using the average of k nearest neighbors.Filtering refers to determine the distribution of some variable at a certain time given the observation up to that time.Approximate continuous belief states are formed from the sampling using Gaussian Kernels.1.Infinite memory is needed to represent continuous state space2. Balance running time and accuracy with the number of samples n.3. Hyperline methods cannot be used because the state space is continuous.
32Policy search Vs Value search It is simpler to just determine how to act instead of the value of acting.Approximate value function method usually produce deterministic policies.The heuristic methods are approximated projections from belief states to world states.It is better to introduce randomness in policy.
33Policy search Vs Value search Policy search can be very difficult.Value search can be better for small POMDP .Value search imposes Bellman equations as constrains.Hybrid
34Policy search Policy search can be implemented by policy iteration. Step1: Evaluate current policyStep2: Improve policy
35Recap Different Heuristics Projecting a belief state to a world state Evaluating the values for belief statesFinding good policy