Presentation is loading. Please wait.

Presentation is loading. Please wait.

Hierarchical POMDP Solutions

Published byInge Sutedja Modified over 5 years ago

Similar presentations

Presentation on theme: "Hierarchical POMDP Solutions"— Presentation transcript:

1 Hierarchical POMDP Solutions
Georgios Theocharous

2 Sequential Decision Making Under Uncertainty
Tests ACTIONS Treatments Symptoms OBSERVATIONS & REWARDS AGENT ENVIRONMENT HIDDEN STATES What is the optimal policy?

3 Manufacturing Processes (Mahadevan, Theocharous FLAIRS 98)
Buffer Machine Observations: Parts in buffers Throughput States: Machine internal state Reward: Reward for consuming Penalize for filling buffers Penalize for machine breakdown Actions: Produce Maintenance What is the optimal policy?

4 Foveated Active Vision (Minut)
States: Objects Observations: Local features Reward: Reward for finding object Actions: Where to saccade next What features to use What is the optimal policy?

5 Many More Partially Observable Problems
Assistive technologies Web searching, preference elicitation Sophisticated Computing Distributed file access, Network trouble-shooting Industrial Machine maintenance, manufacturing processes Social Education, medical diagnosis, health care policymaking Corporate Marketing, corporate policy ….

6 Overview Learning models of partially observable problems is far from a solved problem Computing policies for partially observable domains is intractable We Propose hierarchical solutions Learn models using less space and time Compute robust policies that cannot be computed by previous approaches

7 How? Spatial and Time Abstractions Reduce Uncertainty
Spatial abstraction MIT Temporal abstraction

8 Outline Sequential decision-making under uncertainty
A Hierarchical POMDP model for robot navigation Heuristic macro-action selection in H-POMDPs Near Optimal macro-action selection for arbitrary POMDPs Representing H-POMDPs as DBNs Current and Future directions

9 A Real System: Robot Navigation
0.1 0.8 S15 … 0.1 0.0 0.8 S1 S9 S5 Transition matrix for the Go-Forward action Observations Actions 0.0 0.15 0.7 WWWW … OWOW OOOO OBSERVATION Observation model for S1

10 Belief States (Probability Distributions over states)
True State Belief State

11 Belief States (Probability Distributions over states)
True State Belief State

12 Belief States (Probability Distributions over states)
True State Belief State

13 Learning POMDPs Given As and Zs compute Ts and Os
Estimate probability distribution over hidden states Count number of times a state was visited Update T and O and repeat. It is an Expectation Maximization algorithm: An iterative procedure for doing maximum likelihood parameter estimation over hidden state variables Converges to local maxima A1 A2 T(S1=i,A1=a,S2=j) S1 S2 S3 Z1 Z2 Z3 O(O2=z,S2=i,A1=a)

14 Planning in POMDPs Belief states constitute a sufficient statistic for making decisions (Markov property holds: Astrom 1965) Bellman equation: STATE ESTIMATOR POLICY(p) ENVIRONMENT OBSERVATION (z) ACTION (a) BELIEF STATE (b) AGENT Since we have an infinite state space, the problem becomes computationally intractable (PSPACE hard for finite horizon) (UNDECIDABLE for infinite horizon)

15 Our Solution: Spatial and Temporal Abstraction
Learning A hierarchical Baum-Welch algorithm, which is derived from the Baum-Welch algorithm for training HHMMs (with Rohanimanesh and Mahadevan, ICRA 2001) Structure learning from weak priors (with Mahadevan IROS 2002) Inference can be done in linear time by representing H-POMDPs as Dynamic Bayesian Networks (DBNs) (with Murphy and Kaelbling, ICRA 2004) Planning Heuristic macro-action selection (with Mahadevan, ICRA 2002) Near optimal macro-action selection (with Kaelbling, NIPS 2003) Structure Learning and Planning combined Dynamic POMDP abstractions (with Mannor and Kaelbling)

16 Outline A Hierarchical POMDP model for robot navigation
Sequential decision-making under uncertainty A Hierarchical POMDP model for robot navigation Heuristic macro-action selection in H-POMDPs Near Optimal macro-action selection for arbitrary POMDPs Representing H-POMDPs as DBNs Current and Future directions

17 Hierarchical POMDPs EAST WEST 1. Require less data to train than flat approaches Provide better state estimation than flat approaches Produces robust behavior Faster Planning

18 Hierarchical POMDPs ABSTRACT STATES + ACTIONS
(Fine, Singer, Tishby, MLJ 98)

19 Experimental Environments
600 states 1. 1200 states

20 The Robot Navigation Domain
The robot Pavlov in the real MSU environment The Nomad 200 simulator

21 Learning Feature Detectors (Mahadevan, Theocharous, Khaleeli: MLJ 98)
736 hand-labeled-grids 8-fold cross-validation Classification error (m=7.33, s=3.7)

22 Learning and Planning in H-POMDPs for Robot Navigation
INITIAL H-POMDP LEARNING HAND CODING COMPILATION TOPOLOGICAL MAP PLANNING ENVIRONMENT PLANNING PLANNING EXECUTION EM TRAINED H-POMDP NAVIGATION SYSTEM

23 Outline Heuristic macro-action selection in H-POMDPs
Sequential decision-making under uncertainty A Hierarchical POMDP model for robot navigation Heuristic macro-action selection in H-POMDPs Near Optimal macro-action selection for arbitrary POMDPs Representing H-POMDPs as DBNs Current and Future directions

24 Planning in H-POMDPs (Theocharous, Mahadevan: ICRA 2002)
Abstract actions Hierarchical MDP solutions (using the options framework [Sutton, Precup, Singh, AIJ]) Heuristic POMDP solutions MLS Primitive actions Beliefs: b(s) 0.35 0.3 0.2 0.1 0.05 4, 10 10, 5 23, 100 49, 20 100, 40 p(b)= go-west v(go-west) v(go-east)

25 Plan Execution

26 Plan Execution

27 Plan Execution

28 Plan Execution

29 Intuition Probability distribution at the higher level evolves more slowly The agent does not decide what the best macro-action to do every time step Long term actions result in robot localization

30 F-MLS Demo

31 H-MLS Demo

32 Hierarchical is More Successful
Unknown initial position Success % Environment Algorithm MLS QMDP MLS QMDP

33 Hierarchical Takes Less Time to Reach Goal
Unknown initial position ? Average Steps to Goal Environment Algorithm MLS QMDP MLS QMDP

34 Hierarchical Plans are Computed Faster
Planning Time Environment Goal 1 Goal 2 Goal 1 Goal 2 Algorithm

35 Outline Near Optimal macro-action selection for arbitrary POMDPs
Sequential decision-making under uncertainty A Hierarchical POMDP model for robot navigation Heuristic macro-action selection in H-POMDPs Near Optimal macro-action selection for arbitrary POMDPs Representing H-POMDPs as DBNs Current and Future directions

36 Near Optimal Macro-action Selection (Theocharous, Kaelbling NIPS 2003)
Usually agents don’t require the entire belief space Macro-actions can reduce belief space even more Tested in large scale robot navigation Only small part of the belief-space is required Learn approximate POMDP policies fast High success rate Better policies Does information gathering

37 Dynamic Grids Given a resolution, points are sampled dynamically from regular dicretizations, by simulating trajectories

38 The Algorithm True belief state True trajectory Resulting next true
Simulation trajectories from g of macro A (estimation of value at g) Value of b’’ is interpolated from it’s neighbors Nearest grid point to b

39 Experimental Setup

40 Fewer Number of States

41 Fewer Steps to Goal

42 More Successful

43 Information Gathering

44 Information Gathering (scaling up)

45 Dynamic POMDP Abstractions (Theocharous, Mannor, Kaelbling)
Entropy thresholds start goal Localization macros

46 Fewer Steps to Goal

47 Outline Representing H-POMDPs as DBNs
Sequential decision-making under uncertainty A Hierarchical POMDP model for robot navigation Heuristic macro-action selection in H-POMDPs Near Optimal macro-action selection for arbitrary POMDPs Representing H-POMDPs as DBNs Current and Future directions

48 Dynamic Bayesian Networks
STATE POMDP FACTORED DBN POMDP 0.08 0.01 0.7 0.05 # of parameters # of parameters

49 DBN Inference L 1

50 Representing H-POMDPs as Dynamic Bayesian Networks (Theocharous, Murphy, Kaelbling: ICRA 2004)
EAST WEST STATE H-POMDP FACTORED DBN H-POMDP

51 Representing H-POMDPs as Dynamic Bayesian Networks (Theocharous, Murphy, Kaelbling: ICRA 2004)
EAST WEST STATE H-POMDP FACTORED DBN H-POMDP

52 Representing H-POMDPs as Dynamic Bayesian Networks (Theocharous, Murphy, Kaelbling: ICRA 2004)
EAST WEST STATE H-POMDP FACTORED DBN H-POMDP

53 Representing H-POMDPs as Dynamic Bayesian Networks (Theocharous, Murphy, Kaelbling: ICRA 2004)
EAST WEST STATE H-POMDP FACTORED DBN H-POMDP

54 Representing H-POMDPs as Dynamic Bayesian Networks (Theocharous, Murphy, Kaelbling: ICRA 2004)
EAST WEST STATE H-POMDP FACTORED DBN H-POMDP

55 Complexity of Inference
FACTORED DBN H-POMDP STATE H-POMDP EAST WEST DBN H-POMDP STATE POMDP

56 Hierarchical Localizes better
Original Factored DBN tied H-POMDP Factored DBN H-POMDP DBN H-POMDP STATE POMDP Before training

57 Hierarchical Fits Data Better
Original Factored DBN tied H-POMDP Factored DBN H-POMDP DBN H-POMDP STATE POMDP Before training

58 Directions for Future Research
In the future we will explore structure learning Bayesian model selection approaches Methods for learning compositional hierarchies (recurrent nets, hierarchical sparse n-grams) Natural language acquisition methods Identifying isomorphic processes On–line learning Interactive Learning Application to real world problems

59 Major Contributions The H-POMDP model
Requires less training data Provides better state estimation Fast planning Macro-actions in POMDPS reduce uncertainty Information gathering Application of the algorithms to large scale Robot navigation Map Learning Planning and execution

Download ppt "Hierarchical POMDP Solutions"

Similar presentations

Dialogue Policy Optimisation

Dialogue Policy Optimisation
Markov Decision Process

Markov Decision Process
1 University of Southern California Keep the Adversary Guessing: Agent Security by Policy Randomization Praveen Paruchuri University of Southern California.

1 University of Southern California Keep the Adversary Guessing: Agent Security by Policy Randomization Praveen Paruchuri University of Southern California.
Partially Observable Markov Decision Process (POMDP)

Partially Observable Markov Decision Process (POMDP)
U NIVERSITY OF M ASSACHUSETTS, A MHERST Department of Computer Science Solving POMDPs Using Quadratically Constrained Linear Programs Christopher Amato.

U NIVERSITY OF M ASSACHUSETTS, A MHERST Department of Computer Science Solving POMDPs Using Quadratically Constrained Linear Programs Christopher Amato.
SA-1 Probabilistic Robotics Planning and Control: Partially Observable Markov Decision Processes.

SA-1 Probabilistic Robotics Planning and Control: Partially Observable Markov Decision Processes.
CSE-573 Artificial Intelligence Partially-Observable MDPS (POMDPs)

CSE-573 Artificial Intelligence Partially-Observable MDPS (POMDPs)
Decision Theoretic Planning

Decision Theoretic Planning
LCSLCS 18 September 2002DARPA MARS PI Meeting Intelligent Adaptive Mobile Robots Georgios Theocharous MIT AI Laboratory with Terran Lane and Leslie Pack.

LCSLCS 18 September 2002DARPA MARS PI Meeting Intelligent Adaptive Mobile Robots Georgios Theocharous MIT AI Laboratory with Terran Lane and Leslie Pack.
Ai in game programming it university of copenhagen Statistical Learning Methods Marco Loog.

Ai in game programming it university of copenhagen Statistical Learning Methods Marco Loog.
What Are Partially Observable Markov Decision Processes and Why Might You Care? Bob Wall CS 536.

What Are Partially Observable Markov Decision Processes and Why Might You Care? Bob Wall CS 536.
An Introduction to Markov Decision Processes Sarah Hickmott

An Introduction to Markov Decision Processes Sarah Hickmott
Partially Observable Markov Decision Process By Nezih Ergin Özkucur.

Partially Observable Markov Decision Process By Nezih Ergin Özkucur.
COSC 878 Seminar on Large Scale Statistical Machine Learning 1.

COSC 878 Seminar on Large Scale Statistical Machine Learning 1.
Planning under Uncertainty

Planning under Uncertainty
GS 540 week 6. HMM basics Given a sequence, and state parameters: – Each possible path through the states has a certain probability of emitting the sequence.

GS 540 week 6. HMM basics Given a sequence, and state parameters: – Each possible path through the states has a certain probability of emitting the sequence.
Concurrent Markov Decision Processes Mausam, Daniel S. Weld University of Washington Seattle.

Concurrent Markov Decision Processes Mausam, Daniel S. Weld University of Washington Seattle.
KI2 - 10 Kunstmatige Intelligentie / RuG Markov Decision Processes AIMA, Chapter 17.

KI Kunstmatige Intelligentie / RuG Markov Decision Processes AIMA, Chapter 17.
Machine LearningRL1 Reinforcement Learning in Partially Observable Environments Michael L. Littman.

Machine LearningRL1 Reinforcement Learning in Partially Observable Environments Michael L. Littman.
Bayesian Reinforcement Learning with Gaussian Processes Huanren Zhang Electrical and Computer Engineering Purdue University.

Bayesian Reinforcement Learning with Gaussian Processes Huanren Zhang Electrical and Computer Engineering Purdue University.

Similar presentations

Ads by Google