Probabilistic approaches to reasoning and control: Towards autonomous interactive mobile robots Joelle Pineau Carnegie Mellon University TAMALE Seminar March 28, 2003

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Our vision of robotic-assisted health-care Moving things around Moving things around Enabling use of remote health services Enabling use of remote health services Supporting inter-personal communication Supporting inter-personal communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing information (TV, weather) Providing information (TV, weather) Management support of ADLs Management support of ADLs Reminding to eat, drink, & take meds Reminding to eat, drink, & take meds Providing physical assistance Providing physical assistance Linking the caregiver to resources Linking the caregiver to resources

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Introducing Pearl: A mobile robotic assistant for elderly people and nurses cameras sonars handle bars mobile base carrying tray LCD mouth touchscreen microphone & speakers laser

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots What are the challenges? Interaction with the environment: –navigating robustly –handling dynamic obstacles Interaction with individuals: –communicating by speech –providing cognitive reminders –interpreting and satisfying user requests

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview Cognitive supportNavigationCommunication High-level controller

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview Cognitive supportCommunication High-level controller Localization and map building (Burgard et al., 1999) People detection and tracking (Montemerlo et al., 2002) Navigation

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Navigation and people tracking

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview NavigationCommunication High-level controller Autominder system (Pollack et al., 2002) Cognitive support

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Speech recognition: Sphinx system (Ravishankar, 1996) Speech synthesis: Festival system (Black et al., 1999) System Overview Cognitive supportNavigation High-level controller Communication

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Speech recognition with Sphinx

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The role of the top-level controller Cognitive supportNavigationCommunication ACTION SELECTION - based on the trade-off between: - goals from different modules; - goals with varying costs / rewards; - reducing uncertainty versus accomplishing goals. High-level controller

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Types of uncertainty in robotics Cause #1: Non-deterministic effects of actions Cause #2: Partial and noisy sensor information Cause #3: Inaccurate model of the world and the user

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Robot control under uncertainty using Partially Observable Markov Decision Processes State User + Environment + Robot Action={ say-weather, update-appointment, clarify-query } Speech=“today” Belief State e.g. request-weather-today e.g. P(s t =weather-today)=0.5 P(s t =appointment-today )=0.5

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Existing applications of POMDPs –Maintenance scheduling »Puterman, 1994 –Robot navigation »Koenig & Simmons, 1995; Roy & Thrun, 1999 –Helicopter control »Bagnell & Schneider, 2001; Ng et al., 2002 –Dialogue modeling »Roy, Pineau & Thrun, 2000; Peak&Horvitz, 2000 –Preference elicitation »Boutilier, 2002

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Graphical Model Representation POMDP is n-tuple { S, A, , T, O, R }: What goes on: s t-1 stst a t-1 atat T(s,a,s’) = state-to-state transition probabilities O(s,a,o) = observation generation probabilities R(s,a) = Reward function S = state set A = action set  = observation set What we see: o t-1 Belief update:

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Understanding the belief state A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2 } P(s 1 ) 0 1

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Understanding the belief state A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2, s 3 } P(s 1 ) P(s 2 ) 0 1 1

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2, s 3, s 4 } Understanding the belief state P(s 1 ) P(s 2 ) P(s 3 )

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 P(s 1 ) V 0 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes P(s 1 ) V 1 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes P(s 1 ) V 1 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes P(s 1 ) V 2 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes P(s 1 ) V 2 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes ,348,907 P(s 1 ) V 2 (b) b

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Properties of exact planning Value function is always piecewise-linear convex Many hyper-planes can be pruned away P(s 1 ) V 2 (b) b |S|=2, |A|=3, |  |=2 Iteration# hyper-planes …

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Is pruning sufficient? |S|=20, |A|=6, |  |=8 Iteration# hyper-planes ????? … Not for this problem!

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Certainly not for this problem! Physiotherapy Patient room Robot home |S|=576, |A|=19, |O|=17 State Features: { RobotLocation, ReminderGoal, UserLocation, UserMotionGoal, UserStatus, UserSpeechGoal }

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The two curses of POMDP planning The curse of dimensionality: –the dimension of each hyper-plane = # of states The curse of history: –the number of hyper-planes grows exponentially with the planning horizon Complexity of POMDP value iteration: dimensionalityhistory

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Methods to solve POMDPs Complexity Performance QMDP MDP FIB Grid O(S 2 A) O(S 2 A O ) O(S 2 AB) T POMDP New methods? Objective: Find a policy,  (b), which maximizes reward.

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots New approach: A hierarchy of POMDPs Idea: Exploit domain knowledge to divide one POMDP into many smaller ones. Motivation: Smaller action sets help overcome the curse of history. Assumption: We are given POMDP M = {S,A, ,b,T,O,R} and hierarchy H Act ExamineHealth Navigate Move VerifyPulse ClarifyGoal NorthSouthEastWest VerifyMeds subtask abstract action primitive action

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds Step 1: Select the action set

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds Step 1: Select the action set Step 2: Minimize the state set

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h } Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PLAN  h PLAN  h PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h } Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Step 4: Plan task h

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Results on small dialogue domain |S|=1 2, |A|=20, |O|=3

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Achieving a flexible trade-off Planning time Reward QMDP FIB POMDP PolCA+ D2 PolCA+ D1

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+ in the Nursebot domain Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments.

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Sample scenario

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Comparing user performance

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The effects of confirmation actions

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Addressing the curse of dimensionality

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Ongoing work New POMDP approximation techniques. Parameter estimation for adaptation to user-specific speech patterns and preferences. Exploration of emotion and personality types using a new head. Addition of an arm for object manipulation. Addition of weight-bearing bars for assisted walking.

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Summary We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people. The top-level controller uses a hierarchical variant of POMDPs to select actions. –PolCA+ addresses both the curse of dimensionality and the curse of history. Lessons learned during our experiments: –Uncertainty is crucial when dealing with people –Probabilistic techniques are necessary to reason about uncertainty. –Real belief tracking and planning really matters!

Project information: Navigation software: Papers and more: Joint work with: Michael Montemerlo, Martha Pollack, Nicholas Roy, Sebastian Thrun

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The Nursebot project in its early days

Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Autominder System