1. Probabilistic approaches to reasoning and control: Towards autonomous interactive mobile robots Joelle Pineau Carnegie Mellon University TAMALE Seminar March 28, 2003

2. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Our vision of robotic-assisted health-care Moving things around Moving things around Enabling use of remote health services Enabling use of remote health services Supporting inter-personal communication Supporting inter-personal communication Calling for help in emergencies Calling for help in emergencies Monitoring Rx adherence & safety Monitoring Rx adherence & safety Providing information (TV, weather) Providing information (TV, weather) Management support of ADLs Management support of ADLs Reminding to eat, drink, & take meds Reminding to eat, drink, & take meds Providing physical assistance Providing physical assistance Linking the caregiver to resources Linking the caregiver to resources

3. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Introducing Pearl: A mobile robotic assistant for elderly people and nurses cameras sonars handle bars mobile base carrying tray LCD mouth touchscreen microphone & speakers laser

4. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots What are the challenges? Interaction with the environment: –navigating robustly –handling dynamic obstacles Interaction with individuals: –communicating by speech –providing cognitive reminders –interpreting and satisfying user requests

5. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview Cognitive supportNavigationCommunication High-level controller

6. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview Cognitive supportCommunication High-level controller Localization and map building (Burgard et al., 1999) People detection and tracking (Montemerlo et al., 2002) Navigation

7. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Navigation and people tracking

8. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots System Overview NavigationCommunication High-level controller Autominder system (Pollack et al., 2002) Cognitive support

9. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Speech recognition: Sphinx system (Ravishankar, 1996) Speech synthesis: Festival system (Black et al., 1999) System Overview Cognitive supportNavigation High-level controller Communication

10. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Speech recognition with Sphinx

11. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The role of the top-level controller Cognitive supportNavigationCommunication ACTION SELECTION - based on the trade-off between: - goals from different modules; - goals with varying costs / rewards; - reducing uncertainty versus accomplishing goals. High-level controller

12. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Types of uncertainty in robotics Cause #1: Non-deterministic effects of actions Cause #2: Partial and noisy sensor information Cause #3: Inaccurate model of the world and the user

13. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Robot control under uncertainty using Partially Observable Markov Decision Processes State User + Environment + Robot Action={ say-weather, update-appointment, clarify-query } Speech=“today” Belief State e.g. request-weather-today e.g. P(s t =weather-today)=0.5 P(s t =appointment-today )=0.5

14. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Existing applications of POMDPs –Maintenance scheduling »Puterman, 1994 –Robot navigation »Koenig & Simmons, 1995; Roy & Thrun, 1999 –Helicopter control »Bagnell & Schneider, 2001; Ng et al., 2002 –Dialogue modeling »Roy, Pineau & Thrun, 2000; Peak&Horvitz, 2000 –Preference elicitation »Boutilier, 2002

15. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Graphical Model Representation POMDP is n-tuple { S, A, , T, O, R }: What goes on: s t-1 stst a t-1 atat T(s,a,s’) = state-to-state transition probabilities O(s,a,o) = observation generation probabilities R(s,a) = Reward function S = state set A = action set  = observation set What we see: o t-1 Belief update:

16. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Understanding the belief state A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2 } P(s 1 ) 0 1

17. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Understanding the belief state A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2, s 3 } P(s 1 ) P(s 2 ) 0 1 1

18. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots A belief is a probability distribution over states Where Dim(B) = |S|-1 –E.g. Let S={s 1, s 2, s 3, s 4 } Understanding the belief state P(s 1 ) P(s 2 ) 0 1 1 P(s 3 )

19. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 P(s 1 ) V 0 (b) b

20. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 1 3 P(s 1 ) V 1 (b) b

21. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 1 3 P(s 1 ) V 1 (b) b

22. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 1 3 2 27 P(s 1 ) V 2 (b) b

23. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 1 3 2 27 3 2187 P(s 1 ) V 2 (b) b

24. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Exact planning for POMDPs Simple problem: |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 0 1 1 3 2 27 3 2187 4 14,348,907 P(s 1 ) V 2 (b) b

25. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Properties of exact planning Value function is always piecewise-linear convex Many hyper-planes can be pruned away P(s 1 ) V 2 (b) b |S|=2, |A|=3, |  |=2 Iteration# hyper-planes 01 13 25 39 47 5 13 10 27 15 47 20 59 …

26. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Is pruning sufficient? |S|=20, |A|=6, |  |=8 Iteration# hyper-planes 01 15 2 213 3 ????? … Not for this problem!

27. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Certainly not for this problem! Physiotherapy Patient room Robot home |S|=576, |A|=19, |O|=17 State Features: { RobotLocation, ReminderGoal, UserLocation, UserMotionGoal, UserStatus, UserSpeechGoal }

28. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The two curses of POMDP planning The curse of dimensionality: –the dimension of each hyper-plane = # of states The curse of history: –the number of hyper-planes grows exponentially with the planning horizon Complexity of POMDP value iteration: dimensionalityhistory

29. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Methods to solve POMDPs Complexity Performance QMDP MDP FIB Grid O(S 2 A) O(S 2 A O ) O(S 2 AB) T POMDP New methods? Objective: Find a policy,  (b), which maximizes reward.

30. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots New approach: A hierarchy of POMDPs Idea: Exploit domain knowledge to divide one POMDP into many smaller ones. Motivation: Smaller action sets help overcome the curse of history. Assumption: We are given POMDP M = {S,A, ,b,T,O,R} and hierarchy H Act ExamineHealth Navigate Move VerifyPulse ClarifyGoal NorthSouthEastWest VerifyMeds subtask abstract action primitive action

31. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds Step 1: Select the action set

32. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds Step 1: Select the action set Step 2: Minimize the state set

33. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h } Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters

34. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+: Planning with a hierarchy of POMDPs Navigate Move ClarifyGoal SouthEast West North A Move = {N,S,E,W} S Move = {X,Y} STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds PLAN  h PLAN  h PARAMETERS {b h,T h,O h,R h } PARAMETERS {b h,T h,O h,R h } Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Step 4: Plan task h

35. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Results on small dialogue domain |S|=1 2, |A|=20, |O|=3

36. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Achieving a flexible trade-off Planning time Reward QMDP FIB POMDP PolCA+ D2 PolCA+ D1

37. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots PolCA+ in the Nursebot domain Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments.

38. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Sample scenario

39. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Comparing user performance 0.1 0.18

40. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The effects of confirmation actions

41. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Addressing the curse of dimensionality

42. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Ongoing work New POMDP approximation techniques. Parameter estimation for adaptation to user-specific speech patterns and preferences. Exploration of emotion and personality types using a new head. Addition of an arm for object manipulation. Addition of weight-bearing bars for assisted walking.

43. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Summary We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people. The top-level controller uses a hierarchical variant of POMDPs to select actions. –PolCA+ addresses both the curse of dimensionality and the curse of history. Lessons learned during our experiments: –Uncertainty is crucial when dealing with people –Probabilistic techniques are necessary to reason about uncertainty. –Real belief tracking and planning really matters!

44. Project information:www.cs.cmu.edu/~nursebot Navigation software:www.cs.cmu.edu/~carmen Papers and more:www.cs.cmu.edu/~jpineau Joint work with: Michael Montemerlo, Martha Pollack, Nicholas Roy, Sebastian Thrun

45. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots The Nursebot project in its early days

46. Joelle PineauProbabilistic approaches to reasoning and control for interactive mobile robots Autominder System