8/9/20151 DARPA-MARS Kickoff Adaptive Intelligent Mobile Robots Leslie Pack Kaelbling Artificial Intelligence Laboratory MIT

8/9/20152 DARPA-MARS Kickoff Two projects Making reinforcement learning work on real robots Solving huge problems  dynamic problem reformulation  explicit uncertainty management

8/9/20153 DARPA-MARS Kickoff Reinforcement learning given a connection to the environment find a behavior that maximizes long-run reinforcement Reinf Environment Action Observation

8/9/20154 DARPA-MARS Kickoff Why reinforcement learning? Unknown or changing environments Easier for human to provide reinforcement function than whole behavior

8/9/20155 DARPA-MARS Kickoff Q-Learning Learn to choose actions because of their long-term consequences Given experience: Given a state s, take the action a that maximizes

8/9/20156 DARPA-MARS Kickoff Does it Work? Yes and no. Successes in simulated domains: backgammon, elevator scheduling Successes in manufacturing and juggling with strong constraints No strong successes in more general online robotic learning

8/9/20157 DARPA-MARS Kickoff Why is RL on robots hard? Need fast, robust supervised learning Continuous input and action spaces Q-learning slow to propagate values Need strong exploration bias

8/9/20158 DARPA-MARS Kickoff Making RL on robots easier Need fast, robust supervised learning  locally weighted regression Continuous input and action spaces  search and caching of optimal action Q-learning slow to propagate values  model-based acceleration Need strong exploration bias  start with human-supplied policy

8/9/20159 DARPA-MARS Kickoff Human Policy Start with human-provided policy Environment action state

8/9/ DARPA-MARS Kickoff Do supervised policy learning Human Policy Train Environment Policy action state sa

8/9/ DARPA-MARS Kickoff When the policy is learned, let it drive Human Policy Train Environment Policy action state

8/9/ DARPA-MARS Kickoff Q-Learning Train Environment Q-Value RL Policy action state D s a v

8/9/ DARPA-MARS Kickoff Acting based on Q values Q-Value max index a1a1 a2a2 anan a s

8/9/ DARPA-MARS Kickoff Letting the Q-learner drive Train Environment RL Policy action state D Q-Value s a v max

8/9/ DARPA-MARS Kickoff Train policy with max Q values Train Environment RL Policy action state D Q-Value s a v max s’

8/9/ DARPA-MARS Kickoff Add model learning Train Model Environment Q-Value RL Policy action state D s ss a a r v

8/9/ DARPA-MARS Kickoff Train Model Environment Q-Value RL Policy action state D s a v When model is good, train Q with it s’ a’

8/9/ DARPA-MARS Kickoff Other forms of human knowledge hard safety constraints on action choices partial models or constraints on models value estimates or value orderings on states

8/9/ DARPA-MARS Kickoff We will have succeeded if It takes less human effort and total development time to provide prior knowledge run and tune the learning algorithm than to write and debug the program without learning

8/9/ DARPA-MARS Kickoff Test domain Indoor mobile-robot navigation and delivery tasks quick adaptation to new buildings quick adaptation to sensor change or failure quick incorporation of human information

8/9/ DARPA-MARS Kickoff Solving huge problems We have lots of good techniques for small-to-medium sized problems reinforcement learning probabilistic planning Bayesian inference Rather than scale them to tackle huge problems directly, formulate right-sized problems on the fly

8/9/ DARPA-MARS Kickoff Dynamic problem reformulation working memory perceptionaction

8/9/ DARPA-MARS Kickoff Reformulation strategy Dynamically swap variables in and out of working memory constant sized problem always tractable adapt to changing situations, goals, etc Given more time pressure, decrease problem size Given less time pressure, increase problem size

8/9/ DARPA-MARS Kickoff Multiple-resolution plans Fine view of near-term high-probability events Coarse view of distant low-probability events

8/9/ DARPA-MARS Kickoff Information gathering Explicit models of the robot’s uncertainty allow information gathering actions drive to top of hill for better view open a door to see what’s inside ask a human for guidance Where is the supply depot? Two miles up this road

8/9/ DARPA-MARS Kickoff Explicit uncertainty modeling POMDP work gives us theoretical understanding Derive practical solutions from learning explicit memorization policies approximating optimal control

8/9/ DARPA-MARS Kickoff Huge-domain experiments Simulation of very complex task environment large number of buildings and other geographical structures concurrent, competing tasks such as  surveillance  supply delivery  self-preservation other agents from whom information can be gathered