Learning to Navigate Through Crowded Environments Peter Henry 1, Christian Vollmer 2, Brian Ferris 1, Dieter Fox 1 Tuesday, May 4, University of Washington, Seattle, USA 2 Ilmenau University of Technology, Germany

The Goal Enable robot navigation within crowded environments

Motivation Robots should move naturally and predictably within crowded environments Robots should move naturally and predictably within crowded environments Move amongst people in a socially transparent way Move amongst people in a socially transparent way More efficient and safer motion More efficient and safer motion Humans trade off various factors Humans trade off various factors To move with the flow To move with the flow To avoid high density areas To avoid high density areas To walk on the left/right side To walk on the left/right side To reach the goal To reach the goal

Challenge Humans naturally balance between various factors Humans naturally balance between various factors Relatively easy to list factors Relatively easy to list factors But they can’t specify how they are making the tradeoff But they can’t specify how they are making the tradeoff Previous work typically uses heuristics and parameters are hand-tuned Previous work typically uses heuristics and parameters are hand-tuned Shortest path with collision avoidance [Burgard, et al., AI 1999] Shortest path with collision avoidance [Burgard, et al., AI 1999] Track and follow a single person [Kirby, et al., HRI 2007] Track and follow a single person [Kirby, et al., HRI 2007] Follow people moving in same direction [Mueller, et al., CogSys 2008] Follow people moving in same direction [Mueller, et al., CogSys 2008]

Contribution Learn how humans trade off various factors Learn how humans trade off various factors A framework for learning to navigate as humans do within crowded environments A framework for learning to navigate as humans do within crowded environments Extension of Maximum Entropy Inverse Reinforcement Learning [Ziebart, et al., AAAI 2008] to incorporate: Extension of Maximum Entropy Inverse Reinforcement Learning [Ziebart, et al., AAAI 2008] to incorporate: Limited locally observable area Limited locally observable area Dynamic crowd flow features Dynamic crowd flow features

Markov Decision Processes States States Actions Actions Rewards / Costs Rewards / Costs (Transition Probabilities) (Transition Probabilities) (Discount Factor) (Discount Factor) S0S0 S0S0 S1S1 S1S1 S2S2 S2S2 S3S3 S3S3 Goal

Navigating in a Crowd as an MDP States s i States s i In crowd scenario: Grid cell + orientation In crowd scenario: Grid cell + orientation Actions a i,j from s i to s j Actions a i,j from s i to s j In crowd scenario: Move to adjacent cell In crowd scenario: Move to adjacent cell Cost = An unknown linear combination of action features Cost = An unknown linear combination of action features Cost weights to be learned: θ Cost weights to be learned: θ Path: τ Path: τ Features: f τ Features: f τ

Inverse Reinforcement Learning Inverse Reinforcement Learning (IRL): Inverse Reinforcement Learning (IRL): Given: The MDP structure and a set of example paths Given: The MDP structure and a set of example paths Find: The reward function resulting in the same behavior Find: The reward function resulting in the same behavior (Also called “Inverse Optimal Control”) (Also called “Inverse Optimal Control”) Has been previously applied with success Has been previously applied with success Lane changing [Abbeel ICML 2004] Lane changing [Abbeel ICML 2004] Parking lot navigation [Abbeel IROS 2008] Parking lot navigation [Abbeel IROS 2008] Driving route choice and prediction [Ziebart AAAI 2008] Driving route choice and prediction [Ziebart AAAI 2008] Pedestrian route prediction [Ziebart IROS 2009] Pedestrian route prediction [Ziebart IROS 2009]

Exponential distribution over paths: Exponential distribution over paths: Learning: Learning: Gradient: Match observed and expected feature counts Gradient: Match observed and expected feature counts Maximum Entropy IRL

Locally Observable Features It is unrealistic to assume the agent has global knowledge of the crowd It is unrealistic to assume the agent has global knowledge of the crowd Contrast: Continuum Crowd Simulator explicitly finds a global solution for the entire crowd Contrast: Continuum Crowd Simulator explicitly finds a global solution for the entire crowd We do assume knowledge of the map itself We do assume knowledge of the map itself Training: Only provide flow features for small radius around current position Training: Only provide flow features for small radius around current position Assumes that these are the features available to the “expert” Assumes that these are the features available to the “expert” A single demonstration path becomes many small demonstrations of locally motivated paths A single demonstration path becomes many small demonstrations of locally motivated paths

Locally Observable Dynamic Features Crowd flow changes as the agent moves Crowd flow changes as the agent moves Locally observable dynamic feature training: Locally observable dynamic feature training: 1. Update flow features within local horizon 2. Compute feature gradient within grid 3. Perform stochastic update of weights 4. Take the next step of the observed path

Locally Observable Dynamic IRL The path probability decomposes into many short paths over the current features in the locally observable horizon The path probability decomposes into many short paths over the current features in the locally observable horizon Decompose over timesteps Local Horizon Features for actions within horizon at time t

Locally Observable Dynamic Gradient Uses current estimate of features at time t Uses current estimate of features at time t Computes gradient only within local horizon H Computes gradient only within local horizon H Observed features within H Expected features for actions within H

Map and Features Each grid cell encompasses 8 oriented states Each grid cell encompasses 8 oriented states Allows for flow features relative to orientation Allows for flow features relative to orientation Features Features Distance Distance Crowd flow speed and direction Crowd flow speed and direction Crowd density Crowd density (many others possible…) (many others possible…) Chosen as being reasonable to obtain from current sensors Chosen as being reasonable to obtain from current sensors

Crowd Simulator [Continuum Crowds, Treuille et al., SIGGRAPH 2006]

Simulator Environment

Experimental Setup We used ROS [Willow Garage] to integrate the crowd simulator and IRL learning and planner 1. Extract individual crowd traces and observable features 2. Learn feature weights with our IRL algorithm 3. Use weights for a simulated robot in test scenarios Planning is A* search Planning is A* search Re-planning occurs every grid cell with updated features Re-planning occurs every grid cell with updated features The robot is represented to the crowd simulator as just another person for realistic reactions from the crowd The robot is represented to the crowd simulator as just another person for realistic reactions from the crowd

Quantitative Results Measure similarity to “human” path Measure similarity to “human” path Shortest Path (baseline): Ignores crowd Shortest Path (baseline): Ignores crowd Learned Path: The path from our learned planner Learned Path: The path from our learned planner Mean / Maximum Difference: Over all path cells, difference to closest “human” path cell Mean / Maximum Difference: Over all path cells, difference to closest “human” path cell Shortest Path Learned Path Improvement Mean Difference % Maximum Difference % (Difference is significant at p=0.05 level)

Mall Scenario (Video)

Lane Formation (Video)

Future Work Train on real crowd data Train on real crowd data Overhead video + tracking? Overhead video + tracking? Wearable sensors to mimic robot sensor input? Wearable sensors to mimic robot sensor input? Implement on actual robot Implement on actual robot Is the method effective for raw sensor data? Is the method effective for raw sensor data? Which are the most useful features? Which are the most useful features? Pedestrian prediction Pedestrian prediction Compare / incorporate other recent work [Ziebart IROS 2009] Compare / incorporate other recent work [Ziebart IROS 2009]

Conclusion We have presented a framework for learning to imitate human behavior from example traces We have presented a framework for learning to imitate human behavior from example traces We learn weights that produce paths matching observed behavior from whatever features are made available We learn weights that produce paths matching observed behavior from whatever features are made available Our inverse reinforcement learning algorithm handles locally observable dynamic features Our inverse reinforcement learning algorithm handles locally observable dynamic features Resulting paths are more similar to observed human paths Resulting paths are more similar to observed human paths