1. Selection of Behavioral Parameters: Integration of Case-Based Reasoning with Learning Momentum Brian Lee, Maxim Likhachev, and Ronald C. Arkin Mobile Robot Laboratory Georgia Tech Atlanta, GA This research was funded under the DARPA MARS program.

2. Integrated Multi-layered Learning THE LEARNING CONTINUUM: Deliberative (pre-mission). Behavioral switching. Reactive (online adaptation)...... CBR Wizardry –Guide the operator Probabilistic Planning –Manage complexity for the operator RL for Behavioral Assemblage Selection –Learn what works for the robot CBR for Behavior Transitions –Adapt to situations the robot can recognize Learning Momentum –Vary robot parameters in real time

3. Motivation It’s hard to manually derive behavioral controller parameters. –The parameter space increases exponentially with the number of parameters. You don’t always have a priori knowledge of the environment. –Without prior knowledge, a user can’t confidently derive appropriate parameter values, so it becomes necessary for the robot to adapt on its own to what it finds. Obstacle densities and layout in the environment may be heterogeneous. –Parameters that work well for one type of environment may not work well with another type. A solution is to provide adaptability to the system while remaining fully reactive.

4. Context for Case-based Reasoning (CBR) Spatial and temporal features are used to select stored cases from a case library. Cases contain parameters for a behavior-based reactive controller. Selected parameters are adapted for the current situation. The controller is updated with new parameters that should be more appropriate to the current environment.

5. CBR Module Feature Identification Spatial Feature Matching Temporal Feature Matching Random Selection Process Case Library Case Switching Decision Case Adaptation Case Application Sensors

6. Context for Learning Momentum (LM) A crude form of reinforcement learning. –If the robot is doing well, try doing what it’s doing a little more, otherwise try something different. Behavior parameters are continually changed in response to progress and obstacles. Static rules for pre-defined situations are used to update behavior parameters. Different sets of rules for parameter changes can be used (ballooning versus squeezing).

7. LM Strategies Ballooning –Alter parameters so the robot reacts to obstacles at larger distances than normal to push it out of box canyon situations. Squeezing –Alter parameters so the robot reacts to obstacles only at shorter distances than normal so it can move between closely spaced obstacles. Example ballooning rule : if ( situation == NO_PROGRESS_WITH_OBSTACLES ) obstacle_sphere_of_influence += 0.5 meters else obstacle_sphere_of_influence -= 0.5 meters

8. LM Module Sensors Short Sensor History Situation Matching Behavioral Parameters Parameter Deltas Parameter Adaptation Old parameters Adapted parameters

9. Effects of CBR and LM When Used Separately Reported in ICRA 2001 Effects of CBR –Distances traversed were shorter –Time taken was shorter Effects of LM –Completion rates were much higher for dense obstacles –Completion times were higher than those for successful non-adaptive robots

10. Why Integrate? Want discontinuous switching + continuous searching in the parameter space. CBR is not continuous –Parameter changes are triggered by environment changes or case time-outs. –Case library is manually built to provide only ballpark solutions for different environment types. LM does not make large, discontinuous changes –LM may take a while to adapt to large environmental changes. LM cannot change strategies at run time –The LM strategies of ballooning and squeezing are tuned for different environments.

11. Currently Used Behaviors Move to Goal –Always returns a vector pointing toward the goal position. Avoid Obstacles –Returns a sum of weighted vectors pointing away from obstacles. Wander –Returns vectors pointing in random directions. Bias Move –Returns a vector biasing the robot’s movement in a certain direction (i.e. away from high obstacle densities), and is set by the CBR module. –Only used when CBR is present.

12. Adjustable Behavioral Parameters Move to goal vector gain Avoid obstacle vector gain Avoid obstacle sphere of influence –Radius around the robot inside of which obstacles reacted to Wander vector gain Wander persistence –The number of consecutive steps the wander vector points in the same direction Bias Move vector gain Bias Move X, Bias Move Y –These are the components of the vector returned by Bias Move

13. Integration Core Behavior-Based Controller Behavioral Parameters Sensors Actuators Base System

14. Integration Core Behavior-Based Controller Behavioral Parameters Sensors Actuators CBR Module Updated Parameters Addition of CBR Module

15. Integration Core Behavior-Based Controller Behavioral Parameters Sensors Actuators CBR Module LM Module Updated Deltas and Parameter Bounds Updated Parameters Addition of LM Module

16. Simulation Setup Heterogeneous Environments –varying obstacles density, order, and size –350 x 350 meters Homogeneous Environments –even obstacle distribution –random obstacle placement and size –two environments with 15% density and two environments with 20% density –150 x 150 meters

17. CBR-LM in Simulation

18. Simulation Results For a Heterogeneous Environment

19. Simulation Results For a Heterogeneous Environment

20. For a Homogeneous Environment Simulation Results

21. For a Homogeneous Environment Simulation Results

22. Simulation Observations Beneficial Attributes of CBR are Preserved. –We see quick, radical changes in behavior. –Time taken is about the same as CBR only. Beneficial Attributes of LM are not always apparent. –Results can probably be attributed to a well-tuned case library. –If the case library is good enough, LM should not be needed.

23. RWI ATRV-Jr robot Forward and rear LMS SICK laser scanners Odometry, compass, and gyroscope for localization Straight-line start to goal distance of about 46 meters Physical Robot Experiments Outdoor environment with trees and man-made obstacles CBR-LM, CBR, LM, and non-adaptive systems were compared The squeezing strategy was used in the LM-only experiments. Data was averaged over 10 runs per adaptation algorithm

24. Outdoor Run

25. Physical Experiments Results All valid runs were able to reach the goal. Both CBR and LM beat the non-adaptive system. The CBR-LM integrated system gave the best performance.

26. Difference From Simulation CBR-LM outperformed CBR on the physical robot more than in simulation. –The case library for the real robot may not have been as well tuned as the simulation library.

27. Conclusions A performance increase is not guaranteed. For a well-tuned case library, there may be little for LM to do. Integration of CBR and LM can result in a performance increase –observed up to 29% improvement in steps over CBR Benefits of LM are more likely to be apparent when the CBR case library is not well-tuned (which is likely to be the case for real robots.) LM could be used to dynamically update the case library with better sets of parameters.