Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin Mobile Robot Laboratory Georgia Tech This research was funded under the DARPA MARS program.

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin2 Motivation Constant parameterization of robotic behavior results in inefficient robot performance Manual selection of “right” parameters is difficult and tedious work

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin3 Motivation (cont’d) Use of Case-Based Reasoning (CBR) methodology – an automatic selection of optimal parameters at run-time (ICRA’01) –each case is a set of behavioral parameters indexed by environmental features “ front-obstructed ” case “ clear-to-goal ” case

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin4 Motivation for the Current Research The CBR module –improves robot performance (in simulations and on real robots) –avoids the manual configuration of behavioral parameters The CBR module still required the creation of a case library which –is dependent on a robot architecture –needs extensive experimentation to optimize cases –requires good understanding of how CBR works Solution: to extend the CBR module to learn –new cases from scratch or optimize existing cases –in a separate training process or during missions

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin5 Related Work Use of Case-Based Reasoning in the selection of behavioral parameters –ACBARR [Georgia Tech ’92], SINS [Georgia Tech ’93] –KINS [Chagas and Hallam] Automatic optimization of behavioral parameters –genetic programming (e.g., GA-ROBOT [Ram, et. al.]) –reinforcement learning (e.g., Learning Momentum [Lee, et. al.])

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin6 Behavioral Control and CBR Module CBR Module controls (case output parameters): Weights for each behavior BiasMove Vector Noise PersistenceObstacle Sphere

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin7 Case Indices: Environmental Features Spatial features: traversability vector split environment into K = 4 angular regions compute obstacle density within each region transform the density into traversability Temporal features: Short-term velocity towards the goal Long-term velocity towards the goal f 0 =0.92 f 1 =0.58 f 2 =1.0 f 3 =0.68 f 0 =0.02 f 1 =0.22 f 2 =0.63 f 3 =0.02 V spatial: f 0 =0.92 f 1 =0.58 f 2 =1.00 f 3 =0.68 V temporal ShortTerm : R s =1.0 LongTerm : R l =0.7 V temporal ShortTerm: R s =0.01 LongTerm: R l =1.0 V spatial: f 0 =0.02 f 1 =0.22 f 2 =0.63 f 3 =0.02

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin8 Overview of non-learning CBR Module Case switching Decision tree Case Adaptation current environment Feature Identification spatial & temporal feature vectors Spatial Features Vector Matching (1st stage of Case Selection) Temporal Features Vector Matching (2nd stage of Case Selection) set of spatially matching cases set of spatially and temporally matching cases Case Library all the cases in the library best matching or currently used case Case Application case ready for application case output parameters (behavioral assemblage parameters) Random Selection Process (3rd stage of Case Selection) best matching case

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin9 Making CBR Module to Learn Case output parameters ( behavioral assemblage parameters) Random Selection Biased by Case Success and Spatial and Temporal Similarities best matching or currently used case case ready for application last K cases new or existing best matching case current environment Feature Identification spatial & temporal feature vectors Spatial Features Vector Matching (1st stage of Case Selection) Temporal Features Vector Matching (2nd stage of Case Selection) set of spatially matching cases set of spatially and temporally matching cases Case switching Decision tree best matching case last K cases with adjusted performance history Case Library all the cases in the library Old Case Performance Evaluation New Case Creation (if necessary) Case Adaptation Case Application best matching or currently used case

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin10 Random selection of cases with the probability of the selection proportional to: –spatial similarity with the environment ( 1 st step) –temporal similarity with the environment (2 nd step) –weighted sum of the case past performance and spatial and temporal similarities (3 rd step) Extensive Exploration of Cases: Modified Case Selection Process set of spatially & temporally matching cases: {C 1,, C 4 } C1C1 spatial similarity P(selection) C2C2 C4C4 C3C3 C5C5 set of spatially matching cases: {C 1, C 2, C 4 } temporal similarity P(selection) C1C1 C4C4 C2C2 weighted sum of spatial and temporal similarities and case success P(selection) C1C1 C4C4 best matching case: C 1

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin11 Positive and Negative Reinforcement: Case Performance Evaluation Criteria for the evaluation of the case performance : the average velocity with which the robot approaches its goal during the application of the case –opportunities for intermediate case performance evaluations –may not always be the right criteria such cases exhibit no positive velocity towards the goal the evaluation of the performance is delayed by K (=2) cases –case_success (represents case performance) is: increased if the average velocity is increased or sustained high decreased otherwise

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin12 Maximization of Reinforcement: Case Adaptation Maximize case_success as a noisy function of case output parameters (behavioral assemblage parameters) –maintain the adaptation vector A(C) for each case C –if the last series of adaptations result in the increase of case_success then continue the adaptation: O(C) = O(C) + A(C) –otherwise switch the direction of the adaptation, add a random component and scale proportionally to case_success: A(C) = - ·A(C) + ·R O(C) = O(C) + A(C)

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin13 Maximization of Reinforcement: Case Adaptation (cont’d) Incorporate prior knowledge into the search: –fixed adaptation of the Noise_Gain and Noise_Persistence parameters based on the short- and long-term velocities of the robot Constrain the search: –limit Obstacle_Gain to be higher than the sum of the other schema gains (to avoid collisions)

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin14 The Growth of the Case Library: Case Creation Decision To avoid divergence a new case is created whenever: –case_success of the selected case is high and spatial and temporal similarities with the environment are low to moderate –case_success of the selected case is low to moderate and spatial and temporal similarities are low Limit the maximum size of the library (10 in this work) New case is initialized with: –the spatial and temporal features of the environment –the output parameter values of the selected case

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin15 Experimental Analysis: Example Learning CBR: first run (starting with an empty library)

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin16 Experimental Analysis: Example Learning CBR: a run after 54 training runs on various environments library of ten cases was learned 36 percent shorter travel distance A case of a “clear-to-goal” strategy is learned for such environments A case of a “squeezing” strategy is learned for such environments

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin17 Experiments: Statistical Results Simulation results (after 250 training runs for learning CBR system) Heterogeneous environmentHomogeneous environment Average number of steps Mission completion rate learning CBR CBR non-adaptive learning CBR CBR non-adaptive learning CBR CBR non-adaptive non-adapt. CBR learn

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin18 Real Robot Experiments: In Progress RWI ATRV-Jr Sensors: –SICK laser scanners in front and back –Compass –Gyroscope Experiments in progress, no statistical results yet

Learning Behavioral Parameterization Using Spatio-Temporal Case-Based Reasoning Maxim Likhachev, Michael Kaess, and Ronald C. Arkin19 Conclusions New and existing cases are learned and optimized during a training process or as part of mission executions Performance: – substantially better than that of a non-adaptive system –comparable to a non-learning CBR system Neither manual selection of behavioral parameters nor careful creation and optimization of case library is required from a user Future Work –real robot experiments –case “forgetting” component –integration with other adaptation & learning methods (e.g., Learning Momentum, RL for Behavioral Assemblage Selection)