1. Behavior Coordination Mechanisms – State-of-the- Art Paper by: Paolo Pirjanian (USC) Presented by: Chris Martin

2. Subject of Paper  Overview of different Action Selection Mechanisms (ASMs) that “solve” the Action Selection Problem (ASP)  A major issue in behavior-based control (BBC) systems is the formulation of effective mechanisms for coordination of the behaviors’ activities into strategies for rational and coherent behavior

3. Research Areas Tackling ASP  Ethology  Artificial Life (AL)  Virtual Reality (VR)  Software Agents  Robotics (Physical Agents)

4. What is the ASP?  Deals with the agent selecting “the most appropriate” or “the most relevant” next action at a particular moment in a particular situation  Choosing a “good enough” or “satisficing” behavior

5. “Good enough” behaviors  According to Mae’s, to produce a “good enough” behavior, need the following requirements  Goal orientedness  Situatedness  Persistence  Planning  Robustness  reactivity

6. “Good enough” behaviors (2)  Tyrell adds the following to Mae’s list:  Deal with all types of subproblems  Compromise actions  Opportunism (in contrast to persistence)

7. Actions and Agents  Based on the Webster dictionary, two relevant definitions of action for two parts of the system:  1. “the condition of acting or moving, as opposed to rest” = motor movements  2. “habitual deeds; hence conduct; behavior” = activation of a behavior  Agents have 2 roles in an agency: actions and action selection mechanisms (ASMs)  With respect to its subordinates, agent is ASM  With respect to its superior, agent is action

8. Types of ASMs  Two types: Arbitration and Command Fusion (CF) ASMs  One characteristic defines the division: the number of behaviors handled  Arbitrary ASMs allow one or one set of behaviors to take control at any one time  Command Fusion ASMs allow multiple behaviors to contribute to final control of the robot

9. Arbitration ASMs – 3 Types  1. Priority-based  Action selection consists of higher level behaviors overriding the output of lower level behaviors  2. State-based -- 4 different examples  A. Discrete Event Systems  Behavior selection done using state transitions  Detection of certain events shift the system to a new state and new behavior

10. Arbitration ASMs (2) State-based cont.  B. Temporal Sequencing  At each state a behavior is activated and perceptual triggers cause state transition  C. Bayesian Decision Analysis  Choose action that maximizes expected utility of agent (cost/benefit)  D. Reinforcement Learning  Two kinds:  Hierarchical Q-learning: problem broken into smaller problems each learned separately through Q-learning  W-learning: each module/behavior recommends an action with some weight and action with highest weight selected and executed

11. Arbitration ASMs (3)  3. Winner-Take-All -- one example  Activation Networks  A set of behaviors reduce the difference between the system’s present state and goal state  By exchange of activation energy, the behaviors compete and cooperate to select an action  The system emergently chooses and performs the next step of the sequence

12. Command Fusion ASMs  Combine recommendations from multiple behaviors to form a control action that represents a consensus  Proceeds in 3 steps  Action recommendations  Behavior aggregation  Action selection

13. CF ASMs – 4 Types  1. Voting -- 3 examples  A. DAMN  Behavior votes for or against set of actions  Each behavior assigned weight by mode manager  ‘Voter’ selects ‘best’ action  Experiments show DAMN superior to other ASMs

14. CF ASMs – 4 Types (2) Voting cont.  B. SAMBA  Primitive behaviors produce reactions in form of primitive action maps  Behavior outputs generated from 4 primitive action maps  Command arbiter combines maps by multiplying each by a gain and adding the results  C. Action Voting  Each behavior votes for an action and votes against undesirable actions  The votes are summed and action with highest value is selected

15. CF ASMs – 4 Types (3)  2. Fuzzy Command Fusion -- 2 examples  A. Fuzzy/Multivalued Logic Approach  Control schema  Behavior schema  Planners  B. Fuzzy DAMN  Outputs of behaviors are cast as discrete membership functions over the set of possible actions  Weighted sum replaced with fuzzy inferencing methods  Max-vote replaced with defuzzifcation techniques

16. CF ASMs – 4 Types (4)  3. Multiple Objective Behavior Coordination (MOBC)  Each behavior calculates an objective function over a set of permissible actions  Action that maximizes the objective function is the best “satisficing” objective  Action selection is comprised of generating and then selecting a set of “satisficing” solutions among a set of efficient solutions known as Pareto-optimal solutions

17. CF ASMs – 4 Types (5)  4. Superposition Based Command Fusion -- 2 examples  A. Potential Fields  Approach to motion planning  Robot moves under the influence of an artificial potential field produced by an attractive force at the goal configuration and repulsive forces at obstacles  B. Motor Schemas  Generates a vector which encodes the direction and intensity of motor action (calculates a potential field for current configuration of the robot)  These vectors are added to generate a combined motor action  This is then multiplied by a gain value  Used on AuRA