1. Multi Robot Systems Course Bar Ilan University Mor Vered

2. Motivation In a multi-robot environment path planning or collision avoidance is an important problem. Multi-robot systems researchers have been investigating distributed coordination methods for improving spatial coordination in teams – a.k.a collision avoidance. Such methods adapt the coordination method to the dynamic changes in density of the robots. Basically, the goal is to help avoid collision from any static obstacles or other dynamic objects, such as moving robots.

3. Motivation Spatial conflicts can cause the team’s productivity to drop with the addition of robots. We will see that this phenomenon is impacted by the coordination methods used by the team-members, as different coordination methods yield radically different productivity results.

4. Heuristic Coordination Approaches Noise method - If I am on a given trajectory that is danger of colliding with another agents', add random noise to my direction vector. ”Behavior Based Formation Control for Multi-robot Teams”. Balch, Tucker and Arkin, Ronald C

5. Heuristic Coordination Approaches Aggression method - Describes a controller which breaks deadlocks in favour of the most ‘aggressive’ robot. The robots compete and only one gains access to the resource. When robots come to close to each other, each of the robots chooses an aggression level (randomly); the robot with the lower level concedes its position, preventing a collision. Later showed that it might be best to choose aggression level proportional to the robot's task. For every cycle a robot found itself within 2 radii of a teammate, it selected either an aggressive or timid behavior, with probability of 0.5. If the robot selected to become timid, it backed away for 100 cycles (10 simulated seconds). Otherwise it proceeded forward, executing the aggressive behavior. As robots chose to continue being “aggressive” or to become “timid” every cycle, the probability that two robots would collide in this implementation was near zero. ”Go ahead, Make my day: Robot Conflict Resolution by Aggressive Competition”. Vaughan, Richard and St{\o}y, Kasper and Sukhatme, Gaurav and Mataric, Maja

6. Heuristic Coordination Approaches Repel method - When sensing a possible collision on course the Repel group backtracked for 500 cycles (50 seconds) but mutually repelled using a direction of 180 degrees away from the closest robot.

7. Heuristic Coordination Approaches TimeRand method - This method contained no repulsion methods to directly avoid collision but in case of a collision could take care of the immobile robot. When robots sensed that they did not significantly move for 100 cycles (10 seconds), they proceeded to move with a random walk for 150 cycles (15 seconds).

8. Heuristic Coordination Approaches TimeRepel method - This method also contained no repulsion methods to directly avoid collision but only reacted after the fact to collisions. Once these robots did not move for 150 cycles (15 seconds), they then moved backwards for 50 cycles (5 seconds). All methods taken from “ A Study of Mechanisms for Improving Robotic Group Performance “Rosenfeld, Avi and Kaminka, Gal A and Kraus, Sarit and Shehory, Onn

9. Heuristic Coordination Approaches Homework Come up with your own heuristic coordination approach by next week’s lesson and e-mail it to me.

10. Selecting the Best Approach Assuming we have come up with several solutions. Another problem arising from that is how to select the best coordination method. As we stated before spatial conflicts can cause the team’s productivity to drop with the addition of robots. This phenomenon is impacted by the coordination methods used by the team-members, as different coordination methods yield radically different productivity results. No one collision avoidance method is best in all domain and group size settings. The effectiveness of coordination methods in a given context is not known in advance.

11. Selecting the Best Approach

12. CCC - Combined Coordination Cost Method How to select the best coordination method CCC -quantifies the production resources spent on coordination conflicts ; quantify the cost of group interactions. Multi-attribute cost measure to quantify resources such as time and fuel each group member spends in coordination behaviors during task execution. Facilitates comparison between different group methods. “ A Study of Mechanisms for Improving Robotic Group Performance “Rosenfeld, Avi and Kaminka, Gal A and Kraus, Sarit and Shehory, Onn

13. CCC - Combined Coordination Cost Method Contended that if robots dynamically reduce their CCC, group productivity will be improved. To demonstrate this, they created robotic groups which dynamically adapt their coordination techniques based on each robot’s CCC estimate. Problem with this method – it ignores the gains accumulated from long periods of no coordination needs. The next method tries to fix that “ A Study of Mechanisms for Improving Robotic Group Performance “Rosenfeld, Avi and Kaminka, Gal A and Kraus, Sarit and Shehory, Onn

14. Adaptive Multi-Robot Coordination: A Game-Theoretic Perspective Used a reinforcement-learning approach to coordination algorithm selection. Proved it both on experiments (foraging) and empirically by mathematical equations. “ Adaptive Multi-Robot Coordination: A Game-Theoretic Perspective “ Gal A. Kaminka, Dan Erusalimchik and Sarit Kraus

15. Problem Definition The normal routine of a robot's operation is to carry out its primary task until interrupted by a conflict with another robot which must be resolved by a coordination algorithm. This is called a conflict event. The event triggers a coordination algorithm to handle the conflict. Once it successfully finishes, the robots involved go back to their primary task.

16. Problem Definition Defined several kinds of tasks : 1) Loose-coordination between the robots ( only occasional need for spatial or temporal coordination ). For example; multi-robot foraging. 2) Cooperative task – the robots seek to maximize group utility. For example; exploration. 3) Timed tasks – the task is bound in time. For example; exploration – completely explore a new area as quickly as possible, or patrolling.

17. Problem Definition Divided time into : 1) Active interval – in which the robot was actively investing resources in coordination. 2) Passive interval – in which the robot no longer requires investing in coordination. 3) The robot has a nonempty set of coordination algorithms to select from. The choice of c coordination algorithm selection effects the duration of the active and passive intervals.

18. Reinforcement Learning Computes how an agent ought to take actions in an environment so as to maximize some definition of cumulative reward. Basic reinforcement model consists of : 1) A set of environment states - S. 2) A set of actions - A. 3) Rules of transitioning between states. 4) Rules that determine the immediate reward of a transition. 5) Rules that describe what the agent observes.

19. Reinforcement Learning The trick is - how to define the reward. They introduced a reward function EI effectiveness index, that reduces time and resources spent coordinating and maximizes the time between conflicts that require coordination. Took into consideration time spent on coordination and time not spent on coordination.

20. Reinforcement Learning Instead of reward functions – used Q-Learning. ( a variant of reinforcement learning ). A technique that works by learning an action-value function that gives the expected utility of taking a given action in a given state and following a fixed policy thereafter. Takes into account an accumulative value - of all results taken up until now. “Q-learning”, Watkins, Christopher JCH and Dayan, Peter

21. General reward The general reward is comprised of : 1) The total cost of coordination – cost of internal resources such as battery life and fuel. 2) Time spent on coordinating. 3) Frequency of coordinating.