1. Jaroslaw Kutylowski 1 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität An Overview of Robot Behavior Control with insight into AI-based and algorithm-based approaches

2. Jaroslaw Kutylowski 2 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Agenda What is this talk going to cover? What is behavior? Behavior control -Basic control strategies -Advantages and disadvantages of these strategies -Hybrid strategies Behavior-based control Deliberation-based control Hybrid strategies Final remarks

3. Jaroslaw Kutylowski 3 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität What is behavior? every robot has a goal how to accomplish this goal? good readings from sensors and good control of movement do not suffice  we need proper decision-making

4. Jaroslaw Kutylowski 4 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior control High-level behavioral algorithms Low-level basic algorithms Physics What is behavior? movement control sensor control what to do on sensor input how to coordinate with teammates navigation exploration etc.

5. Jaroslaw Kutylowski 5 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität What is behavior? Behavior control Input: sensory data history of behavior information from teammates information about opponent Output: what to do next? – where to go – where to look – what to send to teammates Behavior control High-level behavioral algorithms Low-level basic algorithms Physics  We will look at different methods for decision making and following these decisions

6. Jaroslaw Kutylowski 6 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität What is behavior? High-level behavioral algorithms Most prominent problems navigation to a point, with obstacles exploring unknown terrain task allocation Behavior control High-level behavioral algorithms Low-level basic algorithms Physics  Many research done in this area  We will review some of the results

7. Jaroslaw Kutylowski 7 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität What is behavior? Low-level basic algorithms Behavior control High-level behavioral algorithms Low-level basic algorithms Physics Typical problems how to walk how to read sensor input how to evaluate visual sensory input  These are problems which we won’t discuss

8. Jaroslaw Kutylowski 8 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior control basic strategies Two main approaches to behavior control: Behavior-based control (reactive) –“world is the world’s best model” –simple actions as reactions to environment –complex behaviors emerge from simple ones –stateless –no communication between teammates, only observation –inspired biologically –emerging from the AI community Deliberation-based control –careful planning of actions –maintaining state and synchronizing it with the environment –complex behaviors planned in advance –communication with teammates –prediction of opponent’s behavior –emerging from the algorithmic community

9. Jaroslaw Kutylowski 9 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior control discussion on behavior-based control Advantages: Simple controller, suitable for architectures with low performance Easy implementation leading to rapid development Easy to test and debug Should adapt well to changing environmental conditions Fast reaction time, well suited for dynamically changing situations e.g. (e.g. robot-soccer) Provable low-level properties (collision-avoidance etc.) Disadvantages: Emergent behavior is impossible to predict No provable properties about emergent behavior Not suitable very well to less dynamic situations where goals are achieved in a long term (e.g. UGV navigation)

10. Jaroslaw Kutylowski 10 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior control discussion on deliberation-based control Advantages: Possibility to plan in advance for long term behavior Complex behaviors are precisely defined and provable Can take advantage of communication with mates Possibility of learning and thus predicting the moves of the opponent Disadvantages: High hardware requirements (computationally intensive algorithms) Possibility of loss of synchronization between internal state and environment Problems hard to solve and implement Can react too slow in very dynamic situations (e.g. robot-soccer)

11. Jaroslaw Kutylowski 11 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior control hybrid strategies The two basic strategies can be combined to hybrid ones Basic behavior controlled by behavior-based strategies (low-level) Deliberation-based methods define a high-level strategy Advantages of both strategies can be combined

12. Jaroslaw Kutylowski 12 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control agenda Typical methods: –Simple state machines and how to define them –Potential fields method –Formation control

13. Jaroslaw Kutylowski 13 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control simple state-machines Most popular method of behavior control in dynamical systems Used by GermanTeam 2002 and later Sample definition: Goalie Goalie-before -kickoff Goalie -playing Return-to goal Position-inside -goal stand go-to-point kick go-to-ball

14. Jaroslaw Kutylowski 14 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control XABSL for defining behavior rules Instead of defining behavioral aspects of software in plain code, usage of meta-languages Software engineering defines UML, Petri-nets, high-level scripting etc. for modeling of behavior XABSL (extensible agent behavior specification language) is defined by the German team Syntax based on XML Defines a state-automaton Language constructs typical for a structural language (if, conditions) Constructs for easy operation on the state-automaton (transitions) Basic behaviors like “go-to-ball” defined in low-level language

15. Jaroslaw Kutylowski 15 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control XABSL for defining behavior rules XABSL is transformed into Intermediate Code, which is executed on the AIBO by a low-level virtual machine AIBO behaves according to the definitions given in XABSL, acting as a state-automaton

16. Jaroslaw Kutylowski 16 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control decisions in state machines Sometimes decisions between certain behavior options must be made These are based on evaluating utility functions for possible options These utility functions can be influenced by non-determinism

17. Jaroslaw Kutylowski 17 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control Potential fields method Objects either attract or repulse the robot These forces constitute the potential field Forces in the field are summed according to physical rules, so that one obtains the resultant force The resultant force indicates the movement direction of the robot, optionally the force strength determines the movement speed Advantages: Smooth movement Elegant solution, very easy to describe

18. Jaroslaw Kutylowski 18 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control Potential fields method Ball Robot Opponent Repulsive force induced by opponent robot Attractive induced by ball Calculated resultant force, direction of movement

19. Jaroslaw Kutylowski 19 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control Potential fields method – problems Local minima Ball Robot Ball Robot

20. Jaroslaw Kutylowski 20 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control Potential fields method – problems No passage between close objects Ball Robot

21. Jaroslaw Kutylowski 21 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control Potential fields method – problems Oscillations Ball Robot Ball Robot

22. Jaroslaw Kutylowski 22 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control formation control Formation control is important for terrain traversal, soccer … Four robots travel in a predefined formation ColumnLineDiamond Robots compute position and positions of others Own formation position is calculated basing on leader position neighbor position unit-center position

23. Jaroslaw Kutylowski 23 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Behavior-based control formation control Robot tries to maintain formation, by staying inside of the dead zone Inside of dead zone  no additional formation maintaining performed Inside of controlled zone  speed vector into dead zone linearly dependent on distance from dead zone If obstacles occur, the avoidance gains priority  As soon obstacle is surrounded, the robot tries to get into formation  Can be realized using potential field, with dead zone attracting and obstacles repelling Dead zone Controlled zone

24. Jaroslaw Kutylowski 24 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based control agenda Typical methods: –Case based reasoning –Hidden Markov Models Algorithmic approaches –task allocation –navigation to a point, with obstacles

25. Jaroslaw Kutylowski 25 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based control case based reasoning During soccer play similar situations can occur quite often Case based reasoning allows a player to store the behavior of opponents and use it when a similar situation occurs once again Sample: Goal Robot with ball Opponent

26. Jaroslaw Kutylowski 26 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based control case based reasoning Advantages: Opponent behavior can be analyzed and player can adapt to its strategies Disadvantages: If opponent uses similar techniques, than the two CBR instances fight against each other, returning improper forecasts No provable results Highly memory and computational intensive Learning process is needed  May be advantageous against simple opponents, but has no provable properties and fails against “intelligent” opponents

27. Jaroslaw Kutylowski 27 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based control Hidden Markov Model method As in CBR, the goal is to predict the behavior of the opponent The HMM method: assume that the opponent has a state machine and uses a set of common behaviors, like go-to-ball, intercept-ball … for each behavior we define a model, which is a state machine with probabilities for transition from state to state for every possible observation the model contains a probability that it occurs in a certain state we cannot directly observe the state of the opponent  so we instantiate HMM behavior models and look whether their execution matches the observations thus we obtain probabilities that the opponent is in a certain state of a certain behavior

28. Jaroslaw Kutylowski 28 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based control Hidden Markov Model method Observations are Distance of robot to ball Robot ball manipulation Distance of robot to goal … The most interesting question is about the value of  Knowing the probability, we can derive some information about future behavior of opponent

29. Jaroslaw Kutylowski 29 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation Task allocation is important when coordination of robots is needed With robot soccer task allocation is mostly reduced to role assignment (first forward, supporting forward, defender) Lot of research on multiprocessor task scheduling and similar assignments, which can be often translated to multi-robot scenarios Models utilized for robot task scheduling: Robots are heterogeneous Tasks require specific skills Tasks appear online Communication is expensive and thus must be minimized Computation power is sparse

30. Jaroslaw Kutylowski 30 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation We look for efficient, online and distributed approximations for task- allocation Taxonomy of task allocation problems: ST-SR – single-task robots, single-robot tasks ST-MR – single-task robots, multi-robot tasks MT-SR – multi-task robots, single-robot tasks MT-MR – multi-task robots, multi-robot tasks uncommon

31. Jaroslaw Kutylowski 31 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation – ST-SR setting Model Set M of workers, s. t. |M| = m Set N of jobs, s. t. |N| = n jobs, with a weight w j for each job skill rating, which defines the fitness of a worker for a job: We want to find such an assignment of workers to jobs, s. t. a sum of the combination of utility function and job weight is maximized  Centralized ILP solvable by Hungarian Method gives runtime of O(mn 2 ), but needs about n 2 messages to be exchanged  Distributed auction mechanisms achieve the same task with only O(n) messages

32. Jaroslaw Kutylowski 32 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation – online ST-SR setting The previous model assumed an offline-setting In reality the online version is much more likely to occur BLE algorithm: If any robot is unassigned, find the robot-task pair with highest utility and weight Assign this robot to this task Go on  This greedy strategy is 2-competitive to the optimal offline algorithm

33. Jaroslaw Kutylowski 33 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation – ST-MR setting Also known as coalition formation Now each job might require a specific skill which is possessed only by some robots Transforming the coalition formation problem to SPP: Let E be a set of all tasks and robots Let F be a family of all robot-task pairs u(f), where f is a set from F, is the utility for robot-task pair SPP Finite set E Family F of subsets of E Utility function u: F→R +  Find a maximum-utility family X of elements in F, s.t. X is a partition of E

34. Jaroslaw Kutylowski 34 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods task allocation – ST-MR setting SPP is NP-complete But there are heuristics and approximations which give good practical results Unfortunately these methods do not have a guaranteed approximation ratio, they only report how far the constructed solution is from the optimum for a particular problem instance

35. Jaroslaw Kutylowski 35 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods navigation to a point Model: The robot should get from a source position to a target position traveling the smallest possible distance There are obstacles with unknown position and size Different assumptions about the abilities of sensors may be made Visual sensors Touch sensors Important measures: Ratio of distance obtained by algorithm and the optimum Distance taking into account the sizes of obstacles

36. Jaroslaw Kutylowski 36 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods navigation – D* algorithm Model Finite undirected graph G(V,E), most often a grid Edge blocking The edge blocking is unknown to the algorithm The blocked edges cannot be traverse Blocked edges can be detected only at adjacent vertices D* algorithm Assume that all the unknown terrain contains no blocked edges Find shortest path Try to go on this path On blocked edges  update terrain map, calculate new path EB 

37. Jaroslaw Kutylowski 37 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods navigation – D* algorithm Sample edge-blocked graph S E

38. Jaroslaw Kutylowski 38 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Deliberation-based methods navigation – D* algorithm Performance of D* Lower bound on competitive ratio Upper bound on competitive ratio Lower bound construction

39. Jaroslaw Kutylowski 39 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Hybrid strategies Two layers of execution The lower runs with reactive behavior-based methods The upper runs with deliberative methods The lower layer assures fast reactions, obstacle avoidance etc. and can basically function without the help of the upper layer The upper layer provides additional support to the lower layer, by analyzing the situation (e.g. case based reasoning) and giving “hints” to the lower layer The hints are only supportive for the working of the behavior-based methods, i.e. they can be (partially) ignored The hint can be modeled as a slight influence on a utility function of executing an option

40. Jaroslaw Kutylowski 40 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Final remarks What you should remember Two basic strategies for behavior control No clear indication which one is best Many research in both areas, with deliberative having more strict proofs and behavior-based having more practical realization  Today’s results aren’t great  Practical realizations more often use simpler methods – there is a gap between the theoretical results and their implementation

41. Jaroslaw Kutylowski 41 HEINZ NIXDORF INSTITUT Universität Paderborn Algorithmen und Komplexität Jaroslaw Kutylowski Heinz Nixdorf Institut & Institut für Informatik Universität Paderborn Fürstenallee 11 33102 Paderborn Tel.: 0 52 51/60 64 68 Fax: 0 52 51/62 64 82 E-Mail: jarekk@upb.de http://www.upb.de/cs/ag-madh Thank you for your attention!