1. 1 Introduction to Robotics 13. Deliberative and Hybrid Control.

2. 2 Deliberative Systems  Purely deliberative systems are considered the classical control architecture, since they were the first to be tried.  In AI, classical deliberative, planner-based architectures were used for reasoning about actions in various non-physical domains, such as chess.  As a result, the same architectures were applied to robotics as well. In the 1960’s: Shakey  In the late 1960's, the state-of-the-art in machine vision was used to process visual information on a robot called Shakey, the forerunner of many AI-inspired robotics projects.  Shakey used a classical planner as the underlying structure to decide what to do.  What is planning?

3. 3 Traditional Deliberative Planners Are often aligned with hierarchical control community within robotics. Hierarchical planning systems typically share a structured and clearly identifiable subdivision of functionality regarding to distinct program modules that communicate with each other in a predictable and predetermined manner. At a hierarchical planner’s highest level, the most global and least specific plan is formulated. At the lowest levels, rapid real-time response is required, but the planner is concerned only with its immediate surroundings and has lost the sight of the big picture.

4. 4 Strategic Global Planning Tactical Intermediate Planning Short-Term Local Planning Actuator Control Actions Global Knowledge Local World Model Intermediate Sensor Interpretations Sensing Real - Time Time Horizon Long - Term Spatial Scope Global Immediate Vicinity Hierarchical Planner World Model

5. 5 Planning as Search Planning is looking ahead, searching  The goal is a state.  The robot's entire state space is enumerated, and searched, from the current state to the goal state.  Different paths are tried until one is found that reaches the goal.  If the optimal path is desired, then all possible paths must be considered in order to find the best one.

6. 6 SPA = Planner-based  Planner-based (deliberative) architectures typically involve three generic sequential steps or functional modules: 1) sensing (S) 2) planning (P) 3) acting (A), executing the plan  Thus, they are called SPA architectures.  SPA has serious drawbacks. It takes a very (prohibitively) long time to search in a real robot's state space, as that space is typically very large. –Real robots may have collections of simple digital sensors (e.g., switches, IRs), a few more complex ones (e.g., cameras), or analog sensors (e.g., encoders, gauges, etc.) –=> "too much information" –=> Generating a plan is slow. Problem 1: Time Scale

7. 7 SPA = Planner-based It takes a lot of space (memory) to represent and manipulate the robot's state space representation. The representation must contain all information needed for planning. => Generating a plan can be large. Space is not nearly as much of a problem as time, in practice. Problem 2: Space Problem 3: Information The planner assumes that the representation of the state space is accurate and up-to-date. => The representation must be constantly updated and checked The more information, the better. => "too little information"

8. 8 SPA = Planner-based The resulting plan is only useful if : a) the environment does not change during the execution of a plan in a way that affects the plan. b) the representation was accurate enough to generate a correct plan. c) the robot's effectors are accurate enough to perfectly execute each step of the plan in order to make the next step possible Problem 4: Use of Plans Deliberation in Summary  In short, deliberative (SPA, planner-based) approaches:  require search and planning, which are slow  encourage open-loop plan execution, which is limiting and dangerous  Note that if planning were not slow (computationally expensive) then execution would not need to be open-loop, since re-planning could be done.

9. 9 Hierarchical Planners vs. BBS Hierarchical Planners Rely heavily on world models, Can readily integrate world knowledge, Have a broad perspective and scope. BB Control Systems afford modular development, Real-time robust performance within a changing world, Incremental growth are tightly coupled with arriving sensory data.

10. 10 Inventing Hybrid Control The basic idea is simple: we want the best of both worlds (if possible). The goal is to combine closed-loop and open-loop execution. That means to combine reactive and deliberative control. This implies combining the different time-scales and representations. This mix is called hybrid control. Hybrid robotic architectures believe that a union of deliberative and behavior-based approaches can potentially yield the best of both worlds.

11. 11 Organizing Hybrid Systems Planning and reaction can be tied: A: hierarchical integration - planning and reaction are involved with different activities, time scales Level N Level 2 Level 1 Level 0 More Reactive More Deliberative A Deliberation  Projection Planner Reactor B Behavioral Advice Configurations Parameters B: Planning to guide reaction - configure and set parameters for the reactive control system. C: coupled - concurrent activities PlannerReactor C

12. 12 Organizing Hybrid Systems In summary, a modern hybrid system typically consists of three components:  a reactive layer  a planner  a layer that puts the two together. => Hybrid architectures are often called three-layer architectures. It was observed that the emerging architectural design of choice is: – multi-layered hybrid comprising of * a top-down planning system and * a lower-level reactive system. – the interface (middle layer between the two components) design is a central issue in differentiating different hybrid architectures.

13. 13 The Magic Middle  The middle layer has a hard job: 1) compensate for the limitations of both the planner and the reactive system 2) reconcile their different time-scales. 3) deal with their different representations. 4) reconcile any contradictory commands between the two.  This is the challenge of hybrid systems => achieving the right compromise between the two ends.

14. 14 The middle layer services.  Some frequently useful planned decisions may need to be reused, so to avoid planning, an intermediate layer may cache and look those up. These can be: Reusing Plans  intermediate-level actions (ILAs): stored in contingency tables.  macro operators: plans compiled into more general operators for future use. Dynamic Re-planning  Reaction can influence planning.  Any "important" changes discovered by the low-level controller are passed back to the planner in a way that the planner can use to re-plan.  The planner is interrupted when even a partial answer is needed in real- time.  The reactive controller (and thus the robot) is stopped if it must wait for the planner to tell it where to go.

15. 15 Planner - Driven Reaction Planning can also influence reaction. Any "important" optimizations the planner discovers are passed down to the reactive controller. The planner’s suggestions are used if they are possible and safe. => Who has priority, planner or reactor? It depends, as we will see... The middle layer services. Types of “Reaction  Planning” Interaction  Selection: Planning is viewed as configuration.  Advising: Planning is viewed as advice giving.  Adaptation: Planning is viewed as adaptation of controller.  Postponing: Planning is viewed as a least commitment process.

16. 16 Universal Plans  Suppose for a given problem, all possible plans are generated for all possible situations in advance, and stored.  If for each situation a robot has a pre-existing optimal plan, it can react optimally, be reactive and optimal.  It has a universal plan (These are complete reactive mappings). Viability of Universal Plans  A system with a universal plan is reactive; the planning is done at compile-time, not at run-time.  Universal plans are not viable in most domains, because:  the world must be deterministic.  the world must not change.  the goals must not change.  the world is too complex (state space is too large).

17. 17 Situated Automata  A formal notion of finite state machines  whose inputs are connected to sensors and  whose outputs are connected to effectors are called situated automata.  Situated means existing in and interacting with a complex world, and automata is the formal name for FSMs (formally: finite state automata).  Situated automata are used to create reactive principled control systems.  Developed by Kaelbling & Rosenschein ‘91. We already mentioned situated automata in BBS. State Register Sensor Input Action The basic structure of Rosenschein’s situated automata design.

18. 18 Control with Situated Automata Situated automata can be constructed in two basic ways: oBy hand (i.e., the designer puts FSMs together), as in the Subsumption Architecture). oBy pre-compiling a complete plan (similar to Universal Plans, but reduced down to circuits of FSMs).  Rosenschein was able to design a compiler that generates finite state machines whose  internal states can be proved to correspond to certain logical propositions about the environment,  provided that the initial state and the correct laws of “physics” are given to the compiler.

19. 19 Situated Automata Rosenschein’s basic design relies on a theorem to the effect that any finite state machine can be implemented as a state register together with a feedforward circuit that updates the state based on the sensory inputs and the current state, and another circuit that calculates the output given the state register. Flakey, the robot capable in 1984 to navigate the halls of SRI was based on situated automata. This requires the use of a special programming language that implements the right semantics and compiles down into FSM circuitry, as Rex and Gapps.

20. 20 Pre - compiled Systems  A key advantage of pre-compiled systems is that domain knowledge, i.e., information that the designer has about the environment, the robot, and the task, can be embedded into the system in a principled way.  Then, the system is compiled into a reactive circuit, so the knowledge does not have to be reasoned about (or planned with) explicitly, in real-time. Advantage: Domain Knowledge Disadvantages  A key disadvantage of pre-compiled systems is that it quickly becomes prohibitively large to enumerate the state space of a real robot, and thus pre-compiling generally does not scale up to complex systems.  Another disadvantage is common to compiled or hard-wired systems: the result is not flexible in the presence of changing environments, tasks or goals.

21. 21 Summary: Hybrid Control Relative Strength. Deliberative planners:  Rely heavily on world models.  Can readily integrate world knowledge.  Have broader perspective and scope.  Reactive & behavior-based systems:  Afford modular development.  Provide real-time robust performance in dynamic world.  Provide for incremental growth.  Tightly coupled to incoming sensory data.

22. 22 Representative Hybrid Architectures.  Selection: Planning is viewed as configuration.  Advising: Planning is viewed as advice giving.  Adaptation: Planning is viewed as adaptation of controller.  Postponing: Planning is viewed as a least commitment process. Let’s discuss an example of each of these strategies: * AuRA for selection, * Atlantis for advising, * Planner - Reactor for adaptation, and * PRS as postponent.

23. 23 Example: AuRA R. Arkin (1986)  Planning is viewed as configuration.  Initial A* planner integrated with schema-based controller.  Provides modularity, flexibility, and adaptability. LearningUser Input Plan Recognition User Profile User Intentions Spatial LearningSpatial Goals Opportunism On-line Adaptation Teleautonomy Mission Alterations Mission Planner Spatial Reasoner Plan Sequencer RE PRE SEN TA TI ON Schema Controller Motor Perceptual ActuationSensing Hierarchical Component Reactive Component

24. 24 AuRA Planning and execution components: –a hierarchical system  mission planner, spatial reasoner, and plan sequencer. * traditional : highest level is a mission planner  establishes high - level (global) goals + constraints. – coupled with a reactive system  the schema controller. * mission planner  acts as an interface to human operator. o In the original implementation it was a rule based system o Now a finite state sequencer was implemented. * spatial reasoner  navigator, uses knowledge stored in LTM to construct a sequence of path legs that the robot must execute. * plan sequencer (pilot)  translates each path leg into a set of motor behaviors for execution. The collection of behaviors (schemas), specified by the sequencer, is sent to the reactive system for execution. – At this point, deliberation ends/stops, and reactive execution begins.

25. 25 A* Search Best-first search using f as the evaluation function and an admissible h function. Evaluation function f (n)= estimated cost of the cheapest solution through node n. f(n)= g(n)+ h(n) –g(n) gives the path cost from start to node n –h(n) is the estimated cost of the cheapest path from n to the goal. If h never overestimates the cost is called admissible heuristic. –Example: estimate the distance between to cities by rhe straight- line distance. A* –exhibit monotonicity (along any path from the root f never decreases). –optimally efficient (algorithm that extend search paths from the root).

26. 26 Example: Atlantis  E. Gat (1991)  Three layers: controller, sequencer, deliberator.  Asynchronous, heterogeneous: reactivity and deliberation  Planning as advice giving, not as command (not a decree).  Tested on NASA rovers. Control SensorsActuators Deliberative Sequencing Results ActivationStatus Invocation

27. 27 Atlantis Control Layer: –Reactive controller charged with managing collection of primitive activities. –Implemented in ALFA, a LISP based program language, similar to Rex, a circuit based language. Sequencing Layer: – Modeled after RAP (Reactive Action Packages, Firby 1989). – RAP is a situation-driven execution reactive method, in which the current situation provides an index into a set of actions regarding how to act in that environment. – Conditional sequencing occurs upon the completion of subtasks or detection of failure. – Notion of cognizant failure was introduced, referring to the robot’s ability to recognize on its own when it has not or cannot complete its task: it has knowledge about its failures. *Monitor (task specific) routines are added to determine if things are going as they should and then interrupt the system if cognizant failure occurs.

28. 28 Atlantis Deliberative Layer. –Deliberation occurs at the sequencing layer’ request. –Consists of traditional LISP based AI planning algorithms, specific to the task at hand. –The planner’s output is viewed only as advice to the sequencer layer: it is not necessarily followed or implemented. Design proceeds from bottom up: –low-level activities capable of being executed within the reactive controller are first constructed. –suitable sequences of primitive behaviors are then constructed. –Followed by deliberative methods that assist in the decision done by the sequencer.

29. 29 Example: Planner-Reactor D. Lyons (1992)  Continuous modification of a reactive control system (sub-optimal).  Planning is a form of reactor adaptation.  Adaptation is on-line rather than off-line deliberation.  Planning is used to remove performance errors when they occur.  Uses Robot Schema (RS) model.  Tested in both assembly cell and grasp planning. Goals Planner World Adaptation Perceptions Reactions REACTOR Action Sensing Perception

30. 30 Planner-Reactor A suboptimal reactor may be present at any time. The planner’s goal is to improve the performance of the reactor at all times (any-time planning). – Any-time planners provide approximate answers in a time critical manner: * at any point a plan is available for execution, and * the quality of the available plan increases over time. – Situations provide the framework for structuring sets of reactions. * can be defined hierarchically, as behavioral structures for use in the reactor and not specific robotic commands. * denote the state of the robotic agent is currently in (regarding a task).

31. 31 Planner-Reactor Adaptation: a) a reactor executes under a set of operating assumptions. b) if any assumptions are violated, the planner modifies the reactor’s control system to remove the violation. –Each assumption has a monitor associated with it during run time to ensure its validity. Reactor Performance with Monitoring Reactor Adapted by Planner and Assumptions Relaxed. Assumptions violation detected. Adapt Reactor Restore Initial Reactor. Violation assumptions Restored. Completed Initial Reactor Construction Adapt ReactorStart Execution Normal Performance

32. 32  Georgeff and A. Lansky (1987)  PRS = Procedural Reasoning System  Reactivity refers to postponement of the elaboration of plans until it is necessary:  a least commitment strategy.  Tested on SRI Flakey Example: PRS INTERPRETER ACTUATORSSENSORS MONITOR BELIEFS DESIRESPLANS INTENTIONS COMMAND GENERATOR OPERATOR INTRFACE

33. 33 PRS Plans are the primary mode of expressing action. –They are continuously determined in reaction to the current situation. –Previously formulated plans undergoing execution can be interrupted and abandoned at any time. Representations of the robot’s beliefs, desires, and intentions are all used to formulate a plan. The plan represents the robot’s desired behaviors instead the traditional AI planner’s output of goal states to be achieved. The interpreter drives system execution, handling the plan switching. A symbolic plan always drives the system. –it is not reactive in the normal sense of tight sensori-motor pair execution –it is reactive in the sense that perceived changing environmental conditions permit the robotic agent to alter its plans on the fly.

34. 34 SSS = Servo Subsumption Symbolic  J. Connell (1992)  3 layers: servo, subsumption, symbolic  World models are a convenience, not a necessity  Symbolic (deliberative) layer handles where-to-go-next decisions. Provides the ability to selectively turn behaviors on or off as well as provide parameters for those that require them.  Subsumption: where-to-go-now  Servo: making it go (continuous time, behaviors execute without any intervention from the symbolic layer.)  Tested on TJ More Examples SOMASS hybrid assembly system  C. Malcolm and T. Smithers (Edinburgh U. 1990)  cognitive/subcognitive components  planning as configuration

35. 35 More Examples Agent architecture  B. Hayes-Roth (Stanford)  physical and cognitive levels  functional boundary blurry  Multi-valued logic  Saffiotti, Konolige, Ruspini (SRI, 1995) Supervenience  L. Spector (1992, U. of Maryland)  Multiple levels of abstraction  Generic Robot Architecture  Noreils and Chatila (1995, France)  3 levels: planning, control system, functional

36. 36 Teleo-reactive agent architecture  Benson and N. Nilsson (1995, Stanford)  Planning yields TR operator tree More Examples  Reactive Deliberation  M. Sahota (1993, U. of British Columbia)  Robosoccer Theoagent  T. Mitchell (CMU, 1990)  Reacts when it can plans when it must  Emphasis on learning

37. 37  Dynamical Systems Approach  Schoner and Dose (1992)  Planning is selecting and parameterizing behavioral fields  Behaviors use vector summation More Examples  Integrated path planning and dynamic steering control  Krogh and C. Thorpe (1986, CMU)  Relaxation over grid-based model with potential fields controller  Planner generated way -points for controller  Many others (including several for UUVs)

38. 38 Hybrids Everywhere?  Hybrid systems are the most popular alternative for single-robot control.  Behavior-based systems are not used by quite as many researchers, but have more specialized niches (e.g., multi- robot systems) and more practical applications.