Models for Control and Verification Ian Mitchell Department of Computer Science The University of British Columbia research supported by National Science and Engineering Research Council of Canada

March 2008Ian Mitchell (UBC Computer Science)2 Outline Classes of models –Well-posed models –Difference Equations –Nonlinear Ordinary Differential Equations –Syntax vs semantics –Visualization Optimal Control –Objective and value functions Verification: Reachability –Forward & backward, tubes & sets, maximal & minimal

March 2008Ian Mitchell (UBC Computer Science)3 Control & Verification Require Modeling Dynamic systems change with time We wish to reason about that change –Control: We seek to guide the evolution to achieve a desired objective –Verification: We seek to confirm the evolution will achieve a desired objective Control and verification require prediction of future evolution –Prediction is achieved by mathematical models System is described by state and time

March 2008Ian Mitchell (UBC Computer Science)4 Discrete vs Continuous Discrete variable –Drawn from a countable domain, typically finite –Often no useful metric other than the discrete metric –Often no consistent ordering –Examples: names of students in this room, rooms in this building, natural numbers Continuous variable –Drawn from an uncountable domain, but may be bounded –Usually has a continuous metric –Often no consistent ordering –Examples: Real numbers [ 0, 1 ], R d, SO(3)

March 2008Ian Mitchell (UBC Computer Science)5 Classes of Models for Dynamic Systems Discrete time and state Continuous time / discrete state –Discrete event systems Discrete time / continuous state Continuous time and state Markovian assumption –All information relevant to future evolution is captured in the state variable Models are deterministic –Future evolution completely determined by initial conditions Not the only classes of models

March 2008Ian Mitchell (UBC Computer Science)6 Well-Posed Models Mathematical models may not behave nicely –May describe impossible evolutions –May not be easy to apply formal reasoning We want to forbid such (eg ignore) models Common desirable traits –There exists a solution for all (or some) time –The solution is unique –The solution depends continuously on the data (initial conditions, dynamics)

March 2008Ian Mitchell (UBC Computer Science)7 Difference Equations Existence: for all t and x, f(t, x)  ; Uniqueness: for all t and x, | f(t, x) | = 1 Continuous dependence on the data: for all t, x, y there exists constant such that –Only makes sense if state space has a continuous metric –Sufficient but not necessary –Might also want to handle mistakes in f

March 2008Ian Mitchell (UBC Computer Science)8 Lipschitz Continuity Called “Lipschitz continuity” with respect to x (or y) Constant is the “Lipschitz constant” Relationship with continuity and differentiability? Continuity Differentiability with bounded derivative Lipschitz continuity no relation

March 2008Ian Mitchell (UBC Computer Science)9 Lipschitz Continuous Functions Which of these functions is Lipschitz continuous? A D B C

March 2008Ian Mitchell (UBC Computer Science)10 Ordinary Differential Equations (ODEs) What about second order ODE? –Newton’s second law: force = (mass)(acceleration) Need to reformulate into first order form –Define new variable z(t) 2 R 2*d May also be useful to remove dependence on t –Define new variable y(t) 2 R d+1 –Called “autonomous system” in mathematics

March 2008Ian Mitchell (UBC Computer Science)11 Standard First Order Form Q45: What is the equivalent first order form of the following ODE for the motion of a pendulum? l m

March 2008Ian Mitchell (UBC Computer Science)12 Standard First Order Form Q46: What is the equivalent first order form of the following high order ordinary differential equation?

March 2008Ian Mitchell (UBC Computer Science)13 Well-Posed ODEs Consider initial value problem (IVP): x(t i ) = x i If f is Lipschitz continuous in x for all t 2 [ t i, t f ] –There exists a unique solution x(t) for t 2 [ t i, t f ] for each x i such that dx /dt exists and dx/dt = f(t,x) –For perturbed initial data y i yielding y(t) –For perturbed dynamics Sufficient but not necessary conditions

March 2008Ian Mitchell (UBC Computer Science)14 Ill-Posed ODEs Why do we care that the ODE is well-posed? –Theory: much depends on the existence of a unique solution –Numerics: approximate solution may not be desired solution, and may not even be near a true solution

March 2008Ian Mitchell (UBC Computer Science)15 Syntax vs Semantics Syntax: what are legal statements? –Boolean expression over variable x 2 { 0, 1 } and boolean expressions f and g: x | 0 | 1 | ¬ f | fg | f + g | f ◦ g | (f) –Arithmetic expression over x 2 R [ 1 and expressions f and g: x | –f | f + g | f – g | fg | f / g | f ◦ g | (f) Semantics: what do those statements mean? –Boolean expression “or” –Arithmetic expression = 9 x0011 y0101 x+yx+y 0111

March 2008Ian Mitchell (UBC Computer Science)16 Checking a Model Well-posed conditions are examples of syntactic checks: tests applied directly to the model –Model does not itself evolve, but is a static entity –Complexity of check depends only on the complexity of the model Alternative: Semantic checks –Requires understanding the evolving solution –Complexity of check depends on the complexity of the solution trajectory

March 2008Ian Mitchell (UBC Computer Science)17 Restricted Classes of Model Many results in control and verification assume a restricted class of models –Permits more checks to be syntactic / static –May simplify checks of semantic / dynamic –Example: Are there any syntactically correct but semantically incorrect boolean expressions? –Example: Are there any syntactically correct but semantically incorrect arithmetic expressions other than 0 / 0? Our nonlinear ODE and DI models are very general –Most of what we discuss (beyond well-posedness) will be semantic / dynamic checks

March 2008Ian Mitchell (UBC Computer Science)18 Visualization Most of the visualization of system evolution will be done in the phase or state space (ignore time) –Pendulum states angle and angular velocity phase Space state vs time pendulum workspace

March 2008Ian Mitchell (UBC Computer Science)19 Outline Classes of models –Well-posed models –Difference Equations –Nonlinear Ordinary Differential Equations –Syntax vs semantics –Visualization Optimal Control –Objective and value functions Verification: Reachability –Forward & backward, tubes & sets, maximal & minimal Dimitri Bertsekas, Dynamic Programming & Optimal Control, Athena Scientific (3 rd edition 2005)

March 2008Ian Mitchell (UBC Computer Science)20 Achieving Desired Behaviours We can attempt to control a system when there is a parameter u of the dynamics (the “control input”) which we can influence –Time dependent dynamics are possible, but we will mostly deal with time invariant systems Without a control signal specification, system is nondeterministic –Current state cannot predict unique future evolution Control signal may be specified –Open-loop u(t) or u: R → U –Feedback, closed-loop u(x(t)) or u: S → U –Either choice makes the system deterministic again

March 2008Ian Mitchell (UBC Computer Science)21 Visualization: Vector Fields Introduction of a free control input changes the vector field plot in the phase space into a field of cones (nondeterministic) Feedback control law changes it back into a (static) vector field Open loop control law does not no inputs (“autonomous” for control engineers) unspecified input signal feedback input signal

March 2008Ian Mitchell (UBC Computer Science)22 Objective Function We distinguish quality of control by an objective / payoff / cost function, which comes in many different variations –eg: discrete time discounted with fixed finite horizon t f –eg: continuous time no discount with target set T

March 2008Ian Mitchell (UBC Computer Science)23 Value Function Choose input signal to optimize the objective –Optimize: “cost” is usually minimized, “payoff” is usually maximized and “objective” may be either Value function is the optimal value of the objective function –May not be achieved for any signal (eg: min should be inf) Set of signals U is contentious –For implementation purposes, we desire restricted classes: bounded, continuous, piecewise constant –Unfortunately, theory applies to (and thus can only guarantee optimality with) very general classes: measurable

March 2008Ian Mitchell (UBC Computer Science)24 Example: LQR for Linear Systems Much of the “optimal control” literature and most classes focus (without mentioning it) on linear systems Corresponding objective functions are usually quadratic where A, B, Q, R, Q f are all matrices of appropriate size Successful but restricted class of problems –Not rigorously part of the results to follow (due to a technicality)

March 2008Ian Mitchell (UBC Computer Science)25 Outline Classes of models –Well-posed models –Difference Equations –Nonlinear Ordinary Differential Equations –Syntax vs semantics –Visualization Optimal Control –Objective and value functions Verification: Reachability –Forward & backward, tubes & sets, maximal & minimal Ian Mitchell, “Comparing Forward and Backward Reachability as Tools for Safety Analysis,” Hybrid Systems Computation and Control, LNCS 4416, Springer-Verlag (2007).

March 2008Ian Mitchell (UBC Computer Science)26 Verification: Safety Analysis Does there exist a trajectory of system H leading from a state in initial set I to a state in terminal set T ? (under some policy for input u(¢))

March 2008Ian Mitchell (UBC Computer Science)27 Typical Systems: ODEs Common model for continuous state spaces Standard existence and uniqueness

March 2008Ian Mitchell (UBC Computer Science)28 Working with Sets Optimal control works with a single optimal trajectory Verification works with sets of trajectories –Takes a nondeterministic (but not probabilistic) viewpoint Basic construct is reachability –Many versions: forward and backward, sets or tubes –What should the input do? Many related concepts in control theory –Invariant sets, controlled invariant sets, stability Safety is not the only verification goal –Liveness is a common goal, but often harder to verify

March 2008Ian Mitchell (UBC Computer Science)29 Forward Reachability Start at initial conditions and compute forward

March 2008Ian Mitchell (UBC Computer Science)30 Backward Reachability Start at terminal set and compute backwards

March 2008Ian Mitchell (UBC Computer Science)31 Exchanging Algorithms Algorithms are (mathematically) interchangeable if system dynamics can be reversed in time For example: Then

March 2008Ian Mitchell (UBC Computer Science)32 Maximal Reachability Input signal u(¢) maximizes size of the set or tube

March 2008Ian Mitchell (UBC Computer Science)33 Maximal Reachability Definition

March 2008Ian Mitchell (UBC Computer Science)34 Maximal Reachability Results Reach sets and tubes provide similar information The following properties are equivalent Any maximal reachability operator can be used to demonstrate safety for all possible inputs

March 2008Ian Mitchell (UBC Computer Science)35 Maximal Reachability Demonstration System Dynamics Forward Reach Set Results Initial and Terminal Sets

March 2008Ian Mitchell (UBC Computer Science)36 Maximal Reachability Demonstration System Dynamics Forward Reach Tube Results Initial and Terminal Sets

March 2008Ian Mitchell (UBC Computer Science)37 Maximal Reachability Demonstration System Dynamics Backward Reach Set Results Initial and Terminal Sets

March 2008Ian Mitchell (UBC Computer Science)38 Maximal Reachability Demonstration System Dynamics Backward Reach Tube Results Initial and Terminal Sets

March 2008Ian Mitchell (UBC Computer Science)39 Minimal Reachability Input signal u(¢) minimizes size of the set or tube

March 2008Ian Mitchell (UBC Computer Science)40 Minimal Reachability Definition

March 2008Ian Mitchell (UBC Computer Science)41 Minimal Reachability Results Reach tubes provide more information –Choice of trajectory length t is quantified first for sets but last for tubes

March 2008Ian Mitchell (UBC Computer Science)42 Minimal Reachability Results Backward reach tubes are the only minimal reachability operator that can prove that there exists an input u(¢) which keeps the system safe –Basic problem with minimal forward reachability: the state lying in the terminal set is chosen before the input, while the state lying in the initial set is chosen after

March 2008Ian Mitchell (UBC Computer Science)43 Minimal Reachability Demonstration System Dynamics (Correct) Backward Reach Tube Results Initial and Terminal Sets

March 2008Ian Mitchell (UBC Computer Science)44 Minimal Reachability Demonstration System Dynamics (Incorrect) Forward Reach Tube Results Initial and Terminal Sets

Models for Control and Verification For more information contact Ian Mitchell Department of Computer Science The University of British Columbia