Metrics for real time probabilistic processes Radha Jagadeesan, DePaul University Vineet Gupta, Google Inc Prakash Panangaden, McGill University Josee Desharnais, Univ Laval

Outline of talk Models for real-time probabilistic processes Approximate reasoning for real-time probabilistic processes

Discrete Time Probabilistic processes Labelled Markov Processes For each state s For each label a K(s, a, U) Each state labelled with propositional information 0.3 0.5 0.2

Discrete Time Probabilistic processes Markov Decision Processes For each state s For each label a K(s, a, U) Each state labelled with numerical rewards 0.3 0.5 0.2

Discrete time probabilistic proceses + nondeterminism : label does not determine probability distribution uniquely.

Real-time probabilistic processes Add clocks to Markov processes Each clock runs down at fixed rate r c(t) = c(0) – r t Different clocks can have different rates Generalized SemiMarkov Processes Probabilistic multi-rate timed automata

Generalized semi-Markov processes. Each state labelled with propositional Information Each state has a set of clocks associated with it. {c,d} s u {c} {d,e} t

Generalized semi-Markov processes. Evolution determined by generalized states <state, clock-valuation> <s,c=2, d=1> Transition enabled when a clock becomes zero {c,d} s u {c} {d,e} t

Generalized semi-Markov processes. <s,c=2, d=1> Transition enabled in 1 time unit <s,c=0.5,d=1> Transition enabled in 0.5 time unit {c,d} s u {c} {d,e} t Clock c Clock d

Generalized semi-Markov processes. Transition determines: a. Probability distribution on next states b. Probability distribution on clock values for new clocks {c,d} s 0.2 0.8 u {c} {d,e} t Clock c Clock d

Generalized semi Markov proceses If distributions are continuous and states are finite: Zeno traces have measure 0 Continuity results. If stochastic processes from <s, > converge to the stochastic process at <s, >

Equational reasoning Establishing equality: Coinduction Distinguishing states: Modal logics Equational and logical views coincide Compositional reasoning: ``bisimulation is a congruence’’

Labelled Markov Processes PCTL Bisimulation [Larsen-Skou, Desharnais-Panangaden-Edalat] Markov Decision Processes Bisimulation [Givan-Dean-Grieg] Labelled Concurrent Markov Chains PCTL [Hansson-Johnsson] Labelled Concurrent Markov chains (with tau) PCTL [Desharnais-Gupta-Jagadeesan-Panangaden] Weak bisimulation [Philippou-Lee-Sokolsky, Lynch-Segala]

With continuous time Continuous time Markov chains CSL [Aziz-Balarin-Brayton-Sanwal-Singhal-S.Vincentelli] Bisimulation,Lumpability [Hillston, Baier-Katoen-Hermanns] Generalized Semi-Markov processes Stochastic hybrid systems CSL Bisimulation:????? Composition:?????

Alas!

Instability of exact equivalence Vs Vs

Problem! Numbers viewed as coming with an error estimate. (eg) Stochastic noise as abstraction Statistical methods for estimating numbers

Problem! Numbers viewed as coming with an error estimate. Reasoning in continuous time and continuous space is often via discrete approximations. eg. Monte-Carlo methods to approximate probability distributions by a sample.

Idea: Equivalence metrics Jou-Smolka, Lincoln-Scedrov-Mitchell-Mitchell Replace equality of processes by (pseudo)metric distances between processes Quantitative measurement of the distinction between processes.

Criteria on approximate reasoning Soundness Usability Robustness

Criteria on metrics for approximate reasoning Soundness Stability of distance under temporal evolution: ``Nearby states stay close '‘ through temporal evolution.

``Usability’’ criteria on metrics Establishing closeness of states: Coinduction. Distinguishing states: Real-valued modal logics. Equational and logical views coincide: Metrics yield same distances as real-valued modal logics.

``Robustness’’ criterion on approximate reasoning The actual numerical values of the metrics should not matter --- ``upto uniformities’’.

Uniformities (same) m(x,y) = |x-y| m(x,y) = |2x + sinx -2y – siny|

Uniformities (different) m(x,y) = |x-y|

Our results

Our results For Discrete time models: Labelled Markov processes Labelled Concurrent Markov chains Markov decision processes For continuous time: Generalized semi-Markov processes

Results for discrete time models Bisimulation Metrics Logic (P)CTL(*) Real-valued modal logic Compositionality Congruence Non-expansivity Proofs Coinduction

Results for continuous time models Bisimulation Metrics Logic CSL Real-valued modal logic Compositionality ??? Proofs Coinduction

Metrics for discrete time probablistic processes

Bisimulation Fix a Markov chain. Define monotone F on equivalence relations:

Defining metric: An attempt Define functional F on metrics.

Metrics on probability measures Wasserstein-Kantorovich A way to lift distances from states to a distances on distributions of states.

Metrics on probability measures

Metrics on probability measures

Example 1: Metrics on probability measures Unit measure concentrated at x Unit measure concentrated at y m(x,y) x y

Example 1: Metrics on probability measures Unit measure concentrated at x Unit measure concentrated at y m(x,y) x y

Example 2: Metrics on probability measures

Example 2: Metrics on probability measures THEN:

Lattice of (pseudo)metrics

Defining metric coinductively Define functional F on metrics Desired metric is maximum fixed point of F

Real-valued modal logic

Real-valued modal logic Tests:

Real-valued modal logic (Boolean) q q

Real-valued modal logic

Results Modal-logic yields the same distance as the coinductive definition However, not upto uniformities since glbs in lattice of uniformities is not determined by glbs in lattice of pseudometrics.

Variant definition that works upto uniformities Fix c<1. Define functional F on metrics Desired metric is maximum fixed point of F

Reasoning upto uniformities For all c<1, get same uniformity [see Breugel/Mislove/Ouaknine/Worrell] Variant of earlier real-valued modal logic incorporating discount factor c characterizes the metrics

Metrics for real-time probabilistic processes

Generalized semi-Markov processes. Evolution determined by generalized states <state, clock-valuation> : Set of generalized states {c,d} s u {c} {d,e} t Clock c Clock d

Generalized semi-Markov processes. {c,d} Path: Traces((s,c)): Probability distribution on a set of paths. s u {c} {d,e} t Clock c Clock d

Accomodating discontinuities: cadlag functions (M,m) a pseudometric space. cadlag if:

Countably many jumps, in general

Defining metric: An attempt Define functional F on metrics. (c <=1) traces((s,c)), traces((t,d)) are distributions on sets of cadlag functions. What is a metric on cadlag functions???

Metrics on cadlag functions x y are at distance 1 for unequal x,y Not separable!

Skorohod metrics (J2) (M,m) a pseudometric space. f,g cadlag with range M. Graph(f) = { (t,f(t)) | t \in R+}

Skorohod J2 metric: Hausdorff distance between graphs of f,g (t,f(t)) f(t) g(t) t

Skorohod J2 metric (M,m) a pseudometric space. f,g cadlag

Examples of convergence to

Example of convergence 1/2

Example of convergence 1/2

Examples of convergence 1/2

Examples of convergence 1/2

Examples of non-convergence Jumps are detected!

Non-convergence

Non-convergence

Non-convergence

Non-convergence

Summary of Skorohod J2 A separable metric space on cadlag functions

Defining metric coinductively Define functional on 1-bounded pseudometrics (c <=1) a. s, t agree on all propositions b. Desired metric: maximum fixpoint of F

Real-valued modal logic

Real-valued modal logic

Real-valued modal logic h: Lipschitz operator on unit interval

Real-valued modal logic

Real-valued modal logic Base case for path formulas??

Base case for path formulas First attempt: Evaluate state formula F on state at time t Problem: Not smooth enough wrt time since paths have discontinuities

Base case for path formulas Next attempt: ``Time-smooth’’ evaluation of state formula F at time t on path Upper Lipschitz approximation to evaluated at t

Real-valued modal logic

Non-convergence

Illustrating Non-convergence 1/2 1/2

Results For each c<=1, modal-logic yields the same distance as the coinductive definition All c<1 yield the same uniformity. In this case, construction can be carried out in lattice of uniformities.

Proof steps Continuity theorems (Whitt) of GSMPs yield separable basis Finite separability arguments yield closure ordinal of functional F is omega. Duality theory of LP for calculating metric distances

Results Approximating quantitative observables: Expectations of continuous functions are continuous Continuous mapping theorems for establishing continuity of quantitative observables

Summary Approximate reasoning for real-time probabilistic processes

Results for discrete time models Bisimulation Metrics Logic (P)CTL(*) Real-valued modal logic Compositionality Congruence Non-expansivity Proofs Coinduction

Results for continuous time models Bisimulation Metrics Logic CSL Real-valued modal logic Compositionality ??? Proofs Coinduction

Questions?

Real-valued modal logic