CS6800 Advanced Theory of Computation Fall 2012 Vinay B Gavirangaswamy Markov Models CS6800 Advanced Theory of Computation Fall 2012 Vinay B Gavirangaswamy

Markov Property Processes future values are conditionally dependent on the present state of the system. Strong Markov Property Similar as Markov Property, where values are conditionally dependent on the stopping time (Markov time) instead of present state. Introduction

Markov Process Process that follows Markov Property. e. g Markov Process Process that follows Markov Property. e.g. First order Markov Process for probability space (, F, P) P(Xt+1=Sj | Xt=Si, Xt-1=Sk ,...) = P(Xt+1=Sj | qt=Si) , for t = 1, 2, 3, … where,  = {S1, S2, S3, …, Sn }, State at time t is Xt=Si Markov Model Model’s a stochastic system which assumes Markov property. It is formally represented by triplet M = (K, , A), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. Markov model is an non deterministic finite state machine. Introduction (Contd.)

Simple Markov Model of Weather Stochastic FSM The transition matrix: Stochastic matrix: Rows sum up to 1 Double stochastic matrix: Rows and columns sum up to 1 Simple Markov Model of Weather

Prediction Based on State Transition Probability If we want to know probability of the sequence SUNNY SUNNY SUNNY SUNNY SUNNY Take initial probability of SUNNY day i.e. on a any given day probability that it will be SUNNY is 0.30 And for use to get another SUNNY day after a SUNNY day is 0.42 So, by using Markov Chain we can say probability of getting 5 consecutive SUNNY day is Prediction Based on State Transition Probability

Types of Markov Model System state is fully observable System state is partially observable System is autonomous Markov chain Hidden Markov model System is controlled Markov decision process Partially observable Markov decision process Types of Markov Model

Types of Markov Model (Contd.) Markov Chain Simplest Markov model System state at time t+1 depends on state at t. Hidden Markov Model Similar to Markov chain but system states are unobservable. Markov decision process Models a system where outcomes are partially random and partially under the control of a decision maker. Types of Markov Model (Contd.)

Types of Markov Model (Contd.) Markov decision process It is formally represented by 4-tuple M = (K, , A, R), where K is a finite set of states  vector of initial probabilities for each of the states. A is a matrix representing transition probabilities. R is reward given for transition from state St to Sp with transition probability given in matrix A. Types of Markov Model (Contd.)

Types of Markov Model (Contd.) Partially observable Markov decision process Similar to Markov decision process where state of system is partially observed. It is formally represented by 4-tuple M = (K, , O , A, ), where K is a finite set of states.  is a set of actions. O is a set of observations. A is a matrix representing transition probabilities.  is a set of conditional observation probabilities. Types of Markov Model (Contd.)

System that transitions from one state to another, between a finite number of possible states. Next state depends only on current state and not its history. Follows Markov property. Formally for a stochastic process { Xt } P{ Xt+1 = j | X0 = k0, . . . , Xt-1 = kt-1, Xt = i } = P{ Xt+1 = j | Xt = i } for every i, j, k0, . . . , kt-1 and for every t. Stationary Markov Chains Pr{ Xt+1 = j | Xt = i } = Pr{ X1 = j | X0 = i } for all t Markov Chain

If the state space S = { 0, 1, . . . , m–1} then we have j pij = 1  i and pij  0  i, j where pij = Pr{ X1 = j | X0 = i } (we must go some where) (each transition has probability  0) Transition Matrix

Representation of a Markov Chain as a Digraph Each directed edge AB is associated with the positive transition probability from A to B. A B C D 0.2 0.3 0.5 0.05 0.95 0.8 1 1 0.8 0.2 0.3 0.5 0.05 0.95 A B C D Representation of a Markov Chain as a Digraph

Properties of Markov Chain states States of Markov chains are classified by the digraph representation (omitting the actual probability values) A, C and D are recurrent states: they are in strongly connected components which are sinks in the graph. A B C D B is not recurrent – it is a transient state Alternative definitions: A state s is recurrent if it can be reached from any state reachable from s; otherwise it is transient. Properties of Markov Chain states

Another example of Recurrent and Transient States B C D A and B are transient states, C and D are recurrent states. Once the process moves from B to D, it will never come back. Another example of Recurrent and Transient States

Irreducible Markov Chains A Markov Chain is irreducible if the corresponding graph is strongly connected (and thus all its states are recurrent). A B C D 1 A B C D E Irreducible Markov Chains

A state s has a period k if k is the GCD of the lengths of all the cycles that pass via s. (in the shown graph the period of A is 2). A Markov Chain is periodic if all the states in it have a period k >1. It is aperiodic otherwise. A B C D E Periodic States

A B C D A Markov chain is ergodic if : the corresponding graph is strongly connected. It is not peridoic Ergodic Markov Chains are important since they guarantee the corresponding Markovian process converges to a unique distribution, in which all states have strictly positive probability. Ergodic Markov Chains

Reversible Markov chain A Markov chain is reversible if Reversible Markov chain

Markov Model Applications Physics Chemistry Testing Information sciences Queuing theory Internet applications Statistics Economics and finance Social science Mathematical biology Games Music Markov Model Applications

"Wkipedia-Markov model," [Online]. Available: http://en. wikipedia "Wkipedia-Markov model," [Online]. Available: http://en.wikipedia.org/wiki/Markov_chain. [Accessed 28 10 2012]. P. Xinhui Zhang, "DTMarkovchains," [Online]. Available: http://www.wright.edu/~xinhui.zhang/. [Accessed 28 10 2012]. References