Topics Review of DTMC Classification of states Economic analysis

Slides:



Advertisements
Similar presentations
Discrete time Markov Chain
Advertisements

1 Introduction to Discrete-Time Markov Chain. 2 Motivation  many dependent systems, e.g.,  inventory across periods  state of a machine  customers.
. Markov Chains. 2 Dependencies along the genome In previous classes we assumed every letter in a sequence is sampled randomly from some distribution.
Lecture 6  Calculating P n – how do we raise a matrix to the n th power?  Ergodicity in Markov Chains.  When does a chain have equilibrium probabilities?
Queueing Models and Ergodicity. 2 Purpose Simulation is often used in the analysis of queueing models. A simple but typical queueing model: Queueing models.
Continuous-Time Markov Chains Nur Aini Masruroh. LOGO Introduction  A continuous-time Markov chain is a stochastic process having the Markovian property.
CS433 Modeling and Simulation Lecture 06 – Part 03 Discrete Markov Chains Dr. Anis Koubâa 12 Apr 2009 Al-Imam Mohammad Ibn Saud University.
Operations Research: Applications and Algorithms
Markov Chains.
Markov Chains 1.
. Computational Genomics Lecture 7c Hidden Markov Models (HMMs) © Ydo Wexler & Dan Geiger (Technion) and by Nir Friedman (HU) Modified by Benny Chor (TAU)
Lecture 12 – Discrete-Time Markov Chains
TCOM 501: Networking Theory & Fundamentals
Solutions Markov Chains 1
Introduction to Discrete Time Semi Markov Process Nur Aini Masruroh.
G12: Management Science Markov Chains.
Chapter 17 Markov Chains.
Flows and Networks (158052) Richard Boucherie Stochastische Operations Research -- TW wwwhome.math.utwente.nl/~boucherierj/onderwijs/158052/ html.
1 Part III Markov Chains & Queueing Systems 10.Discrete-Time Markov Chains 11.Stationary Distributions & Limiting Probabilities 12.State Classification.
Андрей Андреевич Марков. Markov Chains Graduate Seminar in Applied Statistics Presented by Matthias Theubert Never look behind you…
Markov Processes MBAP 6100 & EMEN 5600 Survey of Operations Research Professor Stephen Lawrence Leeds School of Business University of Colorado Boulder,
Entropy Rates of a Stochastic Process
Markov Chain Part 2 多媒體系統研究群 指導老師:林朝興 博士 學生:鄭義繼. Outline Review Classification of States of a Markov Chain First passage times Absorbing States.
Tutorial 8 Markov Chains. 2  Consider a sequence of random variables X 0, X 1, …, and the set of possible values of these random variables is {0, 1,
INDR 343 Problem Session
Lecture 13 – Continuous-Time Markov Chains
. PGM: Tirgul 8 Markov Chains. Stochastic Sampling  In previous class, we examined methods that use independent samples to estimate P(X = x |e ) Problem:
048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion Review.
20. Extinction Probability for Queues and Martingales
2003 Fall Queuing Theory Midterm Exam(Time limit:2 hours)
1 Markov Chains Algorithms in Computational Biology Spring 2006 Slides were edited by Itai Sharon from Dan Geiger and Ydo Wexler.
Finite Mathematics & Its Applications, 10/e by Goldstein/Schneider/SiegelCopyright © 2010 Pearson Education, Inc. 1 of 60 Chapter 8 Markov Processes.
CSE 221: Probabilistic Analysis of Computer Systems Topics covered: Discrete time Markov chains (Sec )
Markov Chains Chapter 16.
INDR 343 Problem Session
Stochastic Process1 Indexed collection of random variables {X t } t   for each t  T  X t is a random variable T = Index Set State Space = range.
CS6800 Advanced Theory of Computation Fall 2012 Vinay B Gavirangaswamy
Lecture 11 – Stochastic Processes
The effect of New Links on Google Pagerank By Hui Xie Apr, 07.
Business Modeling Lecturer: Ing. Martina Hanová, PhD.
Intro. to Stochastic Processes
S TOCHASTIC M ODELS L ECTURE 1 P ART II M ARKOV C HAINS Nan Chen MSc Program in Financial Engineering The Chinese University of Hong Kong (ShenZhen) Sept.
1 Introduction to Stochastic Models GSLM Outline  limiting distribution  connectivity  types of states and of irreducible DTMCs  transient,
Markov Decision Processes1 Definitions; Stationary policies; Value improvement algorithm, Policy improvement algorithm, and linear programming for discounted.
Decision Making in Robots and Autonomous Agents Decision Making in Robots and Autonomous Agents The Markov Decision Process (MDP) model Subramanian Ramamoorthy.
Markov Chains and Random Walks. Def: A stochastic process X={X(t),t ∈ T} is a collection of random variables. If T is a countable set, say T={0,1,2, …
Markov Chains X(t) is a Markov Process if, for arbitrary times t1 < t2 < < tk < tk+1 If X(t) is discrete-valued If X(t) is continuous-valued i.e.
Chapter 61 Continuous Time Markov Chains Birth and Death Processes,Transition Probability Function, Kolmogorov Equations, Limiting Probabilities, Uniformization.
 { X n : n =0, 1, 2,...} is a discrete time stochastic process Markov Chains.
Chapter 3 : Problems 7, 11, 14 Chapter 4 : Problems 5, 6, 14 Due date : Monday, March 15, 2004 Assignment 3.
8/14/04J. Bard and J. W. Barnes Operations Research Models and Methods Copyright All rights reserved Lecture 12 – Discrete-Time Markov Chains Topics.
CS433 Modeling and Simulation Lecture 07 – Part 01 Continuous Markov Chains Dr. Anis Koubâa 14 Dec 2008 Al-Imam.
The generalization of Bayes for continuous densities is that we have some density f(y|  ) where y and  are vectors of data and parameters with  being.
Discrete Time Markov Chains
To be presented by Maral Hudaybergenova IENG 513 FALL 2015.
Flows and Networks (158052) Richard Boucherie Stochastische Operations Research -- TW wwwhome.math.utwente.nl/~boucherierj/onderwijs/158052/ html.
CS433 Modeling and Simulation Lecture 11 Continuous Markov Chains Dr. Anis Koubâa 01 May 2009 Al-Imam Mohammad Ibn Saud University.
Meaning of Markov Chain Markov Chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only.
11. Markov Chains (MCs) 2 Courtesy of J. Bard, L. Page, and J. Heyl.
Markov Processes What is a Markov Process?
8/14/04J. Bard and J. W. Barnes Operations Research Models and Methods Copyright All rights reserved Lecture 11 – Stochastic Processes Topics Definitions.
1 Introduction to Stochastic Models GSLM Outline  transient behavior  first passage time  absorption probability  limiting distribution 
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 1 of 60 Chapter 8 Markov Processes.
Theory of Computational Complexity Probability and Computing Lee Minseon Iwama and Ito lab M1 1.
Business Modeling Lecturer: Ing. Martina Hanová, PhD.
From DeGroot & Schervish. Example Occupied Telephone Lines Suppose that a certain business office has five telephone lines and that any number of these.
Let E denote some event. Define a random variable X by Computing probabilities by conditioning.
Discrete-time markov chain (continuation)
IENG 362 Markov Chains.
Discrete-time markov chain (continuation)
Presentation transcript:

Lecture 12.5 – Additional Issues Concerning Discrete-Time Markov Chains Topics Review of DTMC Classification of states Economic analysis First-time passage Absorbing states

Discrete-Time Markov Chain A stochastic process { Xn } where n  N = { 0, 1, 2, . . . } is called a discrete-time Markov chain if Pr{ Xn+1 = j | X0 = k0, . . . , Xn-1 = kn-1, Xn = i } = Pr{ Xn+1 = j | Xn = i }  transition probabilities for every i, j, k0, . . . , kn-1 and for every n. The future behavior of the system depends only on the current state i and not on any of the previous states.

Stationary Transition Probabilities Pr{ Xn+1 = j | Xn = i } = Pr{ X1 = j |X0 = i } for all n (They don’t change over time) We will only consider stationary Markov chains. The one-step transition matrix for a Markov chain with states S = { 0, 1, 2 } is where pij = Pr{ X1 = j | X0 = i }

Classification of States Accessible: Possible to go from state i to state j (path exists in the network from i to j). Two states communicate if both are accessible from each other. A system is irreducible if all states communicate. State i is recurrent if the system will return to it after leaving some time in the future. If a state is not recurrent, it is transient.

Classification of States (continued) A state is periodic if it can only return to itself after a fixed number of transitions greater than 1 (or multiple of a fixed number). A state that is not periodic is aperiodic. a. Each state visited every 3 iterations b. Each state visited in multiples of 3 iterations

Classification of States (continued) An absorbing state is one that locks in the system once it enters. This diagram might represent the wealth of a gambler who begins with $2 and makes a series of wagers for $1 each. Let ai be the event of winning in state i and di the event of losing in state i. There are two absorbing states: 0 and 4.

Classification of States (continued) Class: set of states that communicate with each other. A class is either all recurrent or all transient and may be either all periodic or aperiodic. States in a transient class communicate only with each other so no arcs enter any of the corresponding nodes in the network diagram from outside the class. Arcs may leave, though, passing from a node in the class to one outside.

Illustration of Concepts Example 1 Every pair of states communicates, forming a single recurrent class; however, the states are not periodic. Thus the stochastic process is aperiodic and irreducible.

Illustration of Concepts Example 2 States 0 and 1 communicate and for a recurrent class. States 3 and 4 form separate transient classes. State 2 is an absorbing state and forms a recurrent class.

Illustration of Concepts Example 3 Every state communicates with every other state, so we have irreducible stochastic process. Periodic? Yes, so Markov chain is irreducible and periodic.

Classification of States Example .6 .7 1 2 .4 3 4 .5 .4 .5 .5 .3 .8 .1 5 .2

A state j is accessible from state i if pij(n) > 0 for some n > 0. In example, state 2 is accessible from state 1 & state 3 is accessible from state 5 but state 3 is not accessible from state 2. States i and j communicate if i is accessible from j and j is accessible from i. States 1 & 2 communicate; also states 3, 4 & 5 communicate. States 2 & 4 do not communicate States 1 & 2 form one communicating class. States 3, 4 & 5 form a 2nd communicating class.

If all states in a Markov chain communicate (i.e., all states are members of the same communicating class) then the chain is irreducible. The current example is not an irreducible Markov chain. Neither is the Gambler’s Ruin example which has 3 classes: {0}, {1, 2, 3} and {4}. First Passage Times Let fii = probability that the process will return to state i (eventually) given that it starts in state i. If fii = 1 then state i is called recurrent. If fii < 1 then state i is called transient.

If pii = 1 then state i is called an absorbing state. Above example has no absorbing states States 0 & 4 are absorbing in Gambler’s Ruin problem. The period of a state i is the smallest k > 1 such that all paths leading back to i have a length that is a multiple of k; i.e., pii(n) = 0 unless n = k, 2k, 3k, . . . If a process can be in state i at time n or time n + 1 having started at state i then state i is aperiodic. Each of the states in the current example are aperiodic

Example of Periodicity - Gambler’s Ruin States 1, 2 and 3 each have period 2. 0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0 2 0 1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1 If all states in a Markov chain are recurrent, aperiodic, & the chain is irreducible then it is ergodic.

Existence of Steady-State Probabilities A Markov chain is ergodic if it is aperiodic and allows the attainment of any future state from any initial state after one or more transitions. If these conditions hold, then For example, State-transition network 1 3 2 Conclusion: chain is ergodic.

Economic Analysis Two kinds of economic effects: those incurred when the system is in a specified state, and those incurred when the system makes a transition from one state to another. The cost (profit) of being in a particular state is represented by the m-dimensional column vector where each component is the cost associated with state i. The cost of a transition is embodied in the m  m matrix . where each component specifies the cost of going from state i to state j in a single step.

Expected Cost for Markov Chain Expected cost of being in state i : Let C = (c1, . . . cm)T ei = (0, 0, 1, 0, 0) be the ith row of the m  m identity matrix, and fn = a random variable representing the economic return associated with the stochastic process at time n. Property 3: Let {Xn : n = 0, 1, . . .} be a Markov chain with finite state space S, state-transition matrix P, and expected state cost (profit) vector C. Assuming that the process starts in state i, the expected cost (profit) at the nth step is given by  E[fn(Xn) |X0 = i] = eiP(n)C.

Additional Cost Results What if the initial state is not known? Property 5: Let {Xn : n = 0, 1, . . .} be a Markov chain with finite state space S, state-transition matrix P, initial probability vector q(0), and expected state cost (profit) vector C. The expected economic return at the nth step is given by   E[fn(Xn) |q(0)] = q(0)P(n)C. Property 6: Let {Xn : n = 0, 1, . . .} be a Markov chain with finite state space S, state-transition matrix P, steady-state vector π, and expected state cost (profit) vector C. Then the long-run average return per unit time is given by   SiS πici = πC.

Insurance Company Example An insurance company charges customers annual premiums based on their accident history in the following fashion: No accident in last 2 years: $250 annual premium  Accidents in each of last 2 years: $800 annual premium  Accident in only 1 of last 2 years: $400 annual premium Historical statistics: If a customer had an accident last year then they have a 10% chance of having one this year; If they had no accident last year then they have a 3% chance of having one this year.

(N, N) (N, Y) (Y, N) (Y, Y) (N, N) 0.97 0.03 0 0 (N, Y) 0 0 0.90 0.10 Problem: Find the steady-state probability and the long-run average annual premium paid by the customer. Solution approach: Construct a Markov chain with four states: (N, N), (N, Y), (Y, N), (Y,Y) where these indicate (accident last year, accident this year). (N, N) (N, Y) (Y, N) (Y, Y) (N, N) 0.97 0.03 0 0 (N, Y) 0 0 0.90 0.10 (Y, N) 0.97 0.03 0 0 (Y, Y) 0 0 0.90 0.10 P =

State-Transition Network for Insurance Company .90 .03 .90 .03 Y, N Y, Y N, N N, Y .97 .10 .97 .10 This is an ergodic Markov chain. All states communicate (irreducible) Each state is recurrent (you will return, eventually) Each state is aperiodic

Solving the Steady–State Equations m i=1 (N,N) = 0.97 (N,N) + 0.97 (Y,N) (N,Y) = 0.03 (N,N) + 0.03 (Y,N) (Y,N) = 0.9 (N,Y) + 0.9 (Y,Y) (N,N) + (N,Y)+(Y,N) + (Y,Y) = 1 pj = å pipij, j = 0,…,m å pj = 1, pj  0,  j m j =1 Solution: (N,N) = 0.939, (N,Y) = 0.029, (Y,N) = 0.029, (Y,Y) = 0.003 & the long-run average annual premium is 0.939*250 + 0.029*400 + 0.029*400 + 0.003*800 = 260.5

Markov Chain Add-in Matrix

Economic Data and Solution

Transient Analysis for Insurance Company

First Passage Times Let ij = expected number of steps to transition from state i to state j If the probability that we will eventually visit state j given that we start in i is less than 1, then we will have ij = +. For example, in the Gambler’s Ruin problem, 20 = + because there is a positive probability that we will be absorbed in state 4 given that we start in state 2 (and hence visit state 0).

Computations for All States Recurrent If the probability of eventually visiting state j given that we start in i is 1 then the expected number of steps until we first visit j is given by mij = 1 + å pirmrj, for i = 0,1, . . . , m–1 rj We go from i to r in the first step with probability pir and it takes mrj steps from r to j. It will always take at least one step. For j fixed, we have linear system in m equations and m unknowns mij , i = 0,1, . . . , m–1.

First-Passage Analysis for Insurance Company Suppose that we start in state (N,N) and want to find the expected number of years until we have accidents in two consecutive years (Y,Y). This transition will occur with probability 1, eventually. For convenience number the states 0 1 2 3 (N,N) (N,Y) (Y,N) (Y,Y) Then, 03 = 1 + p00 03 + p01 13 + p0223 13 = 1 + p10 03 + p11 13 + p1223 23 = 1 + p20 03 + p21 13 + p2223

First-Passage Computations (N, N) (N, Y) (Y, N) (Y, Y) (N, N) 0.97 0.03 0 0 (N, Y) 0 0 0.90 0.10 (Y, N) 0.97 0.03 0 0 (Y, Y) 0 0 0.90 0.10 1 2 3 states Using P = 03 = 1 + 0.9703 + 0.0313 13 = 1 + 0.923 23 = 1 + 0.9703 + 0.0313 So, on average it takes 343.3 years to transition from (N,N) to (Y,Y). Note, 03 = 23. Why? Note, 13 < 03. Solution: 03 = 343.3, 13 = 310, 23 = 343.3

First Passage Probabilities

Game of Craps Probability of win = Pr{ 7 or 11 } = 0.167 + 0.056 = 0.223 Probability of loss = Pr{ 2, 3, 12 } = 0.028 + 0.56 + 0.028 = 0.112

First Passage Probabilities for Craps

Absorbing States An absorbing state is a state j with pjj = 1. Given that we start in state i, we can calculate the probability of being absorbed in state j. We essentially performed this calculation for the Gambler’s Ruin problem by finding P(n) = (pij(n) ) for large n. But we can use a more efficient analysis like that used for calculating first passage times.

Let 0, 1, . . . , k be transient states and k + 1, . . . , m – 1 be absorbing states. Let qij = probability of being absorbed in state j given that we start in transient state i. Then for each j we have the following relationship qij = pij +  pirqrj , i = 0, 1, . . . , k k r = 0 Go directly to j Go to r and then to j For fixed j (absorbing state) we have k + 1 linear equations in k + 1 unknowns, qrj , i = 0, 1, . . . , k.

Absorbing States – Gambler’s Ruin Suppose that we start with $2 and want to calculate the probability of going broke, i.e., of being absorbed in state 0. We know p00 = 1 and p40 = 0, thus q20 = p20 + p21 q10 + p22 q20 + p23 q30 (+ p24 q40) q10 = p10 + p11 q10 + p12 q20 + p13 q30 + 0 q30 = p30 + p31 q10 + p32 q20 + p33 q30 + 0 where P = 0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0 2 0 1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1

Solution to Gambler’s Ruin Example Now we have three equations with three unknowns. Using p = 0.75 (probability of winning a single bet) we have q20 = 0 + 0.25 q10 + 0.75 q30 q10 = 0.25 + 0.75 q20 q30 = 0 + 0.25 q20 Solving yields q10 = 0.325, q20 = 0.1, q30 = 0.025 (This is consistent with the values found earlier.)

What You Should Know About The Mathematics of DTMCs How to classify states. What an ergodic process is. How to perform economic analysis. How to compute first-time passages. How to compute absorbing probabilities.