1. Introduction to Concepts Markov Chains and Processes

2. Why? Core Statistical Processes Appear in Nature
Exemplify Performance Measurement
Discrete and Continuous Time Versions
Fun!

3. DEFINITION {Xn, n>=0} hops around on statepace
{…-2, -1, 0, 1, 2, …} according to
probability Transition Matrix P:
Pi,j = P[Xn+1 = j | Xn = i]
ai = Prob[X0 = i]

4. DEFINITION X’s state completely characterized by its current state
X is “Markov”
P is a “stochastic matrix”
its rows sum to 1

5. EXAMPLE state 0: sunny state 1: rainy.7 .8 .3 1 .2
Tomorrow’s weather depends ONLY on today’s weather.

6. EXAMPLE FROG ON THE ROAD
p = Prob [jump fwd]
q = Prob [jump back]
infinite state space
Prob[X11 = 5]?
Queuing!
state = number customers in system
p = Prob [next event is arrival]
q = Prob [next event is service compl.]

7. MARKOV CHAINS IN NATURE
Neurology
Marketing
Gambling
Inventory
Manpower
Electronic Support Measures
Communications
Service Models
Air-to-Air Combat

8. Chapman-Kolmogorov Equation
For any r < n
Upshot: Pn = n-transition probability matrix
N is the number of states in the statespace.

9. LIMITING DISTRIBUTION
pi exists implies process is aperiodic

10. LIMITING DISTRIBUTION
...completely useless but interesting
better
think about it...

14. CONTINUOUS TIME MARKOV CHAINS
{X(t), t >= 0} is a continuous time process with
> sojourn times S0, S1, S2, ...
> embedded process Xn = X(Sn-1+)
X is a CTMC if Sn ~ Exp(qi) where i=Xn

15. MATRICES
Probability Transition Matrix

16. GENERATOR MATRIX GIVES RISE TO THE NUMERICAL METHODS INVOLVING RAISING
A MATRIX TO A POWER
-- ROW SUMS EQUAL ZERO
-- DIAGONAL-DOMINATE

17. M/M/1 QUEUE l = rate of arrival (# per unit time)
m = rate of service (1/m = avg serve time)

18. FIRST PASSAGE TIME Want to know how soon X(t) gets to a special state:
mi = E[min t: X(t) is “special”|X(0) = i]

19. LIMITING DISTRIBUTION
Corollary of the General Key Renewal Theorem