1. Markov Processes
Markov Chains
1/16/2019 rd

2. Lagrange Multipliers
You need 100 units made at 2 plants with the total cost function C = 0.1x2 + 7x + 15y How should you divide the units between plants x and y to minimize the cost? L(x,y,) = 0.1x2 + 7x + 15y (x + y – 100) Lx = 0.2x  = 0 Ly = 15 +  = 0 =>  = -15 and 0.2x + 7 = 15 or x = 40; y = 60. (solve '(( )( )( )))  ( ) C(40, 60) = $2349 min. See that Lxx and Lyy are > 0; D > 0
1/16/2019 rd

3. Examples
System States Transition p's Market share Brand A vs. Brand B Switch chances TV Market Proportion watching 2 Switch chances Rental returns to various locations different returns Machine breakdowns % not running p of running
1/16/2019 rd

4. Markov Process
Describes a dynamic situation and predicts the movement of a system along different states as time passes. bus F car From To T bus [ ¼ ¾ ] column vector car ( ¼ ¾ ) row vector Presently initial state vector is a for (bus, car) Tij = P(next state is i | current state is j Tbus car = P(riding bus | after riding car) = 0.4e
1/16/2019 rd

5. Markov Assumptions
Finite number of states (none absorbing) State in any period depends only on previous State Transition probabilities are constant over time Changes occur only once during period Transition periods occur with regularity Initial state a0 for example is [ ] sum to 1 T T2 T3 T4 … are the powers of T and approach the fixed point. State Transition Matrix States Column vectors are probability vectors Tij = P(next state is i | current state is j); T21 = 0.4
1/16/2019 rd

6. Markov States
Transient- a state when entered, can never return to this state when out and about. Recurrent - a state when entered, process will definitely return to this state. Absorbing - when entered, will never leave again. States are either recurrent or transient.
1/16/2019 rd

7. Fixed Point
A B if Tx = x fixed point A then Tx = x is also a fixed point B
0.2a + 0.3b = a or 0.8a + 0.7b = b; or 0.8a – 0.3b = a + b = a + b = 1
(solve '((4/5 -3/10 0)(1 1 1)))  (3/11, 8/11)
Steady state solution which is the Limit of the Matrix as n  
(setf matT #2A(( )( )))
(setf fm (expt-matrix matT 10)  #2A(( )( ))
(M* fm #2A((1)(0)))  #2A(( )( ))
Multiply the limit matrix by any probability vector results in fixed vector.
1/16/2019 rd

8. Eigenvalues Given matrix ¼ ½ 3/4x – 1/2y = 0 ¾ ½
¾ ½
TX = tX => (tI – T)X = 0
(quadratic 1 -3/4 -1/4) (1 -1/4) are the eigenvalues
(eigenvalues #2A((1/4 1/2)(3/4 1/2)))  ( )
Eigenvectors are (2, 3) and (1, -1)
(M* #2a((1/4 1/2)(3/4 1/2)) #2A((2)(3)))  #2A((2)(3))
(M* #2a((1/4 1/2)(3/4 1/2)) #2A((1)(-1)))  #2A((-1/4)(1/4))
(expt-matrix #2A((0.25 1/2))(3/4 1/2)) 30)  #2A(( )( ))
(M* (inverse #2A((2 1)(3 -1))) #2A((1/4 1/2)(3/4 1/2)) #2A((2 1)(3 -1))) 
#2A((1 0)(0 -1/4))
(M* (inverse #2A((2 1)(3 -1))) #2A((1/4 1/2)(3/4 1/2)) #2A((2 1)(3 -1)))
#2A((1 0)(0 -1/4))
1/16/2019 rd

9. Eigenvalues
A = 1 4 Ax = tx => t t-3 = t2 -4t -5 = 0; (5, -1)
Eigenvectors: 4x - 4y = 0 or x = y [1 1] x - 4y = 0 or x = 2y [2 -1]
(M* #2a((1 4)(2 3)) #2a((1)(1)))  #2A((5)(5)) ~ (1, 1)
(M* #2a((1 4)(2 3)) #2a((2)(-1)))  #2A((-2)(1))
Diagonalize
(M* (inverse #2a((1 2)(1 -1))) #2a((1 4)(2 3)) #2a((1 2)(1 -1)))
#2A((5 0)(0 -1))
Give another value in lieu of [2 -1].
1/16/2019 rd

10. Toothpaste
G C G owns 60% (a0 = ] of market and T = in that 88% of G's customers remain loyal and 15% of those using the competitors brands switch to G. (eigenvalues #2A(( )( )))  (0.73 1) (expt-matrix #2A(( )( )) 50)  (5/9, 4/9) (M* (inverse #2A(( )( ))) #2A(( ) ( )) #2A(( )( )))  #2A((0.73 0)(0 1.0)) ( ) ( ) is eigenvector; 15/27 =  #2A(( )( )) a0 = ( ). a1 = Ta0 ; an = Tna0 Note that dominant eigenvalue is 1 and determines the stability eigenvector. The dominant eigenvalue is always 1 for a transition matrix
(M* (inverse #2A(( )( ))) #2A(( )( )) #2A(( ) ( )))
Solve 0.12a- 0.15b = 0 and a+ b = 1 => to get (5/9 4/9)
1/16/2019 rd

11. Degenerate Discrete-Parameter Stochastic Process
T= Solve to get a = b = ½ but Tn = Process is strictly stationary and time-independent. Consider T = ½ ½ a + b = 2a ½ ½ a + b = 1 => (a, b) = (½, ½) Consider T = Tn = 1/3 2/3 1a + 2b = 3a => a = b 2/3 1/3 1a + 1b = 1 or (½, ½)
1/16/2019 rd

12. Stochastic Matrix
A matrix is stochastic if its columns (rows) are probability vectors and is called a regular matrix if some power of the matrix has positive vector components . ½ ¼ is regular ½ ¾ Regular matrices have a unique fixed probability vector with positive components and The powers of the regular matrices approach the fixed point. If A and B are stochastic, then the product AB is also.
1/16/2019 rd

13. Stochastic and Regular Matrices
Fixed points of square matrices Let A = |2 2| and u = [2, -1] |1 3| Then |2 2|| 2 |= | 2| |1 3||-1 | |-1| Also A(tu) = tAu => tu or scalar multiples are fixed points. Matrix is ergodic if limit exists; regular if any power has plus entries (expt-matrix #2A((2 2)(1 3)) 20) NOT ergodic,     - 3 yields 2 - 5  + 4 = 0; with eigenvalues = -1, -4; (expt-matrix #2A(( )( )) 20)  #2A(( )( )) ergodic  - ¼ -¾ -¼  - ¾ yields 2 -  = 0;  = 0, 1 (solve '((1 1 1)( )))  ( ) = (2/7, 5/7)
1/16/2019 rd

14. Stochastic and Ergodic
Consider matrix A and Ax = x where x = (a b) a/2 + 2b/3 = a 3a - 4b = 0 1a +1b = 1 (solve '((3 -4 0)(1 1 1)))  (4/7 3/7) (expt-matrix #2A (( )( )) 20) Limit = [4/7 3/7] (M* #2A((1/2 2/3)(0.5 1/3)) #2A((4/7)(3/7))) (4/7 3/7) (M* (expt-matrix #2A((1/2 2/3)(0.5 1/3)) 20) #2A((1/9)(8/9)))  #2A(( )( )) All probability vectors maps to the fixed vector (4/7, 3/7), the eigenvector of eigenvalue 1. (EIGENVALUES = (1 -1/6)
1/16/2019 rd

15. Ergodic
Is matrix ergodic? Does it have a limit? Is it Regular? Does any power contain only positive values? (expt-matrix #2A((0 0.4)(1 0.6)) 2)  Yes Find the limit by writing 0.4b = a; a + b = 1 (solve '((-1 2/5 0)(1 1 1)))  (2/7 5/7)
1/16/2019 rd

16. Fixed Points and Regular Stochastic Matrices
If matrix T is regular stochastic, then T has a unique fixed probability vector with positive components The sequence T, T2, T3 … approach a matrix whose columns are the fixed point. If p is any probability vector, the sequence of vectors Tp, T2p, T3p …approach the fixed point. The fixed point of imply b/2 = a or b = 2a 2a – 1b = 0 1a + 1b = 1 (1/3 2/3) is fixed point
1/16/2019 rd

17. 33. Markov Processes Example 1 Page 448
20% of the residents in region1 move to region 2, and 10% move to region 3. Of the
residents in region 2, 10% move to region 1 and 10% to region 3. In region 3, 20%
move to region 1 and 10% to region 2. Find the probability that a resident in region
1 this year is a resident of region 1 next year; T in 2 years. T2
Given state vector a0 = [ ], find the distribution of the population 3 years
from now. T3a0. Find the steady state vector. #2A((0.3125)(0.4375)(0.25))
a1 = Ta0 … , an = Tna0
From Region To
(setf demo #2A(( )( )( ))) (M* demo demo)
(M* (expt-matrix demo 3) #2A((0.4)(0.3)(0.3))) 
#2A((0.3368)(0.4024)(0.2608)
"Steady State Probabilities"
0.3125
0.4375
0.25
"Equilibrium 1st Passage Times"
0.35
0.382
0.268
"Transition Matrix after 3 Transitions"
Transition Matrix after 3 Transitions
Market Shares: ((0.3368) (0.4024) (0.2608))
1/16/2019 rd

18. Continued Given T = ½ ¼ a/2 + b/4 = a or 2a + b = 4a; 2a – b = 0
find the probability of going from state 2 to state 1 after 2 steps (a12)2 Find the steady state probabilities.
(print-matrix (expt-matrix #2A((1/2 1/4)(1/2 3/4)) 2))  3/ /16
5/8 11/16
(expt-matrix #2A((1/2 1/4)(1/2 3/4)) 30)  / /3
/ /3
1/16/2019 rd

19. 453-31 Flu transition matrix flu no flu
flu for 200 dorm students
no flu
120 students have the flu now, how many will have it 8 days from now? T21 in 8 days? 12 days from now? Assume a 4-day flu cycle
Given 120 have the flu; 120/200 = ao = 3/5 2/5
T2 a0
(prn-mat (M* (expt-matrix #2a((1/10 2/10)(9/10 8/10)) 2) #2A((3/5)(2/5)))  93/500
407/500 => (* /500)  37.2 will get the flu
T3 a0
(prn-mat (M* (expt-matrix #2a((1/10 2/10)(9/10 8/10)) 3) #2A((3/5)(2/5)))
 #2A((907/5000)(4093/5000))
(* /5000)  will get the flu
1/16/2019 rd

20. Steady State Vector
35. Find the unique probability vector of the regular stochastic matrix. Start with a0 = [a, b, 1 - a – b]
=> ½ a ½ b a - b
 ½ - a/2 – b/2 = a 3a + b = 1
a + ½ - a/2 - b/2 = b a – 3b = -1
b = 1 – a – b a + 2b = 1
(solve '((3 1 1)( )(1 2 1)))  (1/5, 2/5, 2/5)
(Expt-Matrix #2A((0 0 1/2)(1 0 1/2)(0 1 0)) 30)  ( )
1/16/2019 rd

21. Flu Steady State Vector
Find the steady state flu vector given transition matrix flu no flu flu a 0.1a + 0.2b = a no flu b 0.9a + 0.8b = b a + b = 1 0.9a = 0.2b => 0.9a = 0.2(1 – a) => a = 0.2/1.1 = b = (expt-matrix #2a(( )( )) 30)  ( )
1/16/2019 rd

22. 2W R W 1R W 2R R W 1R
a b a b a b
a b c
#2A((0 1/6 0)(1 1/2 2/3)(0 1/3 1/3))
a b c a / Ex: b  b: 1/2 * 1/3 + 1/2 * 2/3 = 1/ b /2 2/ b  c: 1 * 1/3
c /3 1/ (0.1, 0.6, 0.3) final states
Compute probability of 2R marbles in Urn A after 3 steps and 2 in urn A after steady state (0.3). a2 = T3p0 = (1/12 23/36 5/18); 0.3
In urn A are 2 White marbles and in urn B there are 3 Red marbles. At each step of the process a marble is randomly selected from each urn and switched. The states are a0 = 0 red in urn a => state a0 = [1 0 0]
States: ai is number of red (state) in urn a
#2A(( )( )( ))
(M* (expt-matrix amat 3) p0)
#2A((1/12)(23/36)(5/18))
1/16/2019 rd

23. Other Notations Find given a0 = (2/3, 0, 1/3) 1/4 1/4 1/2 3/4 1/2 1/2
(prn-mat (expt-matrix #2A((0 1/2 0)(1/2 1/2 1)(1/2 0 0)) 2)) 
1/ / /2
3/ / /2 /
(mmult #2A((1/4 1/4 0)(1/4 1/2 0)(0 1/4 0))
#2A((2/3)(0)(1/3)))
1/16/2019 rd

24. Absorbing Barriers
A player has $2. He bets $1 at a time and wins with probability ½. He stops playing if he loses the $2 or wins $4. Compute probability that he will loose his money at the end of at most 5 plays. Find probability that the game will last more than 7 plays. p0 = (0, 0, 1, ) $2 State ai has i dollars. Player is the T matrix.
a0 a1 a2 a3 a4 a5 a6
a0 1 ½
a ½
a2 0 ½ ½
a ½ ½
a ½ 0 ½ 0
a ½
a ½ 1
Random walk with absorbing barriers at 0 (lose $2) and 6 (win $4).
(M* (expt-matrix player 5) #2A((0) (0)(1)(0)(0)(0)(0)))  #2A((3/8)(5/32)(0)(9/32)(0)(1/8)(1/16))
(M* (expt-matrix player 7) #2A((0) (0)(1)(0)(0)(0)(0))) 
2A((29/64)(7/64)(0)(27/128)(0)(13/128)(1/8))
(sum (flatten '(7/64)(0)(27/128)(0)(13/128)))  27/64
(setf player #2a((1 1/ )(0 0 1/ )(0 1/2 0 1/ )(0 0 1/2 0 1/2 0 0)( /2 0 1/2 0)( /2 0 0)( /2 1)))
1/16/2019 rd

25. Exercise 1
Given P = and the system is in state 2 [0 1] a) Find state vector X2 , the probability of being in state 2 after 2 steps. (setf Y #2A(( )( ))) IV #2A((0)(1))) (M* Y IV)#2A((0.6)(0.4)) (M* Y Y IV)  #2A((0.36)(0.64)) (M* Y #2A((0.6)(0.4)))  #2A((0.36)(0.64)) b) Find (X12)3 probability of going from state 2 to state1 in 3 steps. (expt-matrix #2A (( )( )) 3)  #2A(( )( )) c) Find the equilibrium (steady state) vector (limit L). #2A(( )( )) d) Confirm using Px = x and solve where x = (a, b).
1/16/2019 rd

26. Exercise 2
Cars are classified as good fair or poor with transition matrix
G F P
G F P
Assume 100 cars are Good, 60 Fair and 20 Poor and weekly updated.
Given a0 = [5/9 3/9 1/9]
a) How many cars will be found in each condition next week?
(M* #2A(( )( )( )) #2A((5/9)(3/9)(1/9)))
#2A(( )( )( )) times 180 for weekly condition.
b) Repeat for steady state conditions.
#2A(( )( )( ))
(mapcar #' * '( ) '( )) 
( )
1/16/2019 rd

27. Which to Lease?
Two computer operating condition (OC) and not operating (NO) are given as O NO O NO O NO Which one should be leased? (solve '(( )(1 1 1)))  ( ) (solve '(( )(1 1 1)))  ( )
1/16/2019 rd

28. State Transition Matrix
Compute the transition matrix given the diagram below. 1 2
½ c 1 3 4 1
1/16/2019 rd

29. Absorbing States
Once there, cannot exit. Bankrupt, home destroyed by fire, lake irreversibly polluted, Goal reached, game over, death
1/16/2019 rd

30. Recall: an = Tna0 where T = 0.9 0.2
At a university, 60% of all first-semester students buy their books at the Campus Bookstore, and 40% buy their books at the Off-Campus Bookstore. See transition matrix below. If these trends continue, what percent of the market will each bookstore have at the beginning of the second semester? At the beginning of the third semester? The tenth semester?
Recall: an = Tna0 where T =
and a0 = [ ] . We found that a1 = 0.62
0.38
a2 = T=
0.366
a9 = (solve '(( )(1 1 1)))  (2/3 1/3)
0.3360
What is the distribution in later semesters? At some point, will the distribution stay the same from one semester to the next? Yes, because T is regular.
(setf a0 #2a((0.6)(0.4)) Tmat #2A(( )( )))
1/16/2019
rd‹#›

31. Students at an University
Freshmen Dropouts Sophomores Graduated I Identity Matrix O Null Matrix A Absorbing N Non-absorbing D G F S D G F S
0.6
0.1 F 1 D
0.3
0.05 S
0.55 G 1
0.4 e
e e-13
1/16/2019
rd‹#›

32. Students at an University
Abs Nabs Identity Zero A Nabsorb Abs NAbs The fundamental matrix is F = (I – N)-1 time F S Conditional Prob = FA F S
(inverse #2a((9/10 -3/10)(-2/10 1/2))) --> #2A((50/39 20/39)(10/13 30/13))
1/16/2019 rd

33. Accounts Receivable Analysis
Paid Bad Debt 0-30 days days Paid Bad Debt days days
#2A(( )( ))
Probability of 0-30 days being paid is 89% and 11% bad debt , 74% being paid.
1/16/2019 rd

34. Application: 3 products
A B C
(expt-matrix #2A(( )( )( )) 10) 
#2A(( ) A + B = = 58.3
It costs $150K to promote A only; 280K to promote B only;
Each percentage point is worth $10K
(expt-matrix #2A(( )( )( )) 50) 
#2A(( )) A + B = 77.8, a 19.5 * 10K = $195K 150K => $45K profit
(expt-matrix #2A(( )( )( )) 50) 
#2A(( )) A + B =  $322K 280K => $42K
#2A(( )( )( ))
1/16/2019 rd

35. Markov
From To A B C A B C a) Compute steady state probabilities. (expt-matrix #2A(( )( )( )) 20) #2A(( )( )( )) b) Find market share for C
1/16/2019 rd

36. a) Identify the absorbing states.
FROM
TO Graduate Drop-Out Freshman Sophomore Junior Senior Graduate
Dropout Freshman Sophomore Junior Senior
a) Identify the absorbing states.
b) Interpret transition probabilities for a sophomore.
c) Compute probabilities that a sophomore will graduate; will drop out.
d) Compute steady state probabilities.
e) Currently there are 600 freshmen, 520 sophomores, 460 juniors, 420 seniors.
What percentage of the 2000 students will eventually graduate?
(M* (expt-matrix #2A(( )( )( )( )( )( )) 20) #2a((0)(0)(600)(520)( 460)( 420)))  #2A(( )( )( e-14)( e-13)( e-12)( e-11))
1/16/2019 rd

37. a) Find probabilities of ending in an absorbing state.
Pat walks along a four-blocks of an avenue. If at corner 1, 2, or 3, Pat walks to the left or right with probability ½. When at corner 4 or corner 0, Pat stays.
0 1 ½ ½
2 0 ½ 0 ½ 0
½ 0 0
½ 1
a) Find probabilities of ending in an absorbing state.
b) On average, how long for the process to be absorbed?
c) On average, how many times will the process be in each transient state?
1/16/2019
rd‹#›

38. States of an Ecosystem
Find the states for the year given strontium 90 compartments a0 = [ ] run-off
leaching 0.05
Grasses
Soil
growth 0.01
Dead Organic matter
Streams
1/16/2019 rd

39. Accounts receivable
p b p = paid; 1 = 1-3 days overdue p b = bad debt; 2 = days b I R absorbing O Q non absorbing
I – Q = (I – Q)-1 = F = #2A(( )( ))
RF = (M* #2A ((.5 .3)(0 .3))#2A(( )( ))) 
#2A(( ) If $10K in 1 and $6K in 2 then ( ))
(M* #2A(( )( )) #2a((10e3)(6e3)))
#2A(( )( )) => (expected paid, expected bad debt) = $16K `
1/16/2019 rd