1. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 1 of 60 Chapter 8 Markov Processes

2. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 2 of 60 Outline 8.1 The Transition Matrix 8.2 Regular Stochastic Matrices 8.3 Absorbing Stochastic Matrices

3. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 3 of 60 8.1 The Transition Matrix 1.Markov Process 2.States 3.Transition Matrix 4.Stochastic Matrix 5.Distribution Matrix 6.Distribution Matrix for n 7.Interpretation of the Entries of A n

4. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 4 of 60 Markov Process Suppose that we perform, one after the other, a sequence of experiments that have the same set of outcomes. If the probabilities of the various outcomes of the current experiment depend (at most) on the outcome of the preceding experiment, then we call the sequence a Markov process.

5. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 5 of 60 Example Markov Process A particular utility stock is very stable and, in the short run, the probability that it increases or decreases in price depends only on the result of the preceding day's trading. The price of the stock is observed at 4 P.M. each day and is recorded as "increased," "decreased," or "unchanged." The sequence of observations forms a Markov process.

6. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 6 of 60 States The experiments of a Markov process are performed at regular time intervals and have the same set of outcomes. These outcomes are called states, and the outcome of the current experiment is referred to as the current state of the process. The states are represented as column matrices.

7. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 7 of 60 Transition Matrix The transition matrix records all data about transitions from one state to the other. The form of a general transition matrix is.

8. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 8 of 60 Stochastic Matrix A stochastic matrix is any square matrix that satisfies the following two properties: 1. All entries are greater than or equal to 0; 2. The sum of the entries in each column is 1. All transition matrices are stochastic matrices.

9. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 9 of 60 Example Transition Matrix For the utility stock of the previous example, if the stock increases one day, the probability that on the next day it increases is.3, remains unchanged.2 and decreases.5. If the stock is unchanged one day, the probability that on the next day it increases is.6, remains unchanged.1, and decreases.3. If the stock decreases one day, the probability that it increases the next day is.3, is unchanged.4, decreases.3. Find the transition matrix.

10. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 10 of 60 Example Transition Matrix (2) The Markov process has three states: "increases," "unchanged," and "decreases." The transitions from the first state ("increases") to the other states are

11. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 11 of 60 Example Transition Matrix (3) The transitions from the other two states are

12. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 12 of 60 Example Transition Matrix (4) Putting this information into a single matrix so that each column of the matrix records the information about transitions from one particular state is the transition matrix.

13. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 13 of 60 Distribution Matrix The matrix that represents a particular state is called a distribution matrix. Whenever a Markov process applies to a group with members in r possible states, a distribution matrix for n is a column matrix whose entries give the percentages of members in each of the r states after n time periods.

14. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 14 of 60 Let A be the transition matrix for a Markov process with initial distribution matrix then the distribution matrix after n time periods is given by Distribution Matrix for n

15. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 15 of 60 Example Distribution Matrix for n Census studies from the 1960s reveal that in the US 80% of the daughters of working women also work and that 30% of daughters of nonworking women work. Assume that this trend remains unchanged from one generation to the next. If 40% of women worked in 1960, determine the percentage of working women in each of the next two generations.

16. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 16 of 60 Example Distribution Matrix for n (2) There are two states, "work" and "don't work." The first column of the transition matrix corresponds to transitions from "work". The probability that a daughter from this state "works" is.8 and "doesn't work" is 1 -.8 =.2. Similarly, the daughter from the "don't work" state "works" with probability.3 and "doesn't work" with probability.7.

17. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 17 of 60 Example Distribution Matrix for n (3) The transition matrix is The initial distribution is.

18. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 18 of 60 Example Distribution Matrix for n (4) In one generation, So 50% women work and 50% don't work. For the second generation, So 55% women work and 45% don't work.

19. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 19 of 60 The entry in the i th row and j th column of the matrix A n is the probability of the transition from state j to state i after n periods. Interpretation of the Entries of A n

20. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 20 of 60 Example Interpretation of the Entries Interpret from the last example. If a woman works, the probability that her granddaughter will work is.7 and not work is.3. If a woman does not work, the probability that her granddaughter will work is.45 and not work is.55.

21. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 21 of 60  A Markov process is a sequence of experiments performed at regular time intervals involving states. As a result of each experiment, transitions between states occur with probabilities given by a matrix called the transition matrix. The ij th entry in the transition matrix is the conditional probability Pr(moving to state i|in state j). Summary Section 8.1 - Part 1

22. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 22 of 60  A stochastic matrix is a square matrix for which every entry is greater than or equal to 0 and the sum of the entries in each column is 1. Every transition matrix is a stochastic matrix.  The n th distribution matrix gives the percentage of members in each state after n time periods. Summary Section 8.1 - Part 2

23. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 23 of 60  A n is obtained by multiplying together n copies of A. Its ij th entry is the conditional probability Pr(moving to state i after n time periods | in state j). Also, A n times the initial distribution matrix gives the n th distribution matrix. Summary Section 8.1 - Part 3

24. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 24 of 60 8.2 Regular Stochastic Matrices 1.Regular Stochastic Matrix 2.Stable Matrix and Distribution 3.Properties of Regular Stochastic Matrix

25. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 25 of 60 Regular Stochastic Matrix A stochastic matrix is said to be regular if some power has all positive entries.

26. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 26 of 60 Example Regular Stochastic Matrix Which of the following stochastic matrices are regular? a) All entries are positive so the matrix is regular.

27. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 27 of 60 Example Regular Stochastic Matrix (2) All entries of the square are positive so the matrix is regular. All powers will be one of the above two matrices so the original matrix is not regular.

28. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 28 of 60 Stable Matrix and Distribution If a stochastic matrix A has the properties that 1. as n gets large, A n approaches a fixed matrix, and 2. any initial distribution approaches a fixed distribution for large n, then the fixed matrix is called the stable matrix of A and the fixed distribution is called the stable distribution of A.

29. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 29 of 60 Example Stable Matrix and Distribution In Jordan, 25% of the women currently work. The effect of maternal influence of mothers on their daughters is given by the matrix Find the stable matrix and the stable distribution of A.

30. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 30 of 60 Example Stable Matrix and Distribution (2) It appears that the powers are approaching which is the stable matrix.

31. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 31 of 60 Example Stable Matrix and Distribution (3) For the initial distribution, However, for any initial distribution, which is the stable distribution.

32. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 32 of 60 Properties of Regular Stochastic Matrix Let A be a regular stochastic matrix. 1. The powers A n approach a certain matrix as n gets large. This limiting matrix is called the stable matrix of A. 2. For any initial distribution [ ] 0, A n [ ] 0 approaches a certain distribution. This limiting distribution is called the stable distribution of A.

33. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 33 of 60 Properties of Regular Stochastic Matrix (2) 3. All columns of the stable matrix are the same; they equal the stable distribution. 4. The stable distribution X = [ ] can be determined by solving the system of linear equations

34. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 34 of 60 Example Properties Regular Matrices Use the properties of a regular stochastic matrix to find the stable matrix and stable distribution of

35. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 35 of 60 Example Properties Regular Matrices (2) Solve This gives

36. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 36 of 60 Example Properties Regular Matrices (3) The last equation is (-1) times the second equation so we can solve just the first two equations.

37. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 37 of 60 Example Properties Regular Matrices (4) Therefore, the stable distribution and stable matrix, respectively, are

38. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 38 of 60  A stochastic matrix is called regular if some power of the matrix has only positive entries. Summary Section 8.2 - Part 1

39. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 39 of 60  If A is a regular stochastic matrix, as n gets large the powers of the matrix, A n, approach a certain matrix called the stable matrix of A and the distribution matrices approach a certain column matrix called the stable distribution. Each column of the stable matrix holds the stable distribution. The stable distribution can be found by solving AX = X, where the sum of the entries in X is equal to 1. Summary Section 8.2 - Part 2

40. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 40 of 60 8.3 Absorbing Stochastic Matrices 1.Absorbing State 2.Absorbing Stochastic Matrix 3.Arranging States in an Absorbing Matrix 4.Properties of Absorbing Matrix 5.Stable Matrix of Absorbing Matrix 6.Fundamental Matrix

41. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 41 of 60 Absorbing State A state for which all objects that start in that state, stay in that state is called an absorbing state. That is, an absorbing state is a state that always leads back to itself. A state is absorbing if 1. the corresponding column has a single 1 and the remaining entries are 0, and 2. the 1 must be located on the main diagonal of the matrix.

42. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 42 of 60 Absorbing Stochastic Matrix An absorbing stochastic matrix is a stochastic matrix in which 1. there is at least one absorbing state, and 2. from any state it is possible to get to at least one absorbing state, either directly or through one or more intermediate states.

43. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 43 of 60 Example Absorbing Stochastic Matrix For the given stochastic matrix determine the absorbing states, if any, and whether the matrix is an absorbing stochastic matrix.

44. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 44 of 60 Example Absorbing Stochastic Matrix (2) States 1 and 2 are absorbing because the two columns have a single 1 and that 1 appears on the diagonal. State 4 is not absorbing because, although it has a single 1, it is not on the diagonal. State 3 does not have a single 1.

45. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 45 of 60 Example Absorbing Stochastic Matrix (3) Objects in state 3 can lead to both state 1 with probability.3 and state 2 with probability.1. Objects in state 4 will lead to state 2 with probability 1. Therefore, the matrix is an absorbing stochastic matrix.

46. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 46 of 60 Arranging States in an Absorbing Matrix When considering an absorbing stochastic matrix, we will always arrange the states so that the absorbing states come first, then the nonabsorbing states.

47. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 47 of 60 Example Arranging States Identify R and S for the given absorbing stochastic matrix.

48. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 48 of 60 Example Arranging States The absorbing states are already written first. Identity matrix Zero matrix

49. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 49 of 60 Stable Matrix of Absorbing Matrix Let an absorbing stochastic matrix be partitioned as then the stable matrix of A is where I is the same dimension as R in (I - R) -1.

50. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 50 of 60 Example Stable Matrix Find the stable matrix of

51. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 51 of 60 Example Stable Matrix (2)

52. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 52 of 60 Example Stable Matrix (3) The stable matrix is

53. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 53 of 60 Example Stable Matrix (4) There is no stable distribution matrix as the long term trend depends upon the initial matrix. For example:

54. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 54 of 60 Fundamental Matrix The matrix (I - R) -1 used to compute the stable matrix is called the fundamental matrix and is denoted by the letter F. The ij th entry of F is the expected number of times the process will be in nonabsorbing state i if it starts in nonabsorbing state j. The sum of the entries of the j th column of F is the expected number of steps before absorption when the process begins in nonabsorbing state j.

55. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 55 of 60 Example Fundamental Matrix Interpret the fundamental matrix from the previous example. Note: States 3 and 4 were the nonabsorbing states.

56. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 56 of 60 Example Fundamental Matrix If an object starts in state 3, it is expected to be in state 3 for 2 steps and in state 4 for.2 steps and reach an absorbing state (1 or 2) in 2 +.2 = 2.2 steps. If the object starts in state 4, it will never reach state 3 and expects to be in state 4 for 1 step and reach an absorbing state in 1 step.

57. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 57 of 60 Summary Section 8.3 - Part 1  If the probability of moving from a state to itself is 1, we call that state an absorbing state. An absorbing stochastic matrix is a stochastic matrix with at least one absorbing state and in which from any state it is possible to eventually get to an absorbing state.

58. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 58 of 60 Summary Section 8.3 - Part 2  In an absorbing process, the transition matrix should be arranged so that absorbing states are listed before nonabsorbing states. The transition matrix will have the form where I is an identity matrix, 0 denotes a matrix of zeros, and S and R represent the transitions from nonabsorbing to absorbing states and from nonabsorbing to nonabsorbing states, respectively.

59. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 59 of 60 Summary Section 8.3 - Part 3  The stable matrix of the absorbing matrix in the proceeding transition matrix is

60. Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e 60 of 60 Summary Section 8.3 - Part 4  The fundamental matrix of the absorbing matrix in the proceeding transition matrix is the matrix (I- R) -1. When its columns and rows are labeled with the nonabsorbing states, its ij th entry is the expected number of times the process will be in nonabsorbing state i given that it started in nonabsorbing state j. The sum of the entries in the j th column is the expected number of steps before absorption when the process begins in state j.