Markov Chains and Absorbing States My beard is a Markov Chain Andrey Markov 1856-1922 Nathan Hechtman

About Markov Russian Mathematician Helped prove the central limit theorem Specialized in stochastic processes and probability

Transition Diagrams: Bayesian Probability Maps Transition diagrams are conditional probability trees with one repeated process That process is expressed in a network of conditional probabilities emerging from states (nodes) 1.0 1.0

Transition Diagrams as Matrices 1 .2 .8 .1 .4 .5 1.0 1.0 The network corresponds to the matrix of transformation of the Markov chain Entry (i, j) is the probability from going from node i to node j in a single step

Simple Markov Chain: Initial state matrix, matrix of transformation, and power The final state matrix (matrix resulting after n steps) can be calculated very efficiently this way The matrix of transformation will be square 1 .2 .8 .1 .4 .5 n [C0 C1 C2 C3 C4 C5 C6 C7] Initial state matrix Matrix of transformation

Ergodic/Irreducible Chains Every node in the transition diagram leads to and from every other node with a nonzero probability It does so in a finite amount of steps, but not necessarily one step Ergodic chains correspond to bijective (and invertible) transformations Irreducible (ergodic) Irreducible (ergodic) Reducible (non-ergodic)

Periodic Markov Chains Periodic Markov chains repeat in cycles of length greater than one Periodic chains are a special case of ergodic Markov chains 2n 1 1 =

Regular Markov Chains and Steady State The MOT raised to some power of n has all positive entries Converge to a steady-state matrix Finding the steady-state matrix (v is the initial matrix, P is the MOT): .8 .2 .6 .4 [C1 C2] = [0 0] vP = v v(P-I) = v(I-P) = 0 .8-1 .2 .6 .4-1 [C1 C2] = [0 0] -.2 .2 .6 -.6 [.75 .25] = [0 0]

An Analogy for Absorbing States: Ford and the Bistro

Absorbing States The matrix of transformation contains a row of all zeros, signifying a node or nodes with no ‘children’ All nodes have a pathway to at least one absorbing state, which can be seen as a row with all zeros, except for a 1 along the diagonal Absorbing states exist iff chain is irreducible For example, S4 and S7 1 .2 .8 .1 .4 .5 1.0 1.0

The standard form of the transition matrix Absorbing Markov chain can be expressed with a standard form transition matrix Absorbing states, like S4 and S7 are moved to the top and left: Recall: absorbing states are seen as rows of zeros S0 S1 S2 S3 S4 S5 S6 S7 1 .2 .8 .1 .4 .5 S4 S7 S0 S1 S2 S3 S5 S6 1 .2 .8 .1 .4 .5 Transition Matrix Standard form Transition Matrix

The Standard Form continued Identity Zero Matrix R Q Four Parts: I, 0, R, Q Pk asymptotically approaches P , the limiting matrix S4 S7 S0 S1 S2 S3 S5 S6 1 .2 .8 .1 .4 .5 I, 0: no chance of leaving absorbing states R: probabilities of entering absorbing states Q: probabilities of entering other (pre-absorbing) states Are you absorbed, yet? Standard form transition matrix

F, the fundamental matrix F= (I-Q)-1 is known as the fundamental matrix S4 S7 S0 S1 S2 S3 S5 S6 1 .2 .8 .1 .4 .5 -1 1 -1 -.2 -.8 -.4 -.5 1 .2 .8 .32 .4 .5 F = = Q ( I-Q ) (I-Q) -1

Property of F: expected time before absorption (I-Q)-1 gives the expected number of periods before entering an absorbing state (any absorbing state) The sum of each row: 1 .2 .8 .32 .4 .5 3.74 2.74 1 1.9

P , the limiting matrix Finding FR Identity Zero Matrix FR P = S4 S7 1 .28 .72 .1 .9 1 .2 .8 .32 .4 .5 1 .1 .28 .72 1 .1 .9 = F R FR P

Interpreting P Entry (i, j) is the probability of going from state i to state j after an infinite number of steps Starting in state S0, there is a 72% chance of ending up in S7 Starting in state S2, there is a 100% chance of ending up in S4 There is no chance of ending up in a Non-absorbing state S4 S7 S0 S1 S2 S3 S5 S6 1 .28 .72 .1 .9 P

Sources MDPs: https://www.youtube.com/watch?v=i0o-ui1N35U https://www.youtube.com/watch?v=uvYTGEZQTEs Feller, William. An Introduction to Probability and Its Applications. Tokyo: C.E. Tuttle, 1957. 338-51. Print. Anderson, David. "Markov Chains." Interactive Markov Chains Lecture Notes in Computer Science (2002): 35-55. Web. Wilde, Joshua. “Linear algebra III: Eigenvalues and Markov Chains." Eigenvalues, Eigenvectors, and Diagonalizability (2002): 35-55. Web. http://www.avcsl.com/large-yellow-jumbo-sponge-bone-shape.html http://www.ssc.wisc.edu/~jmontgom/absorbingchains.pdf "Andrey Andreyevich Markov | Russian Mathematician." Encyclopedia Britannica Online. Encyclopedia Britannica, n.d. Web. 24 Nov. 2015.

In Soviet Russia, questions ask you!