1. INDR 343 Problem Session 1

2. 16.2-3
Consider the second version of stock market model presented as an example. Whether the stock goes up tomorrow depends upon whether it increased today and yesterday.
If the stock increased today and yesterday, it will increase tomorrow with probability α1.
If the stock increased today and decreased yesterday, it will increase tomorrow with probability α2.
If the stock decreased today and increased yesterday, it will increase tomorrow with probability α3.
If the stock decreased today and yesterday, it will increase tomorrow with probability α4.

3. 16.2-3
(a) Construct the (one-step) transition matrix of the Markov chain.
(b) Explain why the states used for this Markov chain cause the mathematical definition of the Markovian property to hold even though what happens in the future (tomorrow) depends upon what happened in the past (yesterday) as well as the present (today).

4. 16.3-2
Suppose that a communications network transmits binary digits, 0 or 1, where each digit is transmitted 10 times in succession.
During each transmission, the probability is that the digit entered will be transmitted accurately. In other words, the probability is that the digit being transmitted will be recorded with the opposite value at the end of the transmission.
For each transmission after the first one, the digit entered for transmission is the one that was recorded at the end of the preceding transmission. If X0 denotes the binary digit entering the system, X1 the binary digit recorded after the first transmission, X2 the binary digit recorded after the second transmission, , then {Xn} is a Markov chain.

5. 16.3-2
(a) Construct the (one-step) transition matrix. (b) Use your OR Courseware to find the 10- step transition matrix P(10). Use this result to identify the probability that a digit entering the network will be recorded accurately after the last transmission.

6. Ex: Unconditional Probabilities
An urn always contains 2 balls. Ball colors are red and blue. At each stage a ball is randomly chosen and then replaced by a new ball, which with probability 0.8 is the same color, and with probability 0.2 is the opposite color, as the ball it replaces. If initially both balls are red, find the probability that the fifth ball selected is red.

7. Classification of States
Accessible: Possible to go from state i to state j (path exists in the network from i to j).
Two states communicate if both are accessible from each other. A system is irreducible if all states communicate.
State i is recurrent if the system will return to it after leaving some time in the future.
If a state is not recurrent, it is transient.

8. Classification of States (continued)
A state is periodic if it can only return to itself after a fixed number of transitions greater than 1 (or multiple of a fixed number).
A state that is not periodic is aperiodic.
a. Each state visited every 3 iterations
b. Each state visited in multiples of 3 iterations

9. Classification of States (continued)
An absorbing state is one that locks in the system once it enters.
This diagram might represent the wealth of a gambler who begins with $2 and makes a series of wagers for $1 each.
Let ai be the event of winning in state i and di the event of losing in state i.
There are two absorbing states: 0 and 4.

10. Classification of States (continued)
Class: set of states that communicate with each other.
A class is either all recurrent or all transient..
State i is ergodic if it is recurrent and aperiodic.
A Markov chain is ergodic if all of its states are ergodic.

11. Illustration of Concepts
Example 1
Every pair of states communicates forming a single recurrent class; moreover, the states are not periodic.
Thus the stochastic process is aperiodic and irreducible.

12. Illustration of Concepts
Example 2
States 0 and 1 communicate and form a recurrent class.
States 3 and 4 form separate transient classes.
State 2 is an absorbing state and forms a recurrent class.

13. Illustration of Concepts
Example 3
Every state communicates with every other state, so we have an irreducible stochastic process.
Periodic?
Yes, so Markov chain is irreducible and periodic.

14. Classification of States
Example .6 .7 1 2 .4 3 4 .5 .4 .5 .5 .3 .8 .1 5 .2

15. 16.4-2
Given each of the following (one-step) transition matrices of a Markov chain, determine the classes of the Markov chain and whether they are recurrent.

16. 16.4-3

17. 16.4-5
Consider the Markov chain that has the following (one step) transition matrix.
(a) Determine the classes of this Markov chain and, for each class, determine whether it is recurrent or transient.
(b) Determine the periods of each state.