1. stochastic processes(2)
Dr. Adil Yousif
Lecture 6

2. Cat and Mouse Five Boxes [1,2,3,4,5]
Cat starts in box 1, mouse starts in box 5
Each turn each animal can move left or right, (randomly)
If they occupy the same box, game over (for the mouse anyway)

3. 5 box cat and mouse game States: Stochastic Matrix:
State 1: cat in the first box, mouse in the third box: (1, 3)
State 2: cat in the first box, mouse in the fifth box: (1, 5)
State 3: cat in the second box, mouse in the fourth box: (2, 4)
State 4: cat in the third box, mouse in the fifth box: (3, 5)
State 5: the cat ate the mouse and the game ended: F.

4. State Diagram of cat and mouse
State two in green is the initial state
State five in blood red, is an accepting state
We can bounce around for a while, but eventually the mouse will die

5. Cat, mouse and cheese example
In this example, a mouse is randomly moving from room to room. The
cat and cheese do not move. But, if the mouse goes into the cat’s room, he never comes out.
If he reaches the cheese he also does not come out.

6. For example, whenever the mouse is in room 3 he will go next to room 2,4 or 5 with equal probability.
The mouse moves according to the transition probabilities p(i, j) = P(the mouse goes to room j when he is in room i).

7. the transition matrix:
Every row adds up to 1. This is because the mouse has to go somewhere or stay where he is.

8. (1) There are 5 states (the five rooms).
I numbered them: 1,2,3,4,5.
(2) The mouse moves in integer time, say every minute.
(3) The mouse does not remember which room he was in before.
Every minute he picks an adjacent room at random, possibly going back to the room he was just in.
(4) The probabilities do not change with time.

9. Frog Cell Cycle
Sible and Tyson figure 1 Methods

10. Frog Cell Cycle
Concentration or number of each of the molecule is a state.
Each reaction serves as a transition from state to state.
Whether or not a reaction will occur is Stochastic.
Each state contains a complete picture of every species' quantity

11. Markov? Andrey (Andrei) Andreyevich Markov Russian Mathematician
June 14, 1856 – July 20, 1922

12. Markov Chain Future is independent of the past given the present.
Want to know tomorrow’s weather? Don’t look at yesterday, look out the window.
Requires perfect knowledge of current state.
Very Simple, Very Powerful.
P(Future | Present)

13. Markov Chain
Make predictions about future events given probabilities based on the current state.
Probability of the future, given the present.
Transition from state to state

14. First Order Markov Chain
Make a Markov assumption that the value of the current state depends
only on a fixed number of previous states
In our case we are only looking back to one previous state
Xt only depends on Xt-1

15. Second Order Markov Chain
Value of the current state depends on the two previous states
P(Xt|Xt-1,Xt-2)
The math starts getting very complicated
Can expand to third fourth… Markov chains

16. Random Walk Model
A drunk man leaves a bar late Saturday night He doesn’t know where home is and supports himself from the light posts down Green Street He can only move from one light post to the next
Unfortunately, when he gets to the new light, he forgets where he came from
On average, where does this man wake up Sunday morning?

17. Features of a Random Walk
Memory loss
History reveals no information about the future
Expected change in value is zero
Over any length of time, the best predictor of future value is the current value
This feature is termed a martingale
Variance increases with time
As more time passes, there is potential for being farther from the initial value 6 6

18. Why Random Walks?
A random walk (RW) is a useful model in understanding stochastic processes across a variety of scientific disciplines.
Random walk theory supplies the basic probability theory behind BLAST ( the most widely used sequence alignment theory).

19. Definitions (cont.)
A random walk is defined as restricted walk if the walk is limited to the interval [a, b].
The endpoints a and b are called absorbing barriers if the random walk eventually stays there forever;
or reflecting barriers if the walk reaches the endpoint and bounces back.

20. Example (cont.): simple RW
Ladder point
Ladder Point (LP):the point in the walk lower than any previously reached points.
Excursion: the part of the walk from a LP until the highest point attained before the next LP.
Excursions in Fig: 1, 1, 4, 0, 0, 0, 3;
BLAST theory focused on the maximum heights achieved by these excursions.

21. the one-dimensional simple random walk
The process starts in state X0 at time t = 0. Independently, at each time instance, the process takes a jump Zn: Prob { Zn = -1} = q, Prob { Zn = +1} = p and Prob { Zn = 0 } = 1 - p - q. The state of the process at time n is Xn = X0 + Z1 + Z2 + … + Zn.

22. the one-dimensional simple random walk
In the analysis below assume:
• Probability of a left-step (tails) is q
where p + q = 1
• Probability of a right-step (heads) is p
The following table shows the probabilities associated with the different possible values of k for n = 1, 2, 3, 4:

23. the one-dimensional simple random walk

24. The gambling banker
Consider two urns A and B in a casino game. Initially A contains two white balls, and
B contains three black balls. The balls are then `shu²ed' repeatedly at discrete time
steps according to the following rule: pick at random one ball from each urn, and swap them.

25. The gambling banker
The three possible states of the system during this (discrete time and discrete
state space) stochastic process are shown below:

26. The gambling banker
A banker decides to gamble on the above process. He enters into the following bet: at
each step the bank wins 9M£ if there are two white balls in urn A, but has to pay
1M£ if not. What will happen to the bank?

27. Questions