1. 15 October 2012 Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Discrete Time Markov Chains

2. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  X(n): the state/value of a process at the n -th period (time slot)  X(n) is random and takes values in a finite or countable set  The sequence X(1), X(2), …, X(n) could be a) The values of a stock each day b) The web page a user is currently browsing c) The Access Point(AP) a moving user is currently associated with  We would like a probabilistic model for X(1),X(2),…,X(n) a) X(i) are independent => not so realistic (for above examples) b) X(i+1) depends on X(i) => not a bad model to start with 2 P{X(n+1) = j | X(n) = i,X(n-1) = i n-1,…,X(1) = X 0 } = P{X(n+1) = j | X(n) = i} = p ij (Markov Property)

3. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Assume state space: 0, 1, 2, …  P ij : P{S(n+1) = j | S(n) = i}   Stationary: P{S(n+1) = j | S(n) = i} = P{S(1) = j | S(0) = i}  for any n  p ij define a transition matrix P = 3

4. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Simple weather model  X(n): weather after n days  Simple channel error model  Prob of error = 0.1  Case 1: q = 0.1  Case 2: q = 0.9  What is the transition matrix P in both cases? 4 goodbad p 1-p q 1-q (uncorrelated) (burst of errors)

5. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A simple handover model  cell -> state  need to find transition probabilities  depend on road structure, user profile, statistics 5 user on the phone Cellular Network 0.2 0.3 0.5 0.4 0.7 0.2 0.15 0.2 0.8 0.6 Markov Chain

6. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  So far we know probabilities for next step: S(n)->S(n+1)  Probabilities for S(1) -> S(n)?  If P (2) = {p ij (2) } denotes the 2-step trans. probability matrix, then P (2) = P P  P ij (2) : multiply row I with column j 6

7. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Generalizing for n steps:  Kolmogorov-Chapman equations  P (n) = PP…P = (P) n (n-step transition probabilities)  P (n) = P (m) P (l)  e.g. P (5) = P (4) P (1) = P (2) P (3) = P (2) P P (2) 7

8. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Your iPhone browser can pre-fetch pages to improve speed  The current Web Page being browsed has 3 links  Study of click statistics have shown that  A user clicks link 1 next with prob 0.8  Link 2 with prob 0.1  Link 3 with prob 0.1  It seems reasonable that the browser pre-fetches link 1  Assume now a tiny subset of the Web with the following transition matrix  Start at page A Q: Which 2 web pages should the browser pre-fetch? 8 S (1) = [0.2, 0.2, 0.5, 0.1] S (2) = [0.13, 0.29, 0.24, 0.34] S (0) = [1, 0, 0, 0]

9. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  What happens to p ij (n) as n goes to infinity?  What is the lim P (n) as n->∞?  Q: Does it always converge?? (we’ll see this later)  If limit exists, then for any initial state i  π = {π 0, π 1,…, π m } is called the stationary distribution IF π P = π and Σ i π i = 1  The above equation can be used to find π 9

10. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  The weather system  Data rates supported for outdoor user  e.g. (128Kbps, 320Kbps, 1 Mbps)  What about this one? 10

11. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis (Reachability) State j is reachable by i  P ij (n) > 0 for some n  denote i  j (Communication) states i and j communicate (i j) if i  j and j  i (Recurrence/Transience) Denote f i the probability to ever return to i, starting from i  Transient) state is transient if f i < 1  number of returns is geometric (f i ): why?  Recurrent) state i is recurrent if f i = 1  Positive recurrent) expected time between visits to i is finite  Null recurrent) expected time between visits is infinite (strange!?)  Periodicity) state i is periodic, if exists d such that p ii (n) = 0, if n ≠ kd  Largest d with above property is called period 11

12. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Definition: A subset of states S: {for all i,j in S => i j} is called a class Lemma 1: recurrence is a class property  if i is recurrent and i j, then j is recurrent  Proof (by contradiction): if j transient => can never return to j after some time => cannot be at i either (since there is always a chance to go from i to j) Lemma 2: transience is a class property  Similar argument  Periodicity and positive/null-recurrence are also class properties Irreducible) A Markov Chain is irreducible if it has only 1 class  A finite MC which is irreducible is always positive-recurrent 12 Theorem: If a DTMC is aperiodic, irreducible and positive recurrent (“ergodic”) then it has a stationary distribution π = {π 1,…,π N }

13. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Search for “Network Modeling”  1000s of pages (courses, companies, etc.) contain these keywords  Goal: Make the page I’m interested appear in at least the top 10 pages shown Q: How should pages be ranked? A1: A page is important if many links to this page Q: What is wrong with this metric? A: - link from Yahoo more important than link from /~spyropou - can easily “fool” this! (create many dummy pages pointing to mine) Q: How to fix this? A: weigh a link from x to page p with the number of links into x Q: Can this system be fooled? A: Yes! Create 1000 dummy pages pointing to each other and mine 13

14. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A page p has high rank (is important) if the pages pointing to it also have high rank Q: How is this different from before? A: All the dummy pages pointing to each other would still be low rank, if no “external” link from a high rank page Q: Is a link from a page with 1000 other links as important as a link from a page with 5 other links? A: No! If page i has a link to j and k total outgoing links  P ij = 1/k Q: Where does this lead? How do we calculate the rank? A: rank of page j = Σ i { rank of page i * Prob of link i  j} i.e. r j = Σ i r i P ij  (basis of) Page Rank Algorithm: rank of pages = stationary prob. of an MC with transitions P ij 14

15. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: What about this tiny WWW? A: π A = π N = 2/5, π M = 1/5 Q: But what about these two? A: These two chains are reducible and do not have a stationary distr. 15  Google’s heuristic: introduce additional arrows to avoid such loops and dead ends (not necessarily optimal!) Q: How does Google solve the huge system of eq (million pages)?

16. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 16 [Network] m nodes on a common wire  Or wireless channel  Time is slotted [New Packets] Each node transmits a new packet with probability p (p < 1/m) [Success] If exactly 1 packet is transmitted [Collision] If k>1 (re-)transmissions during a slot  Every collided message is stored, to be resent later [Retransmission] Each collided(back-logged) message is retransmitted with prob q new packet old packet p p p √ X q Does Aloha work??

17. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Q: What should we take as the state of the chain X n ?  A: The number of backlogged messages (at slot n)  Transition probabilities from state 0 (no backlogged msg) P 00 = P 01 = P 0k = 17 (0 or 1 node transmits) 0 (not possible) if k ≤ m (any k of the m nodes transmit) 0 if k > m

18. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis P k,k-1 = (better) P k,k = P k,k+1 =(worse) P k,k+r = (1< r ≤ m)(worse) P k,k+r = 0(r > m)  Assume we are now at state k (i.e. k messages backlogged) 18 (0 new, 1 old) (0 new, 0 old)(1 new, 0 old) (r new, any old) (0 new, ≥2 old) (1 new) (>0 old)

19. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : :  How can we tell if “Aloha works”?  Assume 10 nodes and transmission probability p = 0.05  Load = 0.5 Capacity  Intuition: (necessary) q should be small (re-Tx)  To ensure retransmissions don’t overload medium  Let’s assume q = 0.005 (10 times smaller than p)  Q: Is Aloha Stable? If backlog becomes infinite => delay goes to infinity! 19

20. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  When at state k:  P back (k) : reduce backlog  P fwd (k) : increase backlog (MHB, Ch.10) (Why?)  So??? 20 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

21. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  For large enough k => states k+1, k+2, …, are transient  Markov Chain is transient => Aloha protocol is unstable!  Q: would Aloha work if we make q really (REALLY) small?  A: No!  Intuition: Let E[N] the expected Tx at state k  If E[N] ≥ 1 then situation either stays the same or worse  But E[N] = mp + kq --- what happens if k  ∞? 21 0 p 00 p 01 1 2 p 10 p 21 p 12 … p 0m k k-1 p k,k-1 p k-1,k p k-m,k k+1 p k,k-1 p k-1,k p k,k+2 p k,k+m k+2k+m … p k+1,k+2 p k+2,k+1 : : P back (k) P fwd (k)

22. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Q: How can we fix the problem and make the chain ergodic (and the system stable)? A1: E[N] = mp + kq q < (1-mp)/k i.e. q = f(backlog) A2: or be more aggressive  geometric backoff q = a/k n  a < (1-mp) A3: or even exponential q = β -k  β > 1 Exponential backoff is the basis behind Ethernet Q: Why should q not be too small? A: Retransmission delay goes to infinity 22

23. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Time Averages  N i (t) = number of times in state i by time t  p i = (percentage of time in state i) Ensemble Averages  m ij : expected time to reach j (for 1 st time), starting from i  m ii = expected time between successive visits to i  π i = (prob. of being at state i after many steps Theorem: for an ergodic DTMC (proof based on Renewal Theory) 23

24. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  π i p ij : rate of transitions from state i to state j  π i = percentage of time in state i  p ij = percentage of these times the chain moves next to j  Q: What is Σ j π j p ji ?  A: rate into state i 24 π0π0 p 00 p 01 π1π1 π2π2 p 10 p 21 p 12  From stationary equation: π i =Σ j π j p ji  π i : rate into i  But also π i =Σ j π i p ij (why?)  Theorem: Σ j π j p ji = Σ j π i p ij (rate in = rate out)  Q: why is this reasonable?  A: Cannot make a transition from i, without a transition into i before it (difference in number is at most 1)  Assume a subset of states S: rate into S = rate out of S rate into 1 π 2 p 21 rate out of 1 π 1 p 12 +π 1 p 10

25. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Assume there exist π i : π i p ij = π j p ji (for all i and j)  and Σ i π i = 1  Then:  π i is the stationary distribution for the chain  The above equations are called local balance equations  The Markov chain is called time-reversible  Solving for the Stationary Distribution of a DTMC 25 Stationary Eq. π i = Σ j π j p ji (not always easy) Global Balance Σ j ≠i π i p ij = Σ j ≠i π j p ji (a bit easier) Local Balance π i p ij = π j p ji (easiest! try first!)

26. 15 October 2012 Eurecom, Sophia-Antipolis Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Absorbing Markov Chains 26

27. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A mouse is trapped in the above maze with 3 rooms and 1 exit  When inside a room with x doors, it chooses any of them with equal probability (1/x) Q: How long will it take it on average to exit the maze, if it starts at room i? Q: How long if it starts from a random room? 27 1 2 3 exit

28. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Def: T i = expected time to leave maze, starting from room I  T 2 = 1/3*1 + 1/3*(1+T 1 )+1/3*(1+T 3 ) = 1 + 1/3*(T 1 +T 3 )  T 1 = 1 + T 2  T 3 = 1 + T 2  T 2 = 5, T 3 = 6, T 1 = 6 Q: Could you have guessed it directly? A: times room 2 is visited before exiting is geometric(1/3)  on average, the wrong exit will be taken twice (each time costing two steps) and the 3 rd time the mouse exits 28 1 2 3 exit

29. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A packet must be routed towards the destination over the above network  “Hot Potato Routing” works as follows: when a router receives a packet, it picks any of its outgoing links randomly (including the incoming link) and send the packet immediately. Q: How long does it take to deliver the packet? 29 destination

30. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  First Step Analysis: We can still apply it!  But it’s a bit more complicated: 9x9 system of linear equations  Not easy to guess solution either!  We’ll try to model this with a Markov Chain 30 destination

31. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  9 transient states: 1-9  1 absorbing state: A Q: Is this chain irreducible? A: No! Q: Hot Potato Routing Delay  expected time to asborption? 31 1 2 3 4 5 6 78 9 A 1 1/4 1/2 1/3 1/2 1/4 1/3 1/2 1/3 1/2 1/4 1 1/3

32. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  We can define transition matrix P (10x10) Q: What is P (n) as n  ∞? A: every row converges to [0,0,…,1] Q: How can we get ET iA ?  (expected time to absorption starting from i) Q: How about ? A: No, the sum goes to infinity! 32 1 2 3 4 5 6 78 9 A 1 1/4 1/2 1/3 1/2 1/4 1/3 1/2 1/3 1/2 1/4 1 1/3

33. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Transition matrix can be written in canonical form  Transient states written first, followed by absorbing ones  Calculate P (n) using canonical form Q: Q n as n  ∞? A: it goes to O Q: where does the (*) part of the matrix converge to if only one absorbing state? A: to a vector of all 1s 33

34. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Theorem: The matrix (I-Q) has an inverse  N = (I-Q) -1 is called the fundamental matrix  N = I + Q + Q 2 + …  n ik : the expected number of times the chain is in state k, starting from state i, before being absorbed Proof: 34

35. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Theorem:  Let T i be the expected number of steps before the chain is absorbed, given that the chain starts in state i,  let T be the column vector whose i th entry is T i. then T = Nc,  where c is a column vector all of whose entries are 1 Proof:  Σ k n ik :add all entries in the i th row of N  expected number of times in any of the transient states for a given starting state i  the expected time required before being absorbed.  T i = Σ k n ik. 35

36. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis Theorem:  b ij :probability that an absorbing chain will be absorbed in (absorbing) state j, if it starts in (transient) state i.  B: (t-by-r) matrix with entries b ij. then B = NR,  R as in the canonical form. Proof: 36

37. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  Use Matlab to get matrices  Matrix N =  Vector T = 37 3.2174 2.6957 2.3478 6.6522 4.5652 4.0000 3.8261 3.2174 2.6087 1.3478 3.9130 2.9565 4.0435 4.3043 4.0000 2.5217 2.3478 2.1739 1.1739 2.9565 3.4783 3.5217 3.6522 4.0000 2.2609 2.1739 2.0870 2.2174 2.6957 2.3478 6.6522 4.5652 4.0000 3.8261 3.2174 2.6087 1.5217 2.8696 2.4348 4.5652 4.9565 4.0000 2.7826 2.5217 2.2609 1.0000 2.0000 2.0000 3.0000 3.0000 4.0000 2.0000 2.0000 2.0000 1.9130 2.5217 2.2609 5.7391 4.1739 4.0000 4.8696 3.9130 2.9565 1.6087 2.3478 2.1739 4.8261 3.7826 4.0000 3.9130 4.6087 3.3043 1.3043 2.1739 2.0870 3.9130 3.3913 4.0000 2.9565 3.3043 3.6522 33.1304 27.6087 25.3043 32.1304 27.9130 21.0000 32.3478 30.5652 26.7826

38. Thrasyvoulos Spyropoulos / spyropou@eurecom.fr Eurecom, Sophia-Antipolis  A wireless path consisting of H hops (links)  link success probability p  A packet is (re-)transmitted up to M times on each link  If it fails, it gets retransmitted from the source (end-to-end) Q: How many transmissions until end-to-end success? 38