1 Markov Chains: Transitional Modeling Qi Liu

2 content Terminology Transitional Models without Explanatory Variables Transitional Models without Explanatory Variables Inference for Markov chains Data Analysis :Example 1 (ignoring explanatory variables)Example 1 Transitional Models with Explanatory Variables Data Anylysis: Example 2 (with explanatory variables)Example 2

3 Terminology Transitional models Markov chain K th-order Markov chain Tansitional probabilities and Tansitional matrix Tansitional probabilities and Tansitional matrix

4 Transitional models {y0,y1,…,yt-1} are the responses observed previously. Our focus is on the dependence of Yt on the {y0,y1,…,yt-1} as well as any explanatory variables. Models of this type are called transitional models.

5 Markov chain A stochastic process, for all t, the conditional distribution of Yt+1,given Y0,Y1, …,Yt is identical to the conditional distribution of Yt+1 given Yt alone. i.e, given Yt, Yt+1 is conditional independent of Y0,Y1, …,Yt-1. So knowing the present state of a Markov chain,information about the past states does not help us predict the future P(Yt+1|Y0,Y1, … Yt)=P(Yt+1|Yt)

6 K th-order Markov chain For all t, the conditional distribution of Yt+1 given Y0,Y1,…,Yt is identical to the conditional distribution of Yt+1,given (Yt,…,Yt-k+1) P(Yt+1|Y0,Y1,…Yt)=P(Yt+1|Yt-k+1,Yt-k+2,….Yt) i.e, given the states at the previous k times, the future behavior of the chain is independent of past behavior before those k times. We discuss here is first order Markov chain with k=1.

7 Tansitional probabilities

8 Transitional Models without Explanatory Variables At first, we ignore explanatory variables. Let f(y0,…,yT) denote the joint probability mass function of (Y0,…,YT),transitional models use the factorization: f(y0,…,yT) =f(y0)f(y1|y0)f(y2|y0,y1)…f(yT|y0,y1,…,yT-1) This model is conditional on the previous responses. For Markov chains, f(y0,…,yT) =f(y0)f(y1|y0)f(y2|y1)…f(yT|yT-1) (*) From it, a Markov chain depends only on one-step transition probabilities and the marginal distribution for the initial state. It also follows that the joint distribution satisfies loglinear model (Y0Y1, Y1Y2,…, YT-1YT) For a sample of realizations of a stochastic process, a contingency table displays counts of the possible sequences. A test of fit of this loglinear model checks whether the process plausibly satisfies the Markov property.

9 Inference for Markov chains

10 Inference for Markov chains(continue)

11 Example 1 (ignoring explanatory variables) A study at Harvard of effects of air pollution on respiratory illness in children. The children were examined annually at ages 9 through 12 and classified according to the presence or absence of wheeze. Let Yt denote the binary response at age t, t=9,10,11,12. 1 wheeze;2 no wheeze y9y10y11y12coun t y9y10y11y12count

12 Code of Example 1 Code of 11.7 data breath; input y9 y10 y11 y12 count; datalines; ; proc genmod; class y9 y10 y11 y12; model count= y9 y10 y11 y12 y9*y10 y10*y11 y11*y12 /dist=poi lrci type3 residuals obstats; run; proc genmod; class y9 y10 y11 y12; model count= y9 y10 y11 y12 y9*y10 y9*y11 y10*y11 y10*y12 y11*y12 y9*y10*y11 y10*y11*y12/dist=poi lrci type3 residuals obstats; run; proc genmod; class y9 y10 y11 y12; model count= y9 y10 y11 y12 y9*y10 y9*y11 y9*y12 y10*y11 y10*y12 y11*y12 /dist=poi lrci type3 residuals obstats; run; data breath_new;set breath; a=y9*y10+y10*y11+y11*y12; b=y9*y12+Y10*y12+y9*y11; proc genmod; class y9 y10 y11 y12; model count= y9 y10 y11 y12 a b /dist=poi lrci type3 residuals obstats; run;

13 Data analysis The loglinear model (y9y10,y10y11,y11y12) a first order Markov chain. P(Y11|Y9,Y10)=P(Y11|Y10) P(Y12|Y10,Y11)=P(Y12|Y11) G²= , df=8, with p-value<0.0001, it fits poorly. So given the state at time t, classification at time t+1 depends on the states at times previous to time t.

14 Data analysis (cont…) Then we consider model (y9y10y11, y10y11y12),a second-order Markov chain, satisfying conditional independence at ages 9 and 12, given states at ages 10 and 11. This model fits poorly too, with G²= ,df=4 and p-value<0.001.

15 Data analysis (cont) The loglinear model (y9y10,y9y11,y9y12,y10y11,y10y12,y11y12) that permits association at each pair of ages fits well, with G²=1.4585,df=5,and p-value= Parameter Estimate Error Limits Square Pr > ChiSq y9*y <.0001 y9*y <.0001 y9*y <.0001 y10*y <.0001 y10*y <.000 y11*y <.0001

16 Data analysis (cont) From above, we see that the association seems similar for pairs of ages1 year apart, and somewhat weaker for pairs of ages more than 1 year apart. So we consider the simpler model in which It also fits well, with G²=2.3, df=9, and p-value=

17 Estimated Conditonal Log Odds Ratios

18 Transitional Models with Explanatory Variables

19

20 Data Anylysis Example 2 (with explanatory variables) At ages 7 to 10, children were evaluated annually on the presence of respiratory illness. A predictor is maternal smoking at the start of the study, where s=1 for smoking regularly and s=0 otherwise.

21 Child ’ s Respiratory Illness by Age and Maternal Smoking

22 Data analysis (cont)

23 Code of Example 2 data illness; input t tp ytp yt s count; datalines; ; run; proc logistic descending; freq count; model yt = t ytp s/scale=none aggregate; run;

24 Output from SAS Deviance and Pearson Goodness-of-Fit Statistics Criterion DF Value Value/DF Pr > ChiSq Deviance Pearson Analysis of Maximum Likelihood Estimates Standard Wald Parameter DF Estimate Error Chi-Square Pr > ChiSq Intercept t ytp <.0001 s

25 Analysis

26 The model fits well, with G²=3.1186, df=8, p- value= The coefficient of is with SE , Chi- Square statistic and p-value <.0001,which shows that the previous observation has a strong positive effect. So if a child had illness when he was t-1, he would have more probability to have illness at age t than a child who didn’t have illness at age t-1. The coefficient of s is , the likelihood ratio test of H0 :=0 is ,df=1,with p-value There is slight evidence of a positive effect of maternal smoking.

27 Interpratation of Paramters ß

28 Thank you !