Download presentation

1
**Repeated Measure Design of ANOVA**

AMS 572 Group 5

2
**Outline Jia Chen: Introduction of repeated measures ANOVA**

Chewei Lu: One-way repeated measures Wei Xi: Two-factor repeated measures Tomoaki Sakamoto : Three-factor repeated measures How-Chung Liu: Mixed models Margaret Brown: Comparison Xiao Liu: Conclusion

3
**Introduction of Repeated Measures ANOVA**

Jia Chen

4
What is it ? Definition: - It is a technique used to test the equality of means.

5
When To Use It? It is used when all members of a random sample are measured under a number of different conditions. As the sample is exposed to each condition in turn, the measurement of dependent variable is repeated.

6
**Introduction of One-Way Repeated Measures ANOVA**

Che-Wei, Lu Professor:Wei Zhu

7
**One-Way Repeated Measures ANOVA**

Definition A one-way repeated measures ANOVA instead of having one score per subject, experiments are frequently conducted in which multiple score are gathered for each case. Concept of Repeated Measures ANOVA One factor with at least two levels, levels are dependent. Dependent means that they share variability in some way. The Repeated Measures ANOVA is extended from standard ANOVA.

8
**One-Way Repeated Measures ANOVA**

When to Use Measuring performance on the same variable over time for example looking at changes in performance during training or before and after a specific treatment The same subject is measured multiple times under different conditions for example performance when taking Drug A and performance when taking Drug B The same subjects provide measures/ratings on different characteristics for example the desirability of red cars, green cars and blue cars Note how we could do some RM as regular between subjects designs For example, Randomly assign to drug A or B

9
**One-Way Repeated Measures ANOVA**

Source of Variance in Repeated Measures ANOVA SStotal Deviation of each individual score from the grand mean SSb/t subjects Deviation of subjects' individual means (across treatments) from the grand mean. In the RM setting, this is largely uninteresting, as we can pretty much assume that ‘subjects differ’ SSw/in subjects: How Ss vary about their own mean, breaks down into: SStreatment As in between subjects ANOVA, is the comparison of treatment means to each other (by examining their deviations from the grand mean) However this is now a partition of the within subjects variation SSerror Variability of individuals’ scores about their treatment mean

10
**One-Way Repeated Measures ANOVA**

Partition of Sum of Square 𝑆𝑆 𝑡𝑜𝑡𝑎𝑙 𝑆𝑆 𝑏/𝑡𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑠 𝑆𝑆 𝑤/𝑖𝑛𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑠 𝑆𝑆 𝑡𝑟𝑒𝑎𝑚𝑒𝑚𝑡 𝑆𝑆 𝑒𝑟𝑟𝑜𝑟 Repeated Measures ANOVA 𝑆𝑆 𝑡𝑜𝑡𝑎𝑙 𝑆𝑆 𝑏/𝑡𝑡𝑟𝑒𝑎𝑚𝑒𝑛𝑡 𝑆𝑆 𝑤/𝑖𝑛𝑒𝑟𝑟𝑜𝑟 Standard ANOVA

11
**One-Way Repeated Measures ANOVA**

Standard ANOVA Table Variation SS Df MS F Between 𝑖−1 𝑎 𝑛 𝑖 ( 𝑥 𝑖 − 𝑥 ) 2 a-1 MSA= 𝑆𝑆𝐴 𝑎−1 F= 𝑀𝑆𝐴 𝑀𝑆𝐸 Within 𝑖=1 𝑎 𝑗=1 𝑛 𝑖 ( 𝑥 𝑖𝑗 − 𝑥 𝑖 ) 2 N-a MSE= 𝑆𝑆𝐸 𝑁−𝑎 Total 𝑖=1 𝑎 𝑗=1 𝑛 𝑖 ( 𝑥 𝑖𝑗 − 𝑥 ) 2 N-1 Repeated Measures ANOVA Table SS Df MS F Between ( 𝑎 𝑖𝑗 ) 2 𝑠 − 𝑇 2 𝑁 a-1 MSA= 𝑆𝑆𝐴 𝑎−1 F= 𝑀𝑆𝐴 𝑀𝑆𝐸 Within 𝑎 𝑖𝑗 2 − ( 𝑎 𝑖𝑗 ) 2 𝑠 N-a -Subjects ( 𝑆 𝐼 ) 2 𝑎 − 𝑇 2 𝑁 s-1 -Error 𝑆𝑆 𝑤𝑖𝑛𝑡𝑖𝑛 − 𝑆𝑆 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑠 MSE= 𝑆𝑆𝐸 (𝑎−1)(𝑠−1) Total 𝑆𝑆 𝑏𝑒𝑡𝑒𝑤𝑤𝑛 + 𝑆𝑆 𝑤𝑖𝑡ℎ𝑖𝑛𝑔 N-A

12
**One-Way Repeated Measures ANOVA**

Example: Researchers want to test a new anti-anxiety medication. They measure the anxiety of 7 participants three times: once before taking the medication, once one week after taking the medication, and once two weeks after taking medication. Anxiety is rated on a scale of 1-10,with 10 being ”high anxiety” and 1 being “low anxiety”. Are there any difference between the three condition using significant level 𝛼=0.05? Participants Before Week1 Week2 1 9 7 4 2 8 6 3 5

13
**One-Way Repeated Measures ANOVA**

Participants Before Week1 Week2 1 9 7 4 2 8 6 3 5 Define Null and Alternative Hypotheses 𝐻 0 : 𝜇 𝑏𝑒𝑓𝑜𝑟𝑒 = 𝜇 𝑤𝑒𝑒𝑘1 = 𝜇 𝑤𝑒𝑒𝑘2 𝐻 𝑎 : 𝐻 0 𝑖𝑠 𝑛𝑜𝑡 𝑎𝑙𝑙 𝑒𝑞𝑢𝑎𝑙

14
**One-Way Repeated Measures ANOVA**

Participants Before Week1 Week2 1 9 7 4 2 8 6 3 5 Define Degrees of Freedom N= s=7 𝑑𝑓 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 =𝑎−1=3−1=2 𝑑𝑓 𝑊𝑖𝑡ℎ𝑖𝑛 =𝑁−𝑎=21−3=18 𝑑𝑓 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 =𝑠−1=7−1=6 𝑑𝑓 𝐸𝑟𝑟𝑜𝑟 = 𝑑𝑓 𝑊𝑖𝑡ℎ𝑖𝑛 - 𝑑𝑓 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 =18-6=12 𝑑𝑓 𝑡𝑜𝑡𝑎𝑙 =𝑁−1=21−1=20

15
**One-Way Repeated Measures ANOVA**

Participants Before Week1 Week2 1 9 7 4 2 8 6 3 5 Analysis of Variance 𝑆𝑆 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = ( 𝑎 𝑖𝑗 ) 2 𝑠 − 𝑇 2 𝑁 ,where 𝑎 𝑖𝑗 =𝑜𝑏𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛, 𝑇=𝑡𝑜𝑡𝑎𝑙 𝑠𝑢𝑚 𝑜𝑓 𝑒𝑎𝑐ℎ 𝑔𝑟𝑜𝑢𝑝 𝑆𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 = 𝑎 𝑖𝑗 2 − ( 𝑎 𝑖𝑗 ) 2 𝑠 𝑆𝑆 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 = ( 𝑆 𝐼 ) 2 𝑎 − 𝑇 2 𝑁 ,𝑤ℎ𝑒𝑟𝑒 𝑆 𝑖 =𝑠𝑏𝑢𝑗𝑒𝑐𝑡 𝑖𝑡ℎ

16
**One-Way Repeated Measures ANOVA**

Analysis of Variance(ANOVA Table) SS Df MS F Between 98.67 2 49.34 224.27 Within 10.29 18 -Subjects 7.62 6 -Error 2.67 12 0.22 Total 108.96 20 Error=within-Subjects= =2.67 Total=Beteen+Within= =108.96 Test Statistic: 𝐹 0 = 𝑀𝑆 𝑏𝑤𝑡𝑤𝑒𝑒𝑛 𝑀𝑆 𝑒𝑟𝑟𝑜𝑟 = 𝑆𝑆 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑑𝑓 𝑆𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 𝑑𝑓 = =244.27

17
**One-Way Repeated Measures ANOVA**

SS Df MS F Between 98.67 2 49.34 224.27 Within 10.29 18 -Subjects 7.62 6 -Error 2.67 12 0.22 Total 108.96 20 Critical Region 𝐼𝑓 𝐹 0 𝑖𝑠 𝑔𝑟𝑒𝑎𝑡𝑒𝑟 𝑡ℎ𝑎𝑛 𝐹 2,12,0.05 =3.88, 𝑟𝑒𝑗𝑒𝑐𝑡 𝑛𝑢𝑙𝑙 ℎ𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑠. Now, the 𝐹 0 =224.26>3.88. 𝑇ℎ𝑒𝑟𝑒𝑓𝑜𝑟𝑒, 𝑤𝑒 𝑟𝑒𝑗𝑒𝑐𝑡 𝐻 0 𝑡ℎ𝑎𝑡 𝑚𝑒𝑎𝑛𝑠 𝑡ℎ𝑒 𝑡ℎ𝑟𝑒𝑒 𝑐𝑜𝑛𝑑𝑖𝑜𝑛𝑠 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑑 𝑠𝑖𝑔𝑛𝑖𝑐𝑎𝑛𝑡𝑙𝑦 𝑜𝑛 𝑎𝑛𝑥𝑖𝑒𝑡𝑦 𝑙𝑒𝑣𝑒𝑙

18
**One-Way Repeated Measures ANOVA**

SAS Code DATA REPEAT; INPUT SUBJ BEFORE WEEK1 WEEK2; DATALINES; ; PROC ANOVA DATA=REPEAT; TITLE "One-Way ANOVA using the repeated Statment"; MODEL BEFORE WEEK1 WEEK2= / NOUNI; REPEATED TIME 3 (1 2 3); RUN; The Before, Week1, and Week2 are the time level at each participants. The NOUNI(no univariate) is a request not to conduct a separate analysis for each of the three times variables. In here, we don’t have CLASS statement because our data set does not have an independent variable This indicates the labels we want to printed for each level of times

19
**One-Way Repeated Measures ANOVA**

SAS Result

20
**Two-Factor ANOVA with Repeated Measures**

Wei Xi

21
**Stating of the Hypothesis**

Within-Subjects Main Effect Between-Subjects Main Effect Between-Subjects Interaction Effect Within-Subjects By Between-Subjects Interaction Effects

22
**Two-Factor ANOVA with Repeated Measures on One Factor**

23
Hypothesis

24
**ANOVA TABLE Source DF SS MS F Factor A a-1 SSA SSA/(a-1)**

MSA/MSWA ～ F(a-1),n(a-1) Factor B b-1 SSB SSB/(b-1) MSB/MSE ～ F(b-1),n(a-1)(b-1) AB Interaction (a-1)(b-1) SSAB SSAB/(a-1)(b-1) MSAB/MSE ～ F(a-1)(b-1,n(a-1)(b-1) Subjects within A (n-1)a SSWA SSWA/(n-1)a Error (n-1)a(b-1) SSE SSE/((n-1)a(b-1) Total nab-1 SST

25
Example The shape variable is the repeated variable. This produces an ANOVA with one between-subjects factor. If you were to examine the expected mean squares for this setup, you would find that the appropriate error term for the test of calib is subject|calib. The appropriate error term for shape and shape#calib is shape#subject|calib (which is the residual error since we do not include the term in the model).

26
**SAS Code Data Q1; set pre.Q1; run; proc anova data=Q1;**

title' Two-way Anova with a Repeated Measure on One Factor'; class calib; model shape_1 shape_2 shape_3 shape_4 = calib/nouni; repeated shape 4; means calib;

27
Analysis of SAS Output MANOVA Test Criteria and Exact F Statistics for the Hypothesis of no shape Effect H = Anova SSCP Matrix for shape E = Error SSCP Matrix S=1 M=0.5 N=0 Statistic Value F Value Num DF Den DF Pr > F Wilks' Lambda 25.69 3 2 0.0377 Pillai's Trace Hotelling-Lawley Trace Roy's Greatest Root At α=0.05,we reject the hypothesis and conclude that there is shape Effect MANOVA Test Criteria and Exact F Statistics for the Hypothesis of no shape*calib Effect H = Anova SSCP Matrix for shape*calib E = Error SSCP Matrix S=1 M=0.5 N=0 Statistic Value F Value Num DF Den DF Pr > F Wilks' Lambda 3.31 3 2 0.2404 Pillai's Trace Hotelling-Lawley Trace Roy's Greatest Root At α=0.05,we cannot reject the hypothesis and conclude that there is no shape*calib Effect

28
**Tests of Hypotheses for Between Subjects Effects**

Source DF Anova SS Mean Square F Value Pr > F calib 1 11.89 0.0261 Error 4 Univariate Tests of Hypotheses for Within Subject Effects Source DF Anova SS Mean Square F Value Pr > F Adj Pr > F G - G H - F shape 3 12.80 0.0005 0.0099 0.0011 shape*calib 2.01 0.1662 0.2152 0.1791 Error(shape) 12

29
**Two-Factor ANOVA with Repeated Measures on both Factors**

30
**ANOVA TABLE Source DF SS MS F Subjects n-1 SSS SSS/I-1 MSS/MSE**

Factor A a-1 SSA SSA/(a-1) MSA/MSA*S ～ F(a-1),(n-1)(a-1) Factor B b-1 SSB SSB/(b-1) MSB/MSB*S ～ F(b-1),(n-1)(b-1) AB Interaction (a-1)(b-1) SSAB SSAB/((a-1)(b-1)) MSAB/MSE～F(a-1)(b-1),(n-1)(a-1)(b-1) A*Subjects (n-1)(a-1) SSA*S SSWA/((n-1)a) SSA*S/MSE F(a-1)(n-1),(n-1)(a-1)(b-1) B*Subjects (n-1)(b-1) SSB*S SSWB/((n-1)b) F(n-1)(b-1),(n-1)(a-1)(b-1) Error (n-a)(a-1)(b-1) SSE SSE/((n-1)(a-1)(b-1)) Total nab-1 SST

31
Example Three subjects, each with nine accuracy scores on all combinations of the three different dials and three different periods. With subject a random factor and both dial and period fixed factors, the appropriate error term for the test of dial is the dial#subject interaction. Likewise, period#subject is the correct error term for period, and period#dial#subject (which we will drop so that it becomes residual error) is the appropriate error term for period#dial.

32
**SAS Code Data Q2; Input Mins1-Mins9; Datalines;**

; ODS RTF STYLE=BarrettsBlue; Proc anova data=Q2; Model Mins1-Mins9=/nouni; Repeated period 3, dail 3/nom; Run; ods rtf close;

33
**SAS Output Univariate Tests of Hypotheses for Within Subject Effects**

Source DF Anova SS Mean Square F Value Pr > F Adj Pr > F G - G H - F period 2 14.45 0.0148 0.0563 0.0394 Error(period) 4 Greenhouse-Geisser Epsilon 0.5364 Huynh-Feldt Epsilon 0.6569 At α=0.05,we reject the hypothesis and conclude that there is period Effect Source DF Anova SS Mean Square F Value Pr > F Adj Pr > F G - G H - F dail 2 50.91 0.0014 0.0169 0.0115 Error(dail) 4 Greenhouse-Geisser Epsilon 0.5227 Huynh-Feldt Epsilon 0.5952 At α=0.05,we reject the hypothesis and conclude that there is dail Effect Source DF Anova SS Mean Square F Value Pr > F Adj Pr > F G - G H - F period*dail 4 0.30 0.8715 0.6603 0.7194 Error(period*dail) 8 Greenhouse-Geisser Epsilon 0.2827 Huynh-Feldt Epsilon 0.4006 At α=0.05,we cannot reject the hypothesis and conclude the there is no period*dail Effect

34
**Three-factor Experiments with a repeated measure**

T. Sakamoto

35
**Example of a marketing experiment**

Case of this example A company which produces some Liquid Crystal Display wants to examine the characteristics of its prototype products. Experiment The subjects who belong to a region X or Y see the Liquid Crystal Display A, B, or C. Each type of LCD is seen twice; once in the light and the other in the dark. The preferences of the LCD are measured by the subjects, on a scale from 1 to 5 (1= lowest, 5=highest).

36
**Experimental Design and Data**

Three factors Type of LCD Regions to which the specimens belong In the light / In the dark Repeted measure factor : In the light / In the dark Type of LCD A B C subj light dark REGION X 1 5 4 11 21 2 12 6 22 3 13 23 14 24 15 25 Y 16 26 7 17 27 8 18 28 9 19 29 10 20 30

37
**SAS PROGRAM data lcd; input subj type $ region $ light dark @@;**

datalines; 1 a a a a a 5 3 6 a a a a a 5 4 11 b b b b b 4 6 16 b b b b b 4 4 21 c c c c c 4 3 26 c c c c c 4 3 ; run; proc anova data=lcd; title ’Three-way ANOVA with a Repeated Measure'; class type region; model light dark = type | region /nouni; repeated light_dark; means type | region;

38
OUTPUT(Part 1/4):

39
OUTPUT(Part 2/4):

40
OUTPUT(Part 3/4): 40/81

41
OUTPUT(Part 4/4):

42
Mixed Effect Models How-Chang Liu

43
Mixed Models When we have a model that contains random effect as well as fixed effect, then we are dealing with a mixed model. From the above definition, we see that mixed models must contain at least two factors. One having fixed effect and one having random effect.

44
Why use mixed models? When repeated measurements are made on the same statistical units, it would not be realistic to assume that these measurements are independent. We can take this dependence into account by specifying covariance structures using a mixed model

45
**Definition A mixed model can be represented in matrix notation by:**

𝑦= 𝛽 0 𝑋+ 𝛽 1 𝑍+𝜀 𝑦 is the vector of observations 𝛽 0 is the vector of fixed effects 𝛽 1 is the vector of random effects 𝜀 is the vector of I.I.D. error terms 𝑋 and 𝑍 are matrices relating 𝛽 0 and 𝛽 1 to 𝑦

46
**Assumptions 𝛽 1 ~Normal 0,G 𝜀~Normal 0,R R and G are constants**

We also assume that 𝛽 1 and 𝜀 are independent We get V = ZGZ' + R, where V is the variance of y

47
How to estimate 𝛽 0 and 𝛽 1 ? If R and G are given: Using Henderson’s Mixed Model equation, we have: 𝑋 ′ 𝑅 −1 𝑋 𝑋 ′ 𝑅 −1 𝑍 𝑍 ′ 𝑅 −1 𝑋 𝑍 ′ 𝑅 −1 𝑍+ 𝐺 −1 𝛽 0 𝛽 1 = 𝑋 ′ 𝑅 −1 𝑦 𝑍 ′ 𝑅 −1 𝑦 So 𝛽 0 = (𝑋 ′ 𝑉 −1 𝑋 ) −1 X′ 𝑉 y And 𝛽 1 = 𝐺𝑍′ 𝑉 −1 (y−X 𝛽 0 )

48
**What if G and R are unknown?**

We know that both 𝛽 1 and 𝜀 are normally distributed, so the best approach is to use likelihood based methods There are two methods used by SAS: 1)Maximum likelihood (ML) 2)Restricted/residual maximum likelihood (REML)

49
Example Below is a table of growth measurements for 11 girls and 16 boys at ages 8, 10, 12, 14: Person gender age8 age10 age12 age14 1 F F F F F F F F F F F M M M Person gender age8 age10 age12 age14 15 M M M M M M M M M M M M M

50
Using SAS data pr; input Person Gender $ y1 y2 y3 y4; y=y1; Age=8; output; y=y2; Age=10; output; y=y3; Age=12; output; y=y4; Age=14; output; drop y1-y4; datalines; 1 F F … ; Run;

51
Using SAS proc mixed data=pr method=ml covtest; class Person Gender; model y = Gender Age Gender*Age / s; repeated / type=un subject=Person r; run;

52
**Class Level Information**

Results Model Information Data Set WORK.PR Dependent Variable y Covariance Structure Unstructured Subject Effect Person Estimation Method ML Residual Variance Method None Fixed Effects SE Method Model-Based Degrees of Freedom Method Between-Within As one can see, the covariance matrix is unstructured, as we are going to estimate it using the maximum likelihood method Class Level Information Class Levels Values Person 27 Gender 2 F M

53
**Number of Observations Convergence criteria met.**

Dimensions Covariance Parameters 10 Columns in X 6 Columns in Z Subjects 27 Max Obs Per Subject 4 As one can see, we do not have a Z matrix for this model Number of Observations Number of Observations Read 108 Number of Observations Used Number of Observations Not Used Iteration History Iteration Evaluations -2 Log Like Criterion 1 2 Convergence criteria met. the convergence of the Newton-Raphson algorithm means that we have found the maximum likelihood estimates

54
**Covariance Parameter Estimates**

Cov Parm Subject Estimate Standard Error Z Value Pr Z UN(1,1) Person 5.1192 1.4169 3.61 0.0002 UN(2,1) 2.4409 0.9835 2.48 0.0131 UN(2,2) 3.9279 1.0824 3.63 0.0001 UN(3,1) 3.6105 1.2767 2.83 0.0047 UN(3,2) 2.7175 1.0740 2.53 0.0114 UN(3,3) 5.9798 1.6279 3.67 UN(4,1) 2.5222 1.0649 2.37 0.0179 UN(4,2) 3.0624 1.0135 3.02 0.0025 UN(4,3) 3.8235 1.2508 3.06 0.0022 UN(4,4) 4.6180 1.2573 The table lists the 10 estimated covariance parameters in order. In other words, these are the estimates for R, the variance of 𝜀

55
**Solution for Fixed Effects**

Gender Estimate Standard Error DF t Value Pr > |t| Intercept 0.9356 25 16.93 <.0001 F 1.5831 1.4658 1.08 0.2904 M . Age 0.8268 10.45 Age*Gender 0.1239 -2.83 0.0091 From this table, we see that the boys intercept is at , whole the girls intercept is at =17.42. The estimate of the boys’ slope is at 0.827, while the girls’ slpe is at =0.477 So the girls’ starting point is higher than the girls but their growth rate is only about half of that of the boys

56
**Type 3 Tests of Fixed Effects**

Num DF Den DF F Value Pr > F Gender 1 25 1.17 0.2904 Age 110.54 <.0001 Age*Gender 7.99 0.0091 This is probably the most important table from our results: The gender row tests the null hypothesis that girls and boys have a common intercept. As we can see we cannot reject that hypothesis The Age tests the null hypothesis that age does not affect the growth rate. As we can see, we reject the null hypothesis as the F-value is large. The Age*gender tests reveals that there is a difference in slope at the 1% significance level.

57
**Repeated Measures ANOVA vs. Independent Measures ANOVA**

Magarate Brown

58
**Can we just use standard ANOVA with repeated measures data?**

No, Independent Measures (standard) ANOVA assumes the data are independent. Data from a repeated measures experiment not independent

59
**How are standard ANOVA and repeated measures ANOVA the same?**

Independent measures ANOVA: an extension of the pooled variance t-test Repeated Measures ANOVA: an extension of the paired sample t-test

60
**How are standard ANOVA and repeated measures ANOVA the same?**

Independent measures ANOVA: assumes the population variances are equal (homogeneity of variance) Repeated Measures ANOVA: sphericity assumption that the population variances of all the differences are equal

61
**How are standard ANOVA and repeated measures ANOVA the same?**

Both assume Normality of the population

62
**Advantages to using Repeated Measures instead of Independent Measures**

Limited number of subjects available Prefer to limit the number of subjects Less variability (finger tapping with caffeine example) Can examine effects over time

63
**Drawbacks Practice effect**

Example: subjects get better at performing a task each time with “practice” Differential transfer: “This occurs when the effects of one condition persist and affect participants’ experiences during subsequent conditions.” (format: Example: medical treatments

64
**Resources (to be formatted)**

Similar presentations

Presentation is loading. Please wait....

OK

Break Time Remaining 10:00.

Break Time Remaining 10:00.

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google