Download presentation

Presentation is loading. Please wait.

Published byOsborn Paul Modified over 5 years ago

1
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-1 Statistics for Managers Using Microsoft® Excel 5th Edition Chapter 12 Chi Square Tests and Nonparametric Tests

2
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-2 Learning Objectives In this chapter, you learn: How and when to use the chi-square test for contingency tables How to use the Marascuilo procedure for determining pair-wise differences when evaluating more than two proportions How and when to use the McNemar test How and when to use nonparametric tests

3
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-3 Contingency Tables Useful in situations involving multiple population proportions Used to classify sample observations according to two or more characteristics Also called a cross-classification table.

4
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-4 Contingency Table Example Hand Preference vs. Gender Dominant Hand: Left vs. Right Gender: Male vs. Female 2 categories for each variable, so the table is called a 2 x 2 table Suppose we examine a sample of 300 college students

5
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-5 Contingency Table Example Sample results organized in a contingency table: Hand Preference Gender FemaleMale Left122436 Right108156264 120180300 120 Females, 12 were left handed 180 Males, 24 were left handed sample size = n = 300:

6
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-6 Contingency Table Example If H 0 is true, then the proportion of left-handed females should be the same as the proportion of left-handed males. The two proportions above should be the same as the proportion of left-handed people overall. H 0 : π 1 = π 2 (Proportion of females who are left handed is equal to the proportion of males who are left handed) H 1 : π 1 ≠ π 2 (The two proportions are not the same – Hand preference is not independent of gender)

7
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-7 The Chi-Square Test Statistic where: f o = observed frequency in a particular cell f e = expected frequency in a particular cell if H 0 is true 2 for the 2 x 2 case has 1 degree of freedom The Chi-square test statistic is: Assumed: each cell in the contingency table has expected frequency of at least 5

8
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-8 The Chi-Square Test Statistic Decision Rule: If 2 > 2 U, reject H 0, otherwise, do not reject H 0 The 2 test statistic approximately follows a chi-square distribution with one degree of freedom 2U2U 0 Reject H 0 Do not reject H 0

9
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-9 Computing the Average Proportion Here: 120 Females, 12 were left handed 180 Males, 24 were left handed The proportion of left handers overall is 0.12, that is, 12% The average proportion is:

10
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-10 Finding Expected Frequencies To obtain the expected frequency for left handed females, multiply the average proportion left handed (p) by the total number of females To obtain the expected frequency for left handed males, multiply the average proportion left handed (p) by the total number of males If the two proportions are equal, then P(Left Handed | Female) = P(Left Handed | Male) =.12 i.e., we would expect (.12)(120) = 14.4 females to be left handed (.12)(180) = 21.6 males to be left handed

11
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-11 Observed vs. Expected Frequencies Hand Preference Gender FemaleMale Left Observed = 12 Expected = 14.4 Observed = 24 Expected = 21.6 36 Right Observed = 108 Expected = 105.6 Observed = 156 Expected = 158.4 264 120180300

12
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-12 The Chi-Square Test Statistic Hand Preference Gender FemaleMale Left Observed = 12 Expected = 14.4 Observed = 24 Expected = 21.6 36 Right Observed = 108 Expected = 105.6 Observed = 156 Expected = 158.4 264 120180300 The test statistic is:

13
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-13 The Chi-Square Test Statistic Decision Rule: If 2 > 3.841, reject H 0, otherwise, do not reject H 0 Here, 2 = 0..7576 < 2 U = 3.841, so you do not reject H 0 and conclude that there is insufficient evidence that the two proportions are different. 2 U =3.841 0 =.05 Reject H 0 Do not reject H 0

14
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-14 2 Test for The Differences Among More Than Two Proportions Extend the 2 test to the case with more than two independent populations: H 0 : π 1 = π 2 = … = π c H 1 : Not all of the π j are equal (j = 1, 2, …, c)

15
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-15 The Chi-Square Test Statistic where: f o = observed frequency in a particular cell of the 2 x c table f e = expected frequency in a particular cell if H 0 is true 2 for the 2 x c case has (2-1)(c-1) = c - 1 degrees of freedom Assumed: each cell in the contingency table has expected frequency of at least 1 The Chi-square test statistic is:

16
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-16 Computing the Overall Proportion The overall proportion is: Expected cell frequencies for the c categories are calculated as in the 2 x 2 case, and the decision rule is the same: Decision Rule: If 2 > 2 U, reject H 0, otherwise, do not reject H 0 Where 2 U is from the chi-square distribution with c – 1 degrees of freedom

17
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-17 2 Test with More Than Two Proportions: Example The sharing of patient records is a controversial issue in health care. A survey of 500 respondents asked whether they objected to their records being shared by insurance companies, by pharmacies, and by medical researchers. The results are summarized on the following table:

18
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-18 2 Test with More Than Two Proportions: Example Object to Record Sharing Organization Insurance Companies PharmaciesMedical Researchers Yes410295335 No90205165

19
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-19 2 Test with More Than Two Proportions: Example The overall proportion is: Object to Record Sharing Organization Insurance Companies PharmaciesMedical Researchers Yesf o = 410 f e = 346.667 f o = 295 f e = 346.667 f o = 335 f e = 346.667 Nof o = 90 f e = 153.333 f o = 205 f e = 153.333 f o = 165 f e = 153.333

20
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-20 2 Test with More Than Two Proportions: Example Object to Record Sharing Organization Insurance Companies PharmaciesMedical Researchers Yes No The Chi-square test statistic is:

21
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-21 2 Test with More Than Two Proportions: Example Decision Rule: If 2 > 2 U, reject H 0, otherwise, do not reject H 0 2 U = 5.991 is from the chi- square distribution with 2 degrees of freedom. H 0 : π 1 = π 2 = π 3 H 1 : Not all of the π j are equal (j = 1, 2, 3) Conclusion: Since 64.1196 > 5.991, you reject H 0 and you conclude that at least one proportion of respondents who object to their records being shared is different across the three organizations

22
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-22 The Marascuilo Procedure The Marascuilo procedure enables you to make comparisons between all pairs of groups. First, compute the observed differences p j - p j’ among all c(c-1)/2 pairs. Second, compute the corresponding critical range for the Marascuilo procedure.

23
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-23 The Marascuilo Procedure Critical Range for the Marascuilo Procedure:

24
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-24 The Marascuilo Procedure Compute a different critical range for each pair-wise comparison of sample proportions. Compare each of the c(c - 1)/2 pairs of sample proportions against its corresponding critical range. Declare a specific pair significantly different if the absolute difference in the sample proportions |p j – p j’ | is greater than its critical range.

25
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-25 The Marascuilo Procedure Example Object to Record Sharing Organization Insurance Companies PharmaciesMedical Researchers Yes410 P 1 = 0.82 295 P 2 = 0.59 335 P 3 = 0.67 No90205165

26
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-26 The Marascuilo Procedure Example MARASCUILO TABLE Proportions Absolute DifferencesCritical Range | Group 1 - Group 2 |0.230.06831808 | Group 1 - Group 3 |0.150.0664689 | Group 2 - Group 3 |0.080.074485617 Conclusion: Since each absolute difference is greater than the critical range, you conclude that each proportion is significantly different that the other two.

27
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-27 2 Test of Independence Similar to the 2 test for equality of more than two proportions, but extends the concept to contingency tables with r rows and c columns H 0 : The two categorical variables are independent (i.e., there is no relationship between them) H 1 : The two categorical variables are dependent (i.e., there is a relationship between them)

28
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-28 2 Test of Independence where: f o = observed frequency in a particular cell of the r x c table f e = expected frequency in a particular cell if H 0 is true 2 for the r x c case has (r-1)(c-1) degrees of freedom Assumed: each cell in the contingency table has expected frequency of at least 1) The Chi-square test statistic is:

29
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-29 Expected Cell Frequencies Expected cell frequencies: Where: row total = sum of all frequencies in the row column total = sum of all frequencies in the column n = overall sample size

30
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-30 Decision Rule The decision rule is If 2 > 2 U, reject H 0, otherwise, do not reject H 0 Where 2 U is from the chi-square distribution with (r – 1)(c – 1) degrees of freedom

31
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-31 Example: Test of Independence The meal plan selected by 200 students is shown below: Class Standing Number of meals per week Total 20/week10/weeknone Fresh.24321470 Soph.22261260 Junior1014630 Senior14161040 Total708842200

32
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-32 Example: Test of Independence The hypothesis to be tested is: H 0 : Meal plan and class standing are independent (i.e., there is no relationship between them) H 1 : Meal plan and class standing are dependent (i.e., there is a relationship between them)

33
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-33 Example: Test of Independence Class Standing Number of meals per week Total 20/wk10/wknone Fresh.24.530.814.770 Soph.21.026.412.660 Junior10.513.26.330 Senior14.017.68.440 Total708842200 Expected cell frequencies if H 0 is true: Example for one cell:

34
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-34 Example: Test of Independence The test statistic value is: 2 U = 12.592 for α =.05 from the chi-square distribution with (4 – 1)(3 – 1) = 6 degrees of freedom

35
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-35 Example: Test of Independence Decision Rule: If 2 > 12.592, reject H 0, otherwise, do not reject H 0 Here, 2 = 0.709 < 2 U = 12.592, so do not reject H 0 Conclusion: there is insufficient evidence that meal plan and class standing are related. 2 U =12.592 0 =0.05 Reject H 0 Do not reject H 0

36
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-36 McNemar Test Used to test for the difference between two proportions of related samples (not independent) You need to use a test statistic that follows the normal distribution

37
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-37 McNemar Test Contingency Table Condition (Group) 1 Condition (Group) 2 YesNoTotals YesABA+B NoCDC+D TotalsA+CB+Dn Where A = number of respondents who answered yes to condition 1 and condition 2 B = number of respondents who answered yes to condition 1 and no to 2 C = number of respondents who answered no to condition 1 and yes to 2 D = number of respondents who answered no to condition 1 and condition 2 n = number of respondents in the sample

38
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-38 McNemar Test Contingency Table The sample proportions are: Condition (Group) 1 Condition (Group) 2 YesNoTotals YesABA+B NoCDC+D TotalsA+CB+Dn

39
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-39 McNemar Test Contingency Table The population proportions are: π 1 = proportion of the population who answer yes to condition 1 π 2 = proportion of the population who answer yes to condition 2 Condition (Group) 1 Condition (Group) 2 YesNoTotals YesABA+B NoCDC+D TotalsA+CB+Dn

40
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-40 McNemar Test Test Statistic To test the hypothesis: H 0 : π 1 = π 2 H 1 : π 1 ≠ π 2 Use the test statistic:

41
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-41 McNemar Test Example Suppose you survey 300 homeowners and ask them if they are interested in refinancing their home. In an effort to generate business, a mortgage company improved their loan terms and reduced closing costs. The same homeowners were again surveyed. Determine if change in loan terms was effective in generating business for the mortgage company. The data are summarized as follows:

42
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-42 McNemar Test Example Survey response before change Survey response after change YesNoTotals Yes1182120 No22158180 Totals140160300 Test the hypothesis (at the 0.05 level of significance): H 0 : π 1 ≥ π 2 : The change in loan terms was ineffective H 1 : π 1 < π 2 : The change in loan terms increased business

43
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-43 McNemar Test Example Survey response before change Survey response after change YesNoTotals Yes1182120 No22158180 Totals140160300 The critical value (.05 significance) is Z = -1.96 The test statistic is: Since Z = -4.08 < -1.96, you reject H 0 and conclude that the change in loan terms significantly increase business for the mortgage company.

44
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-44 Wilcoxon Rank-Sum Test Test two independent population medians Populations need not be normally distributed Distribution free procedure Used when only rank data are available

45
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-45 Wilcoxon Rank-Sum Test Can use when both n 1, n 2 ≤ 10 Assign ranks to the combined n 1 + n 2 sample values If unequal sample sizes, let n 1 refer to smaller-sized sample Smallest value rank = 1, largest value rank = n 1 + n 2 Assign average rank for ties Sum the ranks for each sample: T 1 and T 2 Obtain test statistic, T 1 (from smaller sample)

46
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-46 Checking the Rankings The sum of the rankings must satisfy the formula below Can use this to verify the sums T 1 and T 2 where n = n 1 + n 2

47
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-47 Wilcoxon Rank-Sum Test Hypothesis and Decision H 0 : M 1 = M 2 H 1 : M 1 ≠ M 2 H 0 : M 1 ≤ M 2 H 1 : M 1 > M 2 H 0 : M 1 ≥ M 2 H 1 : M 1 < M 2 Two-Tail TestLeft-Tail TestRight-Tail Test M 1 = median of population 1; M 2 = median of population 2 Reject T 1L T 1U Reject Do Not Reject Reject T 1L Do Not Reject T 1U Reject Do Not Reject Test statistic = T 1 (Sum of ranks from smaller sample) Reject H 0 if T 1 ≤ T 1L or if T 1 ≥ T 1U Reject H 0 if T 1 ≤ T 1L Reject H 0 if T 1 ≥ T 1U

48
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-48 Wilcoxon Rank-Sum Test Example with Small Samples Sample data are collected on the capacity rates (% of capacity) for two factories. Are the median operating rates for two factories the same? For factory A, the rates are 71, 82, 77, 94, 88 For factory B, the rates are 85, 82, 92, 97 Test for equality of the population medians at the 0.05 significance level

49
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-49 Wilcoxon Rank-Sum Test Example with Small Samples CapacityRank Factory AFactory BFactory AFactory B 711 772 823.5 823.5 855 886 927 948 979 Rank Sums:20.524.5 Tie in 3 rd and 4 th places Ranked Capacity values:

50
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-50 Wilcoxon Rank-Sum Test Example with Small Samples Factory B has the smaller sample size, so the test statistic is the sum of the Factory B ranks: T 1 = 24.5 The sample sizes are: n 1 = 4 (factory B) n 2 = 5 (factory A) The level of significance is α =.05

51
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-51 Wilcoxon Rank-Sum Test Example with Small Samples n2n2 n1n1 One- Tailed Two- Tailed 45 4 5.05.1012, 2819, 36.025.0511, 2917, 38.01.0210, 3016, 39.005.01--, --15, 40 6 Lower and Upper Critical Values for T 1 from Appendix Table E.8: T 1L = 11 and T 1U = 29

52
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-52 Wilcoxon Rank-Sum Test Example with Small Samples H 0 : M 1 = M 2 H 1 : M 1 ≠ M 2 Two-Tail Test Reject T 1L =11T 1U =29 Reject Do Not Reject Reject H 0 if T 1 ≤ T 1L = 11 or if T 1 ≥ T 1U = 29 =.05 n 1 = 4, n 2 = 5 Test Statistic (Sum of ranks from smaller sample): T 1 = 24.5 Decision: Conclusion: Do not reject at a = 0.05 There is not enough evidence to prove that the medians are not equal.

53
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-53 Wilcoxon Rank-Sum Test Using Normal Approximation For large samples, the test statistic T 1 is approximately normal with mean and standard deviation: Must use the normal approximation if either n 1 or n 2 > 10 Assign n 1 to be the smaller of the two sample sizes Can use the normal approximation for small samples

54
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-54 Wilcoxon Rank-Sum Test Using Normal Approximation The Z test statistic is Where Z approximately follows a standardized normal distribution

55
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-55 Wilcoxon Rank-Sum Test Using Normal Approximation Use the setting of the prior example: The sample sizes were: n 1 = 4 (factory B) n 2 = 5 (factory A) The level of significance was α =.05 The test statistic was T 1 = 24.5

56
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-56 Wilcoxon Rank-Sum Test Using Normal Approximation The test statistic is Z = 1.10 is less than the critical Z value of 1.96 (for α =.05) so you do not reject H 0 – there is not sufficient evidence that the medians are not equal

57
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-57 Kruskal-Wallis Rank Test Tests the equality of more than 2 population medians Use when the normality assumption for one-way ANOVA is violated Assumptions: The samples are random and independent variables have a continuous distribution the data can be ranked populations have the same variability populations have the same shape

58
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-58 Kruskal-Wallis Rank Test Obtain overall rankings for each value In event of tie, each of the tied values gets the average rank Sum the rankings for data from each of the c groups Compute the H test statistic

59
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-59 Kruskal-Wallis Rank Test The Kruskal-Wallis H test statistic: (with c – 1 degrees of freedom) where: n = total number of values over the combined samples c = Number of groups T j = Sum of ranks in the j th sample n j = Size of the j th sample

60
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-60 Kruskal-Wallis Rank Test Decision rule Reject H 0 if test statistic H > 2 U Otherwise do not reject H 0 Complete the test by comparing the calculated H value to a critical 2 value from the chi-square distribution with c – 1 degrees of freedom 2U2U 0 Reject H 0 Do not reject H 0

61
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-61 Kruskal-Wallis Rank Test Example Do different branch offices have a different number of employees? Office size (Chicago, C) Office size (Denver, D) Office size (Houston, H) 23 41 54 78 66 55 60 72 45 70 30 40 18 34 44

62
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-62 Kruskal-Wallis Rank Test Example Do different branch offices have a different number of employees? Office size (Chicago, C) Ranking Office size (Denver, D) Ranking Class size (Houston, H) Ranking 23 41 54 78 66 2 6 9 15 12 55 60 72 45 70 10 11 14 8 13 30 40 18 34 44 3514735147 = 44 = 56 = 20

63
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-63 Kruskal-Wallis Rank Test Example The H statistic is

64
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-64 Kruskal-Wallis Rank Test Example Since H = 6.72 >, reject H 0 Compare H = 6.72 to the critical value from the chi-square distribution for 3 – 1 = 2 degrees of freedom and =.05: There is evidence of a difference in the population median number of employees in the branch offices

65
Statistics for Managers Using Microsoft Excel, 5e © 2008 Prentice-Hall, Inc.Chap 12-65 Chapter Summary Developed and applied the 2 test for the difference between two proportions Developed and applied the 2 test for differences in more than two proportions Examined the 2 test for independence Used the McNemar test for differences in two related proportions Used the Wilcoxon rank sum test for two population medians Small Samples Large sample Z approximation Applied the Kruskal-Wallis H-test for multiple population medians In this chapter, we have

Similar presentations

© 2021 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google