1. Tim Wiemken, PhD MPH CIC Assistant Professor Division of Infectious Diseases University of Louisville, Kentucky Planning Your Study Statistical Issues

2. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

3. Video

4. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

5. Types of Analysis DescriptiveDescribes a sample

6. Types of Analysis Descriptive

7. Types of Analysis Inferential/An alytical Analyzes associations, predictions, differences

8. Types of Analysis Inferential/An alytical STATISTICAL SIGNIFICANCE

9. Types of Analysis Inferential/An alytical

10. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

11. Hypothesis Testing It is very hard to prove something causes something else

12. Hypothesis Testing 2.Hypothesis testing for inferential statistics It is much easier to disprove that something doesn’t cause something else.

13. Types of Analysis 2.Hypothesis testing for inferential statistics EXAMPLE Innocent until proven guilty (misnomer)

14. Hypothesis Testing 2.Hypothesis testing for inferential statistics EXAMPLE Innocent until proven guilty (misnomer) Should be: Innocent until proven not innocent

15. Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence.

16. Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence. b. Disprove innocence beyond a reasonable doubt.

17. Hypothesis Testing 2.Hypothesis testing for inferential statistics a. Assume innocence. b. Disprove innocence beyond a reasonable doubt. c. The conclusion you are left with is guilty.

18. Hypothesis Testing 2.Hypothesis testing for inferential statistics Hypothesis tests work the same way

19. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis

20. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Not guilty Guilty

21. Hypothesis Testing 2.Hypothesis testing for inferential statistics Does azithromycin cause cardiac complications?

22. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis

23. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

24. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Hard to prove Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

25. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis Easy to to disprove Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

26. Hypothesis Testing 2.Hypothesis testing for inferential statistics Null hypothesis Alternative hypothesis If disproven, only option is to accept the alternative! Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death

27. Hypothesis Testing Disproving (rejecting) a null hypothesis is done using statistical tests. Bivariate Tests (univariate) Chi-squared Student’s t-test Fisher’s Exact test Mann-Whitney U-test ANOVA

28. Hypothesis Testing Disproving (rejecting) a null hypothesis is done using statistical tests. Bivariate Tests (univariate) Chi-squared Student’s t-test Fisher’s Exact test Mann-Whitney U-test ANOVA Multivariate Tests (regression) Linear regression Logistic regression Cox Regression Poisson Regression

29. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

30. Error Reject Null Do not reject Null TrueFalse Null Hypothesis (reality) StudyStudy Correct

31. Error Do not reject the null when it is indeed true Null hypothesis Azithromycin is not associated with cardiac death This is the truth Correct!

32. Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Null Hypothesis (reality)

33. Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is false Reject! This is the truth Correct!

34. Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality)

35. Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is true = incorrectly conclude there is an association Reject incorrectly! This is the truth Incorrect!

36. Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β)

37. Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Do not reject the null when it is false= incorrectly conclude there is no association This is the truth Incorrect! Do not reject

38. Error What you need to know α= level of significance Pretty much always 0.05 or 5%

39. Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Reject the null when it is true = incorrectly conclude there is an association Reject incorrectly! This is the truth Incorrect!

40. Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality) Willing to accept a 5% chance of rejecting the null when it is true

41. Error Reject Null Do not reject Null TrueFalse StudyStudy Correct Error I (α) Null Hypothesis (reality) 1 in 20 studies (5%) will lead to a false conclusion that there is an association when there actually is not

42. Error What you need to know β Type II error – usually 0.2 or 20%

43. Error Null hypothesis Alternative hypothesis Azithromycin is not associated with cardiac death Azithromycin is associated with cardiac death Do not reject the null when it is false= incorrectly conclude there is no association This is the truth Incorrect! Do not reject

44. Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β) Willing to accept a 20% chance of not rejecting the null when it is false

45. Error Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Error I (α) Error II (β) 1 in 5 studies (20%) will lead to a false conclusion that there is not an association when there actually is

46. Power What you need to know β More importantly, 1-β = statistical power

47. Power What you need to know 1-β Power = the ability to detect an association when one is present

48. Power Reject Null Do not reject Null TrueFalse Null Hypothesis StudyStudy Correct Correct (Power) Error I (α) Error II (β) 4 in 5 studies (80% Power) will detect an association when one is really present

49. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

50. Effect Size Effect Size = Smallest difference between your study groups worth saying is statistically significant.

51. Effect Size 1.Effect size is critical to be able to properly calculate the sample size.

52. Effect Size 2.Utilize data from pilot studies or literature to determine an effect size

53. Effect Size Usually takes the format: –20% mortality in group 1 –10% mortality in group 2 10% Effect Size (20%-10%)

54. Effect Size Usually takes the format: –Average blood glucose in group 1: 200mg/dL –Average blood glucose in group 2: 150mg/dL Effect Size = 50mg/dL

55. Objectives 1.Describe types of statistical analysis 2.Provide an overview of hypothesis testing 3.Understand type 1 error, type 2 error, and statistical power 4.Describe effect size 5.Understand how to calculate sample size

56. Example 1 Design: Unblinded randomized controlled trial

57. Example 1 Design: Unblinded randomized controlled trial Population: Stool transplantation versus Oral Vancomycin for C. difficile infection

58. Question: Is stool transplantation more effective than oral vancomycin at preventing C. difficile infection relapse? Example 1 Design: Unblinded randomized controlled trial Population: Stool transplantation versus Oral Vancomycin for C. difficile infection

59. From the literature: Stool transplantation will reduce relapse from X% (oral vancomycin alone) to Y% (stool transplantation alone). Example 1

60. Null hypothesis: ?? Example 1 Alternative hypothesis: ??

61. Null hypothesis: Stool transplant is not associated with less C. difficile relapse than oral vancomycin. Example 1 Alternative hypothesis: Stool transplants are associated with less C. difficile relapse than oral vancomycin.

62. Example 1 Sample Size: How many patients do you need?

63. What you need to know: What percent of relapse do you expect in each group? Example 1 Sample Size: How many patients do you need?

64. What you need to know: What percent of relapse do you expect in each group? Example 1 Sample Size: How many patients do you need? Pilot study or literature review (or guess)!

65. Example 1 Back to our example. Are we testing the difference between proportions or means?

66. Example 1 Back to our example. Are we testing the difference between proportions or means?

67. Example 1 How many patients do you need? Left side: 20% (standard of care) 10% (treatment group) Right side: 20% (standard of care) 15% (treatment group)

68. Example 1 How many patients do you need? Left side: 20% (standard of care) 10% (treatment group) 199 per group Total = 398 Right side: 20% (standard of care) 15% (treatment group) 905 per group Total = 1809

69. Example 1 If we are “powered” to detect a 10% difference 199 per group Total = 398 If there is actually a 5% difference, we won’t have enough power to detect the difference = lack of significance may be type II error

70. Example 2 Hypothesis: Patients who come to clinic at least 3 times a year will have an increase in CD4+ cell count. The average CD4 will be X if they do come to clinic and Y if they do not.

71. What you need to know: What is the average CD4 count you expect in each group? Example 2 Sample Size: How many patients do you need?

72. What you need to know: What is the average CD4 count you expect in each group? Example 2 Sample Size: How many patients do you need? Left side: 750 (≥3 visits/year) 500 (<3 visits/year) Right side: 750 (≥3 visits/year) 650 (<3 visits/year)

73. What you need to know: What is the standard deviation for both groups? Example 2 Sample Size: How many patients do you need? Left side: 200 Right side: 200

74. Example 2 Sample Size: How many patients do you need? Left side: Total: 21 Right side: Total: 126

75. Example 2 We have seen what mean alterations do to the sample size. Left side: Total: 21 Right side: Total: 126

76. Example 2 What happens if we change the standard deviation? Use 300 instead of 200

77. Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Left side: Total: 46 (from 21) Right side: Total: 283 (from 126)

78. Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Use 100 instead of 200

79. Example 2 What happens if we change the standard deviation? Use 300 instead of 200 Use 100 instead of 200 Left side: Total: 6 (from 21) Right side: Total: 32 (from 126)

80. At the end of the day… FOR AN UNDERPOWERED STUDY You don’t know if there is not an association or if your non-significant findings are due to your small sample size. Under- powered I’m not significant!

81. At the end of the day… Studies can be OVERPOWERED Any difference can be found to be statistically significant. Over- powered I’m ALWAYS significant!

82. At the end of the day… Studies can be OVERPOWERED There is a difference between statistically significant and clinically significant.

83. Conclusion Sample size estimations are important

84. Conclusion Obtaining appropriate estimates of proportions or means/standard deviations is CRITICAL

85. Conclusion Improper estimation of these values will lead to error

86. Conclusion Call your statistician early in the planning process.

87. Conclusion Remember your statistician is not a magician. He or she will give you a sample size based the values you provide.

88. Questions?