1. Sample Size Considerations for Answering Quantitative Research Questions Lunch & Learn May 15, 2013 M Boyle

2. Fiscal Year 2007 68.8 Fiscal Year 2008 110.9 Fiscal Year 2009 179.8 Fiscal Year 2010 193.9 Fiscal Year 2011 191.1 Sum 744.6 National Children’s Study in the US Proposed Birth Cohort 100,000 to age 21 Planning Costs 2000-2006: $54.7M Implementation Costs 2007-2011: $744.6 Sample Size Justification: ?

3. What is Statistical Power? The statistical power of a test is the probability of correctly rejecting H 0 when it is false. In other words, power is the likelihood that you will identify a statistically significant effect when one exists

4. Types of Power Analysis A priori : Used to plan a study (Question: What sample size is needed to obtain a certain level of power)? Post hoc : Used to evaluate a study faced with a constrained sample size (Question: Do you have a large enough sample to detect a meaningful effect)? [Types of constraints: (1) a completed study; (2) a proposed study with limited number of eligible subjects; (3) a proposed study faced with limited resources]

5. Elements of Power Calculations Effect size ∆ Measurement variability SD Type I error Alpha ( α) typically specified at p=0.05, 2-tailed Type II error Beta ( β) typically specified at p=0.20 Power = 1- β; typically 0.80 Sample Size

6. Hypothesized distributions, effect sizes and error rates Effect Size ∆ Type I Type II Measurement Variability +/- 1 SD

7. Decisions Medical Diagnosis Hypothesis Testing Disease Status Present Absent Population Status H 0 false H 0 true [H 1 true] +ve Test Result -ve Accept H 0 Decision Reject H 0 [Accept H 1 ] correct false +ve false -ve correct correct Type II 1-α β Type I correct α 1-β (power)

8. Example Power Calculation H 0 : At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care. H 1 : At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher. Effect size ∆ = SD = Alpha ( α) = Beta ( β) = Power = Sample Size ?

9. Example Power Calculation H 0 : At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care. H 1 : At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher. Effect size ∆ = 5 SD = 15 Alpha ( α) = 0.05 2-tailed Beta ( β) = 0.20 Power = 80 Sample Size 146 per group

10. n=292

11. FACTORS THAT INFLUENCE SAMPLE SIZE PLANNING AND STATISTICAL POWER

12. Sample Size Planning and Power 1. Error rates Type I ( α) -smaller α requires larger sample sizes -2-sided tests requires larger sample sizes Type II ( β) Statistical power: -smaller β (more power) requires larger sample sizes [Use conventional levels & worry about the trade- offs between effect size and sample size]

13. Sample Size Planning and Power 2. Effect Size ∆ “ What is the minimally important effect based on clinical, biological or social implications of the findings?”

14. Sample Size Planning and Power 1.Effect size ∆ What do you know about the nature of the effect – its scale of measurement and its perceived importance to practice, policy, resource allocation (e.g., infant mortality; dollars; self-esteem)? What do previous empirical studies tell you about achievable effects?

15. Sample Size Planning and Power 1. Effect Size ∆ Can you generate a consensus among your investigative team on a minimally important effect? Is it reasonable to use conventional estimates of small, medium and large? Are you limited by the dollar amount you can request?

16. Sample Size Planning and Power 2.The measurement scale of the dependent variable: discrete, ordinal, interval -interval level measurements require smaller samples 3.The variability of the dependent variable in the general population (SD, Variance) -lower variability requires smaller sample sizes

17. Sample Size Planning and Power 4.The statistical test -simple estimation; differences between groups; correlation and prediction. The test must be appropriate for the question and data. A key element in sample size planning 5. Sample distribution, for example, exposed versus not exposed) -balanced is the most powerful

18. Sample Size Planning and Power 6. Attrition loss of subjects -higher attrition leads to lower power 7. Measurement reliability -complicated: if true variance is constant and error variance is reduced statistical power will increase

19. Sample Size Planning and Power 8. Study costs – what the market will bear 9. Analytical complexity – what to do when your models require much more information than you can get?

20. Adding Complexity Multilevel Model y ij = β 0j + β 1 z 0j + (u 0j + e ij ) H 0 The association between neighbourhood affluence measured on resident 4-16 year olds in 1983 and years of education assessed in 2001 will be = 0.00 standard units Neigh Affluence x y H 0 ∆ = β 1 z 0j > 0.20

21. Estimates 2-level balanced data, nested model Significance level = 0.025 (to get 0.05 2-tailed) Number of simulations per setting = 100 Response variable = normal Estimation method = IGLS Fixed intercept = yes Random intercept = yes Number of explanatory variables = 1 Type of predictor = continuous

22. Estimates Mean of the predictor = 0.0 Variance of the predictor at level 1 = 0.0 Variance of the predictor at level 2 = 1.0 Smallest/Largest # units at L1 (increment) Smallest/Largest # units at L2 (increment) Estimate β 0 = 0 Estimate β 1 = 0.15 Estimate L2 variance 0.05 Estimate L1 variance 0.95

24. Comments Ask specific, quantifiable research questions Consult with colleagues about clinical, biological and social importance of your outcomes Move from simple to complex hypotheses. Complex models – SEM, Multilevel – can require you to provide an enormous number of parameters. When estimating sample size requirements for complex models, you will inevitably use standardized variables

25. Comments Estimating sample size requirements is part game, subject to practical constraints (limited resources and subjects) and convincing reviewers that you know what your doing Take a ‘reasoned’ approach – most reviewers will have no clue what you are going on about The hardest part of the process is acquiring the information you need.