ECON ECON Health Economic Policy Lab Kem P. Krueger, Pharm.D., Ph.D. Anne Alexander, M.S., Ph.D. University of Wyoming

Overview of Study Designs & Threats to Internal and External Validity

Objectives  Define internal and external validity and explain the relationship between the two  Describe each type of extraneous variables that threaten the internal validity of a study

Objectives  Describe each of the following study designs and explain the strengths and weaknesses of each one: an experimental study an experimental study a quasi-experimental study a quasi-experimental study a case-control epidemiology study a case-control epidemiology study a cohort epidemiology study a cohort epidemiology study a correlational study a correlational study a descriptive study a descriptive study

Internal Validity The minimum requirements that are necessary for a study to produce results that are believable.

Threats to Internal Validity HistoryMaturationTestingInstrumentation Statistical regression Selection bias Experimental mortality

History  A Specific event or events that occur between the first and last observation in an experiment that could produce a change in the dependent variable.  This change could be erroneously attributed to the intervention (the independent variable).

Maturation  A change occurring in the study subjects as a function of time rather than a function of a specific event.  If this change impacts the dependent variable, it may be erroneously attributed to the intervention (the independent variable).

Testing  When subjects are tested or measured multiple times during an experiment, their scores on subsequent tests or measures may improve because they remember previous answers or they are more skilled at the task they are performing because due to practice.  This improvement may be erroneously attributed to the intervention (ind var).

Instrumentation  Changes in the calibration of a measurement instrument or changes in observers during a study may produce changes in the independent variable that are erroneously attributed to the intervention.

Statistical Regression  If people in a group have extremely high scores they will most likely “regress to the mean” on their own.  The change in the score may be erroneously attributed to the intervention.

Selection Bias  Differential selection of subjects  If severely ill subjects are differentially selected for the placebo group and mildly ill subjects are selected for the intervention group, the difference between the groups could be erroneously attributed to the intervention.

Experimental Mortality  The differential loss of subjects from one group may affect the independent variable.  If severely ill patients drop out of the intervention group because they cannot tolerate the side effects of the study drug, the average health status of that group will improve because the sick people are gone. This improvement may be erroneously attributed to the drug (independent variable).

External Validity  External validity deals with the generalizability of the results of the study. To what population and setting can the results of this study be applied?

Study Designs

Levels of Evidence  Experimental Designs Randomized Controlled Studies Randomized Controlled Studies Crossover Studies Crossover Studies  Quasi-Experimental Designs Case Control Case Control Cohort Studies Cohort Studies  Correlational Studies  Desciptive Studies Case Reports Case Reports Cross Sectional Surveys Cross Sectional Surveys

Experimental Designs  Randomized controlled trial  Pretest-posttest control group design ROXO ROO  Double or single blind

Experimental Designs  Advantages Control for all threats to internal validity Control for all threats to internal validity Random selection or assignment is used so there is an unbiased distribution of confounders Random selection or assignment is used so there is an unbiased distribution of confounders Blinding usually occurs in RCTs (double or single) Blinding usually occurs in RCTs (double or single) Prospective data collection Prospective data collection Can be used to determine cause and effect Can be used to determine cause and effect

Experimental Designs  Disadvantages Expensive (time and money) Expensive (time and money) If volunteers are recruited they may not represent the population. If volunteers are recruited they may not represent the population. Placebo controlled studies may be unethical Placebo controlled studies may be unethical

Crossover Studies (An Experimental Design)  Randomized trial, each subject serves as his / her own control ROX 1 O X 2 O ROX 2 O X 1 O  Double or single blind with a washout period after first treatment  Subject’s response X 1 is compared to his / her response to X 2

Crossover Studies (An Experimental Design)  Advantages Since the subjects serve as their own control, intra-group variation is lower and fewer subjects are needed Since the subjects serve as their own control, intra-group variation is lower and fewer subjects are needed Random selection or assignment is used so there is an unbiased distribution of confounders Random selection or assignment is used so there is an unbiased distribution of confounders Blinding usually occurs (double or single) Blinding usually occurs (double or single) Threats to internal validity are controlled Threats to internal validity are controlled Can determine cause and effect Can determine cause and effect

Crossover Studies (An Experimental Design)  Disadvantages Carry over / order effects Carry over / order effects Order of treatment may matter Order of treatment may matter Cross over timing Cross over timing Subjects have more opportunities to drop out Subjects have more opportunities to drop out If volunteers are recruited they may not represent the population. If volunteers are recruited they may not represent the population.

Quasi-Experimental Designs  Advantages Randomization may not always be possible, so this is the next strongest study design Randomization may not always be possible, so this is the next strongest study design Controls for most threats to internal validity Controls for most threats to internal validity Confounding variables can be controlled by matching the groups on key variables Confounding variables can be controlled by matching the groups on key variables

Quasi-Experimental Designs  Disadvantages Cause and effect cannot be established Cause and effect cannot be established Selection bias may occur if the researcher is not careful Selection bias may occur if the researcher is not careful

Cohort Studies Exposure Disease ? ? ExposureDisease ? ? Prospective Cohort Study Retrospective Cohort Study

Measures of Disease Association  Relative Risk (RR): estimates the magnitude of an association between exposure and disease and indicates the likelihood of developing the disease in the exposed group relative to those who are not exposed

Cohort Studies  Disadvantages Identifying controls is difficult Identifying controls is difficult Exposure may be linked to a hidden confounder Exposure may be linked to a hidden confounder Blinding is difficult Blinding is difficult No randomization No randomization Large sample size is needed for rare diseases Large sample size is needed for rare diseases

Case-Control Studies ExposureDisease ? ? Measures of Disease Association Odds Ratio (OR): in a case-control study it is not possible to estimate the RR, so it must be estimated using an odds ratio

Measures of Disease Association  Odds Ratio (OR): in a case-control study it is not possible to estimate the RR, so it must be estimated using an odds ratio

Case-Controlled Studies  Advantages Relatively easy to conduct (save time & $) Relatively easy to conduct (save time & $) Few ethical problems Few ethical problems  Disadvantages Cannot prove causality Cannot prove causality Study is susceptible to recall bias and selection bias Study is susceptible to recall bias and selection bias Confounders may not be controlled Confounders may not be controlled

Correlational Studies  Look for associations or relationships between variables  Advantages Often use secondary data Large samples  Disadvantages Limited by available data Cannot determine cause and effect relationship

Descriptive Studies  Describe the situation at a given point in time and may identify things for further study  Advantages May be easier to conduct Sample may be easy to identify  Disadvantages Data collection instrument must be valid and reliable Cannot determine cause and effect relationship

Questions?