Choice of study design: randomized and non-randomized approaches Iná S. Santos Federal University of Pelotas Brazil PAHO/PAHEF WORKSHOP EDUCATION FOR CHILDHOOD OBESITY PREVENTION: A LIFE-COURSE APPROACH Aruba, June 2012

Outline of the presentation Introduction –Types of evidence –Internal and external validity Randomized controlled trials Non-randomized designs Victora et al. Evidence-based Public Health: moving beyond randomized trials. Am J Public Health 2004;94(3): Habicht JP et al. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Intern J Epidemiology 1999;28:10-18

Part I Introduction –Types of evidence –Internal and external validity

Types of epidemiological evidence for Public Health Type of evidenceType of epidemiological study Frequency of diseaseDescriptive Frequency of exposureDescriptive Exposure/disease relationship Experimental (or observational) Coverage of interventionDescriptive Efficacy of interventionExperimental (or observational) Programme effectivenessObservational

Valididy: internal and external External population Target population Actual population Sample

Validity Internal validity –Are the study results true for the target population? –Are there errors that affect the study findings? Systematic error (bias, confounding) Random error (precision) External validity –Generalizability –Are the study results applicable to other settings?

Validity Internal validity –May be judged on the basis of the study methods External validity –Require a “value judgment”

Part II Randomized controlled trials (RCTs)

Internal validity in probability studies Issue: Comparability of Probability study (RCT) Bias avoided PopulationsRandomizationSelection bias ObservationsBlindingInformation bias EffectsUse of placeboHawthorne effect Placebo effect RCTs are the gold standard for internal validity

RCT (from Cochrane Collaboration) In a RCT participants are assigned by chance to receive either an experimental or control treatment. When a RCT is done properly, the effect of a treatment can be studied in groups of people who are the same at the outset, and treated in the same way, except for the intervention being studied. Any differences then seen in the groups at the end of the trial can be attributed to the difference in treatment alone, and not to bias or chance.

Randomised controlled trials Prioritise internal validity –random allocation reduces selection bias and confounding –blinding reduces information bias Gained popularity through clinical trials of new drugs Essential for determining efficacy of new biological agents Adequate for short causal chains –biological effects of drugs, vaccines, nutritional supplements, etc. drug  pharmacological reaction  disease cure or alleviation

Pooling data from RCTs Systematic review –Comprehensive search for all high-quality scientific studies on a specific subject E.g. on effects of a drug, vaccine, surgical technique, behavioral intervention, etc Meta-analysis –Groups data from different studies to determine an average effect –Improves the precision of the available estimates by including a greater number of people –But: data from different studies cannot always be combined

What does a RCT show? The probability that the observed result is due to the intervention But additional evidence is required to make this result conceptually plausible –Biological plausibility –Operational plausibility

Special issues in RCTs “Intent-to-treat” analyses –Individuals/groups should remain in the group to which they were originally assigned Units of analyses –It is incorrect to use group allocation (e.g., health centers, communities, etc) and to analyse the data at individual level –This has implications for sample size calculation and for analysis methods

CONSORT Statement Allocation Rationale Eligibility Interventions Objectives Outcomes Sample size Randomization –Sequence generation –Concealment –Implementation –Blinding (masking) Statistical methods Participant flow Recruitment Baseline data Numbers analyzed Outcomes and estimation Ancillary analyses Adverse events Interpretation Generalizability Overall evidence

Major steps in Public Health trials Central-level provision of intervention to local outlets (e.g. health facilities) Local providers’ compliance with delivery of intervention Recipient compliance with intervention Biological effect of intervention Source: Victora, Habicht, Bryce, AJPH 2004

Example of Public Health Intervention: Nutrition Counselling Trial Health workers are trained Nutritional status improves HW knowledge increases HW performance improves Maternal knowledge increases Child diets change Energy intake increases National programme is implemented Source: Santos, Victora et al. J Nutr 2001 Utilization is adequate HWs are trainable Equipment is available Food is available Central team is competent Lack of food is a cause of malnutrition

Example of Public Health Intervention: Nutrition Counselling Trial Health workers are trained Nutritional status improves HW knowledge increases HW performance improves Maternal knowledge increases Child diets change Energy intake increases National programme is implemented Source: Santos, Victora et al. J Nutr =0.21

Are RCT findings generalizable to routine programmes? The dose of the intervention may be smaller –behavioural effect modification provider behaviour recipient behaviour The dose- response relationship may be different –biological effect modification The longer the causal chain, the more likely is effect modification Source: Victora, Habicht, Bryce, AJPH 2004

Curvilinear associations Trials often done here Results often applied here Source: Victora, Habicht, Bryce, AJPH 2004

Why do RCTs have a limited role in large-scale effectiveness evaluations Often impossible to randomize –unethical, politically unacceptable, rapid scaling up Evaluation team affects service delivery –service delivery is at least “best-practice” Effect modification is the rule –are meta-analyses of complex programmes meaningful? –need for local data Need for supplementary approaches for evaluations in Public Health

Part III Non-randomized designs (Quasi-experiments)

Types of inference in impact evaluations Adequacy (descriptive studies) – the expected changes are taking place Plausibility (observational studies) – observed changes seem to be due to the programme Probability (RCTs) – randomised trial shows that the programme has a statistically significant impact Source: Habicht, Victora, Vaughan, IJE 1999

Ensuring internal validity in probability and plausibility studies Issue: Comparability Probability (RCT) Plausibility (quasi-experiment) PopulationsRandomizationMatching Understanding determinants of implementation Handling contextual factors ObservationsBlindingAvoiding information bias EffectsUse of placeboBeing aware of Hawthorne bias and of the placebo effect

Adequacy evaluations Questions: –Were the initial goals achieved? E.g.: reduce underfive mortality by 20% –Were the observed trends in impact indicators in the expected direction? of adequate magnitude?

Plausibility evaluations Question: –Is the observed impact likely due to the intervention? Require ruling out influence of external factors: –need for comparison group –adjustment for confounders Also known as quasi-experiments

Adequacy/plausibility designs (1) Design: cross-sectional Measurement points: once Outcome: difference or ratio Control group: –Individuals who did not receive the intervention –Groups/areas without the intervention –Dose-response analyses, if possible

ORT and diarrhea deaths in Brazil Spearman r = -0,61 (p=0,04) Each dot = 1 state

Adequacy/plausibility designs (2) Design: longitudinal (before-and-after) Measurement points: twice or more Outcome: change Control group: –The same or similar individuals, before the intervention –The same groups/areas, before the intervention –Time-trend analyses, if possible

Hib vaccine in Uruguay In Uruguay, reported Hib cases declined by over 95 percent after the introduction of routine infant Hib immunisation in Source: PAHO, 2004

Adequacy/plausibility designs (3) Design: longitudinal-control Measurement points: twice or more Outcome: relative change Control group: –The same or similar individuals, before the intervention –The same groups/areas, before the intervention –Time-trend analyses, if possible

Adequacy/plausibility designs (4) Design: case-control Measurement points: once Comparison: exposure to intervention Groups: –Cases: individuals with the disease of interest –Controls: sample of the population from which cases originated

Stunting in Tanzania Source: Schellenberg J et al Stunting prevalence among children aged months p (mean haz) = 0.05

Transparent Reporting for Evaluations with Nonrandomised Designs (TREND) Similar to CONSORT guidelines Include –conceptual frameworks used –intervention and comparison conditions –research design –methods of adjusting for possible biases AJPH, March 2004 Source: Des Jarlais, Lyles, Crepaz and the TREND Group, AJPH 2004

Conclusions (1) RCTs are essential for –clinical studies –community studies for establishing the efficacy of relatively simple interventions RCTs require additional evidence from non-randomised studies for increasing their external validity

Conclusions (2) Given the complexity of many Public Health interventions, adequacy and plausibility studies are essential in different populations –even for interventions proven by RCTs Adequacy evaluations should become part of the routine of decision-makers –and plausibility evaluations too, when possible

THANK YOU