1. Quasi-Experimental Designs 101: What Works? The Need To Know Team January 31 – February 1, 2005 Patricia J. Martens PhD

2. Outline Reviewing X’s and O’s Quasi-experimental time series designs with comparison groups The Population Health Research Data Repository: what data do we have? Brainstorming ideas

3. Key features of study designs Artificial manipulation? (experimental or observational) Experimental: Are the groups randomly assigned to receive or not receive the intervention? (randomized controlled trial) Are the groups selected to be as similar as possible, not randomly? (quasi-experimental comparison groups)

4. Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

5. Key Features of Study Designs Observational: – Information collected concurrently or over a time period? (cross-sectional or longitudinal) – If over a time period, i.e. longitudinal, do you go from exposure to disease (cohort) or from disease back in time to examine exposures (case- control)? – Do you start now and go forward (prospective), or do you have a “cohort” somewhere in the past and you follow them forward (historical prospective)?

6. Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

7. Study design: observational Cross-sectional studies – studying all factors at once - both the hypothesized explanatory and outcome variables Prospective studies – going forward in time, following a cohort and observing the effect of exposure to a future outcome Case-control studies – going backwards in time from the cases/controls to look at differential exposures

8. Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

9. Study design: “What Works” proposal Randomized Controlled (Clinical) Trial – designing a specific intervention and randomly assigning people to receive it or not to receive it Quasi-experimental – using a comparison group which is not randomly assigned – Each RHA is a comparison group – A quasi-experimental time series with many comparison groups (all other RHAs in the province) Diagrammed and described by Campbell & Stanley (1963)

10. X is an intervention O is an outcome measure X O Let’s play X’s and O’s

11. O X O Let’s play X’s and O’s

12. O X O O Let’s play X’s and O’s

13. R means randomly assigned RO X O R O O (pretest-posttest control group design) Let’s play X’s and O’s

14. _ _ _ _ means not randomly assigned (quasi-experimental comparison) O X O - - - - O Let’s play X’s and O’s

15. O X O - - - - O quasi-experimental pretest- posttest design (non-randomized control group) (non-equivalent pretest-posttest comparison group design) Let’s play X’s and O’s

16. 0 10 20 30 40 12 Hospital BFHI Compliance Scores Time (8 month interval) BFHI Compliance site Arborg Pine Falls Ten Steps and WHO Code each assigned 4 points, for total compliance of 44 control intervention Split-unit anova: p=0.0009 Martens 2001 Examples of a quasi-experimental pretest- posttest comparison group study to determine effectiveness of hospital policy/education program

17. O O X O O Time series (quasi experiments) Let’s play X’s and O’s

18. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 199219931994199519961997 Breastfeeding Initiation 1992-97 year proportion initiating breastfeeding 1994 Breastfeeding study: pregnant women interviewed ? ? ? Video and breastfeedng booklet completed, used in individual prenatal instruction by CHN CHN at conference, uses new techniques to address prenatal feeding intent ? ? CHN hired PC Training begun ? * p<0.05, one-tailed, adjusted for birth weight and parity * Martens 2002 Example of a quasi-experimental time series to determine effectiveness of a community-based breastfeeding strategy

19. Time series (quasi experiment with comparison group) O O X O O - - - - - - - - - - - - - - - O O Let’s play X’s and O’s

20. From CIHR proposal submission September 2004 Example of a quasi-experimental time series with comparison groups to determine effectiveness of a regional teen pregnancy reduction program

21. Additions of small amounts of phosphorus to one section of ELA Lake 226 caused surface blooms of blue-green algae, and vividly demonstrated the importance of phosphate as a cause of excessive algal growth or eutrophication. This experiment spurred legislation controlling the input of phosphorus to many water bodies. http://www.umanitoba.ca/institutes/fisheries/eutro.html A demonstration of the work of Dr. David Schindler and the Experimental Lakes project in NW Ontario

22. Study design: Low internal validity Anecdote/case study Pre-experimental just doing a pretest and posttest on one group and seeing its effect Cross-sectional a snapshot in time: can’t tell which comes first, but only that they are “associated”

23. Study design: medium internal validity Time series; Time series with qualitative layer – looking over time to see change, with information about when interventions occurred in the time frame Case-control – going backwards in time from the cases/controls to look at different exposures to possible risk factors Observational (prospective) – going forward in time, observing the effect of exposure on a cohort to a future outcome

24. Study design: high internal validity Randomized Controlled (clinical) Trials, RCT designing a specific intervention and randomly assigning people to receive it or not to receive it following people to observe the outcome of interest Quasi-experimental comparison group studies using a comparison group which is not randomly assigned, but very similar at onset

25. Internal validity Low High Cross-sectional Pre-experimental Anecdote/case study Time series with comparison Observational (prospective) Case-control Time series with qualitative layer Randomized Controlled Trials RCT Quasi-experimental comparison group studies

26. “There is nothing so useless as doing efficiently that which should not be done in the first place.” Peter Drucker

27. MCHP’s … “paperclips” “Population Health Research Data Repository” Population- Based Health Registry Hospital Home CarePharmaceuticals Cost Vital Statistics Provider Nursing Home Medical Family Services Education Immunization National surveys Census Data EA/DA level

28. Brainstorming: “What Works” proposal Pick (a) a policy; and (b) a program – Think of something that your region has done in the past, somewhere between 1997 and the present (hopefully, with a few years of data AFTER the onset of this) What OUTCOME measures would you think this would impact? – Think of what you would expect to see if this intervention was “working” – Are there specific target groups to which this intervention applies? (e.g. teens, people living in a certain district of your region?) – What measures of this intervention would be available through the Repository data? Brainstorm and report! (see sheet for recording)

29. Policy or Program Outcome Measure(s) Target Group Outcome available in Repository ? Other comments Teen pregnancy reduction Teen pregnancy rate 12-19 year olds? Certain district? pregnancies or live births? Maybe birth control pill use in Rx data?