Quasi-experimental study designs in evaluating medicines use interventions

Lloyd Matowe 1 & Craig Ramsay 2 1 Faculty of Pharmacy, Kuwait University 2 HSRU, University of Aberdeen

BACKGROUND  The RCT is seen as the gold standard methodology for evaluating medicine use interventions  There are situations where it is impractical to carry out RCTs  For example in situations where:  It is impractical to randomize subjects  Difficult to find an appropriate control  Intervention has been retrospectively implemented  Quasi-experiments are often used as an alternative

Simple before and after studies Time Intervention

Interrupted time series analysis Time Intervention

before Controlled before and after design BeforeAfter studycontrol control study Intervention

 There is suspicion that quasi-experiments are often conducted and analysed inappropriately

OBJECTIVE  The objective of this study is to evaluate the use of quasi-experimental designs in medicines use interventions

METHODS  A rigorous systematic review  Subset of a large study on the effectiveness of various policy implementation strategies  Search strategies:  EPOC, MEDLINE, EMBASE, HEALTHSTAR, SIGLE  Bibliographies of previously published reviews  Studies included: CBAs & ITS  Interventions: policies/ guidelines to improve medication use  Outcomes- Objective measures of provider behaviour and/ or patient outcome  Time series regression used to reanalyse serial data where reanalysis was necessary

RESULTS - 1  Search strategy identified hits  hits identified as potentially relevant  285 studies included  44 studies were on medicines use  17 were CCTs or RCTs  27 studies were quasi-experiments  15 ITS  12 CBAs  24 studies were on multiple interventions  Study areas included: Antibiotics, antipsychotics, analgesics, GIT drugs and anti-hypertensives

RESULTS - 2  Time series Overall methodological quality poor Overall methodological quality poor 8 of 15 studies analysed inappropriately 8 of 15 studies analysed inappropriately Effect size different in all 8 studies Effect size different in all 8 studies Common errors included: Common errors included: -Failure to test for autocorrelation and trend -Analysis using non time series methods -Failure to ascertain intervention as independent from other interventions

Trends Time Intervention

RESULTS-3  Controlled before and after studies Overall methodological quality poor Overall methodological quality poor 7 of 12 had unit of analysis errors 7 of 12 had unit of analysis errors Other common errors included: Other common errors included: - Failure to protect against contamination - Failure to take baseline measurements - Failure to perform power calculation

before Controlled before and after design BeforeAfter studycontrol control study Intervention

SUMMARY  Quasi-experiments are often designed analysed inappropriately  Methods and design are often inadequately reported  Study methodological weaknesses are often inadequately reported

RECOMMENDATIONS  For ITS studies w e recommend: Use of time series regression Use of time series regression Pre-testing for autocorrelation and trend Pre-testing for autocorrelation and trend Ascertain that the intended intervention was independent of other interventions Ascertain that the intended intervention was independent of other interventions  For CBA w e recommend: Protection against contamination Protection against contamination Measurement of baseline characteristics Measurement of baseline characteristics Power calculation Power calculation  For both ITS and CBAs methodological weaknesses should be reported

ACKNOWLEDGEMENTS  I would like to thank Professor Jeremy Grimshaw, from Ottawa Health Research Institute and Center for Best Practices, University of Ottawa, Canada for giving me access to the studies used for this evaluation