Presentation on theme: "Systematic Review: Workshop 1 26 July 2007 www.warwick.ac.uk/go/chrisbridle/sr Chris Bridle, PhD, CPsychol Institute of Clinical Education Warwick Medical."— Presentation transcript:
Systematic Review: Workshop 1 26 July Chris Bridle, PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick Tel: +44(24)
Workshop Overview Systematic review 70 mins Break15 mins Programme structure10 mins Protocol development45 mins Questions -
Why is research synthesis needed? 2 million articles published in 20,000 health journals pa Equivalent to a pile of paper 900 metres high If you tried to stay current by reading two articles each day, within 1 year you would fall 55 centuries behind! Put another way, if you try to read everything of possible relevance, you would have to read 5,500 articles per day. Seeing is believing... ‘What burns me up is that the answer is right here somewhere, staring us in the face … … if only Chris B was here! 381m
What and When? What? Thalidomide When? Before 1960 The evidence was there, but the methods for making it accessible and meaningful were not OR (fixed) OR Study95% CI(fixed) [95% CI] [0.35, 14.51] [0.24, 18.96] [0.21, 12.71] [0.35, 13.43] [0.43, 11.69] Total (95%CI) 3.06 [1.13, 8.31] Heterogeneity: Chi²=0.23, df=4 (P=0.92) Effect: Z=2.20 (P=0.03) Favours treatment Favours control Treatment Control (Fewer More) ◊
Don’t forget the slides handout!
Why we need research synthesis Vast quantity of research Single trials rarely definitive Hidden effects Research and practice built upon biased &/or unfounded assumption Need for research synthesis How can research evidence be synthesized?
Chalk and Cheese
You Decide Traditional Review: Take a simmering topic, extract the juice of one argument, add the essence of one filling cabinet, sprinkle liberally with your own publications and sift out the work of noted detractors A non-scientific method prone to bias in evidence retrieval, ‘evaluation’, synthesis and interpretation Systematic Review: A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, evaluate and synthesise relevant primary research A scientific methodology that is reproducible and amenable to rigorous evaluation
Formulate Review Question Develop Protocol Search, Identify & Select Evidence Data extraction Quality Assessment Data Synthesis & Interpretation Conclusions & Recommendations Write Review Report Key stages in conducting a systematic review
Formulating a Review Topic / Question Example question: Effects of interventions for overweight children This is a bad question! It is much too broad, would take years to answer, and results unlikely to be meaningful Needs to be narrower, i.e. more focussed What effects? Effects of what exposure? Among who? Compared to what / who? Etc., etc., etc.
Focussing the Review Answerable questions include: A description of the population P An identified intervention I An explicit comparison C Relevant outcomes O Overweight children Community walking Orilstat Weight loss Community walking versus orilstat for weight loss among overweight children
PICO Components Identify the PICO components: Effectiveness of homeopathy Stage-based interventions for smoking cessation Psychosocial interventions for prevention and treatment of childhood obesity Effectiveness of atypical antipsychotics for treatment of mania in bipolar disorder
The Review Protocol The review protocol is the first major milestone of any systematic review Provides a rigid, well-specified plan for how each stage of the review will be conducted – a roadmap Helps to avoid or minimise bias at each stage of the review – but only if followed rigidly Should be sent for external peer review, e.g. advisory group comprising researchers, practitioners, users, etc. Should be published, e.g. project webpage, research register, or peer-reviewed journal
Comprehensive Literature Search Necessary to reduce bias: Publication, language, and geographical Search media: Electronic, manual, and personal Publication type: Journal, conference, dissertation, grey, and ongoing
From question to search terms For each PICO component identify and record: Abbreviations, e.g. CBT, TTM, MS, CFS Synonyms, e.g. CBT - Theory or Therapy Related terms: cognitive theory/therapy, behavior theory/therapy Transatlantic differences, e.g. pressure sores / decubitus ulcers Spelling, e.g. behavior or behaviour Set limits? Publication, e.g. published; language; date, etc. Study, e.g. sample size, follow-up time, type of outcome assessment, etc. Evidence reliability, e.g. study design; specific criteria … Hierarchy of evidence
Systematic Review RCT Cohort Studies Case-Control Studies Case Reports Cross Sectional Surveys Expert Opinion Anecdotal Experimental Study Observational Studies You, … and me Drunk Bob + + Potential For Bias - -
Building a Search Strategy Record everything!* Search history, e.g. database, host and date Search strategy, e.g. terms and combinations Search results Design a search-term table Enter search terms in relevant column Combine search terms for each column with ‘or’ = more Combine column results with ‘and’ = less * See example
Preparing to be flooded How will you manage the many (thousands) of references and abstracts your search has identified? Bibliographic software Endnote RefWorks ProCite RevMan
Study Selection What references to retrieve? Study information limited to Abstract or Title Assess potential relevance of studies against inclusion / exclusion criteria (i.e. P.I.C.O) – retrieve relevant full text papers What studies to include? Use full text papers to assess the relevance of studies against the study inclusion / exclusion criteria - PICO The remaining papers are those relevant to your review Keep precise records from start to finish Account for all references (studies)
Example: Flow chart of study papers through a review Study papers are to a review, what participants are to a trial
Data Extraction Recording of information presented in primary studies Ranges from superficial reporting to systematic recording Systematic extraction / recording is important for data synthesis and interpretation Important to strike the right balance between reporting / recording too much or too little information Determined by the ‘nature’ of the review and its intended use / audience
Extracting Study Data Important data to extract: Participants: demographic / disease characteristics Intervention: description of intervention and control Analysis: statistical method, number of Ps, ITT analysis Results: be selective and sensible, e.g. primary outcome Authors’ conclusions: word-for-word reporting of conclusions Reviewer’s comments: validity of results, calculation of summary (effect) statistic, etc.
ParticipantsInterventionResultsConclusion / Comment N Randomised: 290 (I=150, C=140) Age: m=43. Gender: 30% female Type: UK Community (Patient) Recruitment: Non-smoking related attendance at GP surgery I: 3 x 30 min weekly stage-based, group MI with take-home intervention pack. C: GP advice Provider: GP Nurse Setting:Surgery Follow-up:2 month Outcome: Abstinence 3 wks, self-report Dropout: 82 (I=53, C=29) N Analysed: 208 (I=97, C=111) Abstinence: 31 (I=19, C=12) (p<0.05) Reviewer: ITT OR=1.54 (95% CI, 0.63 to 4.29) Brief, stage-based MI with take-home material is an effective smoking cessation intervention. Comment: High attrition (I, OR=2.09) and ns effect with ITT Unknown inclusion criteria Unclear tailoring (group-MI) Follow-up too short – 80% relapse/failure to 12 months Authors ’ conclusions are inconsistent with data Data Extraction: Example
Tabulated Data: Example Study P’sExposureContextControl Outcome (Abstinence) Summary effect OR (95%CI) Validity Smith, , UK GP patients Group MI + written advice Nurse, GP surgery, 3 pw Usual care Self-report at 2 months 1.54 (0.63,4.29) Poor Jones, , UK community Group MI Researcher, community centre, 2 pw No intervention Biochemical validation at 12 months 1.03 (0.33,1.22) Good Davis, , UK students Stage-based Written material No intervention Self-report at 2 months 2.54 (1.33,4.89) Poor Scott, , UK GP patients Individual MI Counsellor, home visit, 1pw No intervention Self-report at 1 month 1.87 (1.12,3.19) Poor Tabulated data are usually presented in the text of the report / paper
Critical Appraisal Estimates potential for bias Extent to which results attributable to factors other than named intervention Independent application of a priori criteria Criteria depend on study design Common criteria for RCTs: Adequate randomisation; Concealed allocation; a priori sample size calculation; Blinding; Baseline comparability; Adequate follow-up; Intention-to-treat analysis
Importance of Criteria Quality Item % Overestimation of intervention effect Flawed randomisation41 Unclear randomisation30 Lack of blinding17 No a priori sample size calculation30 Failure to use ITT analysis25 Trials with poor quality of reporting20 Khan et al (1995) Moher et al (1998)
Uses of Quality Assessment Determine the strength of conclusions A threshold for inclusion Potential source of heterogeneity As weights in statistical analyses – unwise! Recommendations for improving future research
Data Synthesis Quantitative: statistically combine results of several studies into a single numerical estimate of effect, e.g. meta-analysis Only used for (some) quantitative data Qualitative: a narrative summary and synthesis of primary studies Used for both quantitative and qualitative data
Selecting a Method of Synthesis Decision based on degree of heterogeneity Variation within the group of included studies Types of heterogeneity: Clinical: participants, interventions and outcomes, e.g. age, intervention intensity, outcome definition, etc. Methodological: how studies were conducted, e.g. study design, study quality Statistical: measured intervention effect – you should calculate a summary statistics for each study yourself
Workplace exercise intervention for mild depression
Assessing Statistical Heterogeneity Visual inspection Overlap of CIs Chi-square statistic Significant p<.01 Power, too much, too little Statistic Vs degrees of freedom Meaningless question Is heterogeneity present? Of course it is! We want to know how much heterogeneity is present, where, and is it important?
Dealing With Heterogeneity Subgroup analyses Comparison of separate meta-analyses based on subgroups of the studies Sensitivity analyses Repeating meta-analyses with amended criteria to determine robustness of review findings
Subgroup analyses: Mortality results of zidovudine trials in HIV, stratified by infection stage (early or late).
Sensitivity analyses: Case-control studies relating residential exposure to electromagnetic fields to childhood leukaemia, stratified by methodological quality.
Caution Subgroup and sensitivity analyses need to interpreted with caution Strong conclusions are often drawn from subgroup / sensitivity analyses –misleading and dangerous Results based on indirect comparisons Any differences are observational in nature Prone to bias and confounding
Narrative Synthesis Used for both quantitative and qualitative data but no formal method of synthesis Studies meaningfully grouped and explored for potential sources of effect heterogeneity Participants: age, gender, ethnicity, SES Interventions: setting, provider, intensity, duration Outcomes: definition, measured, collection, follow-up Methods: quality, design, sample size, protocol adherence
Conclusions Must be based solely on the evidence reviewed Should highlight implications for research, practice, policy, decision makers, etc. May be graded according to the strength of the evidence May indicate a time period within which the findings of the review are relevant
Formulate Review Question Develop Protocol Search, Identify & Select Evidence Data extraction Quality Assessment Data Synthesis & Interpretation Conclusions & Recommendations Write Review Report Scoping Search Develop Research Bid Advisory Group Advisory Group Review Drafts Reasons for Exclusion Independently Assessed Check for accuracy Independently Assessed Investigate Heterogeneity Core Systematic Review Stages Review Processes - Inputs Review Outputs Publish Protocol Publish Review
Critical Appraisal of SRs QuestionIs the review question clear, specific and answerable? SearchHave attempts to identify relevant evidence been sufficiently comprehensive? EvaluationHave included studies been critically appraised against appropriate criteria? SynthesisIs the method of synthesis appropriate, and have potential sources of heterogeneity been investigated? ConclusionsDo conclusions reflect both the quality and quantity of evidence? ProcessIs the review process independent and transparent so as to limit potential bias?
Summary Systematic reviews are important: Meet the need for research synthesis Use scientifically rigorous methods Provide evidence-based answers to meaningful questions Identify gaps and weaknesses in evidence base Improve research quality Enhance effectiveness of clinical practice Strengthen bids for research funding
Good Guidance Centre for Reviews & Dissemination: Report 4 Cochrane Collaboration: Reviewer’s Handbook Cochrane Behavioral Medicine: Recommended paper WMS Systematic Review Training Materials
Any questions? Now, or later Chris Bridle PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick Tel: (024) Web:
Programme Overview Date Review WorkshopIntervening Tasks & Objectives July Planning: Developing a review protocol; Formulating a review question / objective; Methods for conducting the review Develop protocol with completed background, specified objective/question; Outline methods. Submit draft for assessment, amend and submit to expert for comment. Sept Conducting: Search strategy; Data extraction; Quality assessment (Data synthesis; Heterogeneity; Interpretation) Update protocol methods (search, extraction template, assessment criteria) and submit. Conduct review as specified in approved protocol, submit draft review report (data synthesis plan). Oct Reporting: Report structure; Manuscript guidelines; Dissemination strategy; Targeted publication; (Data synthesis) Synthesise data; Write draft report; Identify peer- review jrnl; Obtain manuscript guidelines. Submit draft report and dissemination strategy for internal review and respond to feedback. Nov Disseminating: Publishing the review; Target audiences; Dissemination media; Responding to peer-review; Medical education Present completed review to peers; Reflect on programme experience / evaluation & feedback. Write full report for internal peer-review, submit for publication and go to the pub to celebrate.