Presentation on theme: "The Bahrain Branch of the UK Cochrane Centre In Collaboration with"— Presentation transcript:
1 The Bahrain Branch of the UK Cochrane CentreIn Collaboration withReyada Training & Management Consultancy, Dubai-UAECochrane Collaboration and Systematic Review Workshop, February 2007, Dubai - UAEW02Dr. Zbys Fedorowicz, Dr. Dunia Al Hashimi, Dr. Ahmed Al Asfoor
3 Reviews and Meta-Analyses Review: An article that summaries a number of different primary studies and may draw conclusions on the effectiveness of a particular intervention.A review may or may not be systematic.Meta - analysis by definition is a process of synthesizing research results by various statistical methods to retrieve, select, and combine results from previous separate but related studies.Meta - analyses are reviews that use systematic methods to summarizes the results of previous studies
4 Types of ReviewsNarrative (Traditional)Systematic/Overview
5 A Systematic Review:A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critical appraise relevant primary research, and extract and analyze data from the studies that are included in the review.Statistical Methods (meta-analysis) may or may not be used.
6 Differs from traditional reviews in that they use a replicable, scientific and transparent approach to minimise bias.
7 Feature Narrative Review Systematic Review QuestionsOften broad in ScopeOften a focused questionSources and SearchNot usually specified, potentially biasedComprehensive sources and explict search strategySelectionCriterion-based selection, uniformly appliedAppraisalVariableRigorous critical appraisalSynthesisOften a qualitative summaryQuantitative summary*InferencesSometimes evidence-basedUsually evidence-based* Quantitative summary that includes a statistical analysis is a meta-analysis
8 Literature review, systematic review, meta-analysis Meta-analyses can be narrowly defined as quantitative pooling of results of individual studies. It is also sometimes called as overviews or systematic reviews. We know, in a systematic review, all relevant research are systematically searched, assessed and synthesized. Therefore, a narrowly defined meta-analysis is only part of a systematic review. This clarification is important since many systematic reviews are not able to quantitatively combined study results, and some meta-analyses are not systematic.
9 Anatomy of a Systematic Review The QuestionSurvey the review literatureFurther define questionDesign the ProtocolSearch and Study SelectionData ExtractionAnalysisReporting
10 The Question Systematic reviews usually have a focused: Population/problem, setting(s).Intervention (treatment, diagnosis, technology, etc.)Outcome (patient, economic, usability, etc.)(EBM uses PICO as a guide to question formation)
11 2. Literature ReviewCheck to see if a review has already been done on your question.This process can help re-define and focus your question.You can gain an understanding of the literature in the subject area.
13 4. Develop the ProtocolThe protocol is your research plan and should include: -The backgroundThe problemThe methodologyProtocols should be developed prospectively in limit bias.
14 Protocols should include: Specific questions to be answeredStrategies and sources for finding informationInclusion/Exclusion criteria includingPopulation, settings, problemExposure, interventionControlsOutcome specified : definitions, what is being measured, how it’s being measured, where and by whom.
15 Cont.. – Type(s) of study design Study selection Types of studies to include –experimental, observational, qualitative, etc.Defined outcomes measuring what, how, and by whomStudy selectionData Extraction strategyAnalysis strategyPresentation of results
16 5. Finding the information Start with the questionDecide on sources to searchDevelop search strategiesKeep records and use a bibliographic software program to manage the information
17 Search StrategyIs an iterative process developed through input from the review team.Generates the lists of published (and unpublished) research studiesAffects the validity of the results of the reviewThe protocol guides the search by stating sources to search, types of information to include (published & unpublished), languages, study design, key concepts.
18 Search strategy cont’d The thoroughness of the searching is one of the defining differences between systematic and traditional reviewsMust decide whether to go for aSensitive search strategy – increases retrieval and irrelevant resultsORSpecific search strategy – increases precision but may miss relevant material
19 Sources of Information Databases of primary research articlesHand searching core subject journalsChecking cited referencesConference proceedingsContacting researchers and expertsDissertationsGrey LiteratureWeb of Science – citation trackingInternet
21 Why so thorough?Missing studies can have significant effect on the results of the reviewPublication BiasThe tendency for studies with positive results to get published and studies showing no difference or negative results not getting published.
22 Thorough searching much research is never published not all research is published in journalsnot all research published in journals is indexed on major databasesnot all research indexed on databases can be easily retrieved
23 Sources of bias in trials Target population Sources of biasAllocationIntervention Controlgroup groupSelection bias Exposed to Not exposed tointervention interventionPerformance bias Follow-up Follow-upAttrition bias Outcomes OutcomesDetection bias
24 ‘Bias’ publication bias language bias Positive resultslanguage biasPositive results more likely to be published in English language publicationsgeographical coverage bias of journals and databases
25 Why do we search widely?Savoie et al estimated that 29.2% of items in their review were uncovered by:searching the webhandsearchingscanning reference listspersonal communicationsearching specialised databases and web sites.
26 Publication bias Song et al. Publication and related biases. Methods to dealing with publication and related biases in systematic reviews:literature searchinglocating unpublished studiesassessment of the risk of publication and related biasesdetecting publication bias
27 Why search widely…. Allen & Hanburys found that: Wallace et al. only 51% of the clinical trials relating to their respiratory products were published in journals indexed by MEDLINE, EMBASE or CINAHL (46% are in grey lit)Wallace et al.11 of 65 trials in end stage renal disease reviews were found by searching beyond major databases.Allen and Hanburys ref from Prentice et al. Accessibility of trial data to EBM reviews.Song review: findingsResearch findings and dissemination profilesThe empirical evidence demonstrates that studies with significant results or favourable results are more likely to be published orcited than those with non-significant or unfavourable results. Studies with significant results are often published earlier than thosewith non-significant results. Limited and often indirect evidence indicates only the possibility of full publication bias, outcomereporting bias, duplicate publication bias, language bias and database bias. There is some evidence concerning the existence ofcitation bias and media attention bias.Consequences of publication and related biasesThe important consequences of publication bias include the avoidable suffering of patients and the waste of limited resources.However, there is little empirical evidence relating to the impact of publication and related biases on health policy, clinical decisionmaking and the outcome of patient management.Sources of publication biasInvestigators, peer reviewers, editors and funding bodies may all be responsible for the existence of publication bias. Someevidence suggests that authors or investigators may be the main source of this bias, for not writing up or not submitting studies withnull or unimportant results. However, it should be recognised that the decision to write up an article and then submit it may beaffected by pressure from research sponsors and instruction from journal editors. Evidence shows that the interest of researchsponsors can restrict the dissemination of research findings. The large potential variation in results obtained across similar studiesthat can easily be conducted and abandoned will further exacerbate the biased selection of findings for publication.Prevention of publication biasBecause of their space limitations and need to maintain newsworthiness, it is unlikely that conventional paper journals can solve theproblem of the selective publication of studies that produce striking results. For the purpose of reducing publication bias,peer-reviewed electronic journals that are without limitations of space are required. More importantly, editorial policy needs to bechanged to accept for publication clinical trials that are based on methodological criteria only and not on the impact of theirfindings.Clearly, the ideal solution to publication bias is the prospective, universal registration of all studies at their inception. Although theregistration of all studies cannot be realised in the near future, there are many encouraging signs that there will be more registriesestablished as a result of initiatives from government or industry. Large-scale confirmatory studies may be an alternative in theprevention of the consequences of publication bias.Methods for reducing or detecting publication biasThe methods available for dealing with publication and related biases in systematic reviews include literature searching, locatingunpublished studies, assessment of the risk of publication and related biases, several methods for detecting publication bias inmeta-analyses, and updating systematic reviews. The statistical methods are by nature indirect and exploratory, and often basedon certain strict assumptions that can be difficult to justify in the real world. The attempt at identifying or adjusting for publicationbias in a systematic review should mainly be used for the purpose of sensitivity analysis.Survey of published systematic reviewsThis survey indicates that literature searching was clearly inadequate in some published systematic reviews. Potential publicationbias was ignored and the available methods for dealing with such bias were not used in most of these reviews. When they are usedto estimate possible publication bias at the stage of literature review, the available methods were far from adequate and theirusefulness was strictly limited. The problem of publication and related biases was dealt with more often in reviews containing ameta-analysis than in the narrative systematic reviews.ConclusionsAlthough the extent, direction and impact of publication and related biases are uncertain and may vary greatly depending oncircumstances, it seems reasonable to conclude that studies with significant or favourable results are more widely disseminatedthan those with non-significant or unfavourable results. The potential problem of publication and related biases should be takeninto consideration in the field of health technology assessment. All funded or approved studies should be prospectively registered.The risk of publication bias should be assessed in all systematic reviews.
28 Why search widely.... Long lead times before publication: publication gaps after conference presentationindexing publication lag before recorded in databasesCheng et al, 19988.1% of a set of conference papers achieved publication within 12 months29% within 2 years and 40% within 5 years
29 Delay and non publication Cheng et al, 1998only 32% of abstracts presented at chronic fatigue conferences were subsequently published in fullPetticrew et al, 199950.6 % of oral paper presented at the Society for Social Medicine 1996 achieved publication
30 Key factors thorough searches searching beyond published articles citation searchingInternet searchinghand searchingsearching for ongoing research
31 Sources of Information REVIEWS:Cochrane Database of Systematic Reviews (CDSR)DARECochrane Central Database of Controlled Trials (CCDCT)other sources such as Health TechnologyAssessment sourcesHTA ( orCCOHTA – Canadian Coordinating Office for Health Technology Assessment (has a number of publications)International Network of Agencies for Health Technology Assessment
33 OTHER: Reference Lists & Citations Web of Science cited reference searchesHand searching key journalsResearchers in the subject areaConference Proceedings (PapersFirst, Proceedings First)Dissertations (Dissertation Abstracts International)Internet
34 Grey Literature Databases of grey literature Grey Literature Report SIGLE System for Information on Grey Literature in Europe contains citations to reports and non-conventional (grey) literature issued informally throughout EC member countries.NetPrints -A repository of non-peer reviewed original researchOrganizations ….Clinical Trials
35 6. Information from included studies Bibliographic detailsStudy characteristics- design/methods- participants- interventions- outcome measuresStudy results- means, SD and/or SE- no. of events and N
36 Make sense of a review Are the results of the review valid? B. What are the results?C. Will the results help locally?
37 A. Are the results of a review valid? Did the review address a clearly focused issue?2. Did the authors look for the appropriate sort of papers?3. Do you think the important, relevant studies were included?4. Did the review’s authors do enough to assess the quality of the included studies?5. If the results of the review have been combined, was it reasonable to do so?
38 B. What are the results? 6. What is the overall result of the review? ‘Bottom line’ results; NNT, OR, RR, RD?7. How precise are the results?Confidence interval, p values
39 C. Will the results help locally? Can the results be applied to the local population?Were all important outcomes considered?Are the benefits worth the harms and costs?