Presentation is loading. Please wait.

Presentation is loading. Please wait.

Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules.

Similar presentations


Presentation on theme: "Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules."— Presentation transcript:

1 Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide

2 Overview of a Medical Test Review Prepare Topic Develop the Topic and Structure the Review Choose the Important Outcomes Research Sources Search for and Select Studies for Inclusion Search for Studies Extract Data From Studies Analyze and Synthesize Studies Assess Risk of Bias as a Domain of Quality Assess Applicability Grade the Body of Evidence Meta-analysis of Test Performance Evidence With a “Gold Standard” — or — Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard Decision Modeling Report Medical Test Review Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

3  Recognize the importance of a multiple- approach method of searching  Understand the current state of indexing and reporting and its effect on the most pragmatic methods of conducting searches Learning Objectives Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

4  The overall goal is to locate all published studies relevant to key questions.  General guidance on searching for systematic reviews also applies to reviews of medical tests.  A balance must be achieved between recall and precision:  Recall = how much of the relevant literature is retrieved.  Precision = how much of the retrieved literature is relevant.  An optimal balance depends on the context. For comparative effectiveness reviews, the goal is to conduct a comprehensive and exhaustive search while minimizing the resources needed to review retrieved citations. General Search Guidance (1 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at

5  Develop the search strategy for the study with the research team according to the analytic framework and the inclusion and exclusion criteria.  Search electronic databases for applicable studies:  MEDLINE  and the Cochrane Central Register of Controlled Trials should always be included in the search strategy.  Other useful databases include EMBASE , CINAHL , and PsycINFO .  Tailor the search to each database.  Use both controlled (i.e., MeSH  subject headings) and uncontrolled vocabulary.  Use delimiters such as language only if a specific case can be made. General Search Guidance (2 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at

6  Identifying additional studies:  References from studies included for review  Citation databases (e.g., SciVerse  Scopus or Web of Science  )  Unpublished or “grey” literature, such as:  Clinical trial registries (e.g., ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform)  Conference proceedings/abstracts (e.g., ProQuest COS Conference Papers Index) General Search Guidance (3 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at

7  Searches are required on each relevant test strategy being considered.  One of two approaches is used to conduct the search:  Search all possible tests used to evaluate the disease. This requires knowledge of all possible test strategies available.  Search on the disease or condition, and then filter for the medical test.  Searching is often an iterative process, requiring refinements to search criteria. Challenges Specific to Searches for Studies of Medical Tests (1 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

8 The first approach for identifying studies of medical tests is to search for specific named tests.  Benefits:  It is straightforward; test names are used to locate studies.  A specific search for the concept of diagnosis/screening/ prognosis may not be necessary.  Drawbacks:  New approaches may be missed because of evolving test strategies.  Using the test name alone may be insufficient, especially for gene product tests that are associated with multiple diseases. Challenges Specific to Searching for Studies of Medical Tests (2 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

9  The second approach for identifying studies of medical tests is to search for a specific disease or condition.  Benefits:  This approach is preferred if specific medical tests are unknown.  It can be used in combination with searches for known test names.  Drawback: Searches are broad and increase the amount of work needed to filter down to the relevant studies.  This approach is used more often by reviewers. Challenges Specific to Searching for Studies of Medical Tests (3 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

10 1.Do not rely on search filters alone. 2.Do not rely on controlled vocabulary (subject headings) alone. 3.Search in multiple locations. Principles of Searching for Studies of Medical Tests Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

11  Search filters (or “hedges”):  Are pre-prepared and tested searches.  Can be combined with searches on a particular disease or condition.  Most search filters have been developed for MEDLINE .  Additional search filters have been developed specifically for diagnostic imaging and EMBASE .  Currently, search filters are inappropriate for systematic reviews because they:  May miss relevant studies.  Do not perform as well for studies published before 1990 because of nonstandardized reporting and indexing of medical test studies, which may improve in the future. Search Principle 1: Do Not Rely on Search Filters Alone (1 of 2) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

12 Category Optimization of Search Sensitivity/Specificity of SearchPubMed  Search String DiagnosisBroad Sensitivity/breadth (of “Diagnosis” search) 98%/74%(sensitiv*[Title/Abstract] OR sensitivity and specificity[MeSH Terms] OR diagnos*[Title/Abstract] OR diagnosis[MeSH:noexp] OR diagnostic* [MeSH:noexp] OR diagnosis,differential[MeSH:noexp] OR diagnosis[Subheading:noexp]) Narrow Specificity/narrowness (of “Diagnosis” search) 64%/98%(specificity[Title/Abstract]) Search Principle 1: Do Not Rely on Search Filters Alone (2 of 2)  Example: Filter used by PubMed  Clinical Queries  Category selected: “Diagnosis”  Two choices of scope to optimize search: “Broad” or “Narrow” Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

13  It is important to use all known variants of a test name, as in the examples below that refer to hematocrit:  Abbreviations (Hct, Crit, PCV)  Generic names (hematocrit, packed cell volume)  Proprietary names (e.g., LighTouch ® HCT)  International terms/spellings (haematocrit)  Controlled vocabulary may not contain all variants.  Controlled vocabulary is unreliable due to nonstandardized reporting and indexing.  Textwords help identify tests not yet indexed or indexed improperly.  Filters may suggest appropriate textwords. Search Principle 2: Do Not Rely on Controlled Vocabulary (Subject Headings) Alone Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

14  Searching more than one database and tailoring searches to each is always advisable. Otherwise, there is a risk of bias due to little overlap between the many databases.  Until reporting/indexing is standardized, the best method is to combine highly sensitive searches and manual article screening.  Articles can still miss relevant articles, so additional sources of information are necessary.  Other sources of information:  Additional bibliographic databases  Citation tracking (i.e., reading through references of relevant articles)  Regulatory documents  Search for diagnostic documents at a U.S. Food and Drug Administration Web site. Search Principle 3: Search in Multiple Locations (1 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

15 Open-Access Databases DatabaseURLTopic Coverage IBIDS (International Bibliographic Information on Dietary Supplements) Information/IBIDS.aspx Dietary supplements PubMed  Biology Health sciences TOXNET (Toxicology Data Network) Toxicology Environmental health adverse effects World Health Organization Global Health Library International biomedical topics Global Index Medicus Examples of specialized databases that have free access: Search Principle 3: Search in Multiple Locations (2 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

16 Subscription Databases DatabaseURLTopic Coverage CINAHL  (Cumulative Index to Nursing and Allied Health) Nursing Allied health EMBASE  Biomedical with emphases on drugs and pharmaceuticals More non – U.S. coverage than MEDLINE  IPA (International Pharmaceutical Abstracts) Drugs and pharmaceuticals MANTIS (Manual Alternative and Natural Therapy Index System) Osteopathy Chiropractic Alternative medicine PsycINFO  index.aspx Psychological literature Examples of specialized databases that require a subscription: Search Principle 3: Search in Multiple Locations (3 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

17 Citation-Tracking Databases DatabaseURLAccess Google Scholarwww.scholar.google.comFree PubFocuswww.pubfocus.comFree PubReMinerbioinfo.amc.uva.nl/human-genetics/pubreminerFree SciVerse  Scopuswww.info.scopus.comSubscription required Web of Science  thomsonreuters.com/products_services/science/ science_products/a-z/web_of_science Subscription required Examples of citation-tracking databases: Search Principle 3: Search in Multiple Locations (4 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

18  AHRQ Evidence Report/Technology Assessment No. 142: Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure  All medical tests were known.  The search consisted of all possible variations on test names.  Using a search string to capture the diagnostic testing concept was not necessary.  AHRQ Comparative Effectiveness Review No. 2: Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities  All medical tests were not known.  The PubMed  search string relied on textwords used to capture the diagnostic testing concept: “diagnosis OR diagnose OR diagnostic OR di[sh] OR “gold standard” OR “ROC” OR “receiver operating characteristic” OR sensitivity and specificity[mh] OR likelihood OR “false positive” OR “false negative” OR “true positive” OR “true negative” OR “predictive value” OR accuracy OR precision” Examples of Search Strategies Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

19  Diagnostic search filters cannot be depended on exclusively.  When the full range of tests is known, searching for a specific test using all variant names may be sufficient.  Combining highly sensitive database searches (using textwords), hand searching, and searching lists of cited references is currently the best way to identify relevant studies.  Do not rely on controlled vocabulary alone.  Search multiple locations and databases.  Be sure to check the Web site, which is a useful source. Key Messages Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at

20 1.Generally, when designing a literature search, higher precision is desirable relative to recall. a.True b.False Practice Question 1 (1 of 2)

21 Explanation of Question 1: The statement is false. Maintaining balance between recall and precision is usually desirable. Practice Question 1 (2 of 2)

22 2.Search filters are inappropriate for systematic reviews. a.True b.False Practice Question 2 (1 of 2)

23 Explanation of Question 2: The statement is true. Search filters may miss relevant studies and do not perform as well for studies published before 1990 because of nonstandardized reporting and indexing. Practice Question 2 (2 of 2)

24 3.Which of the following are true statements regarding controlled vocabulary: i.A controlled vocabulary may not contain all variants. ii.A controlled vocabulary is reliable thanks to standardized reporting and indexing. iii.Textwords help identify tests not yet indexed or indexed improperly. Choices: a.a and b b.a and c c.b and c d.All three statements are true. Practice Question 3 (1 of 2)

25 Explanation of Question 3: The correct answer is b. Controlled vocabulary is actually unreliable for identifying variants of a test name because of nonstandardized reporting and indexing. Practice Question 3 (2 of 2)

26 4.Which of the following is not a principle of searching for studies outlined in this module: a.Do not rely on search filters alone. b.Do not rely on controlled vocabulary alone. c.Search in multiple locations. d.Avoid using Google Scholar as a search tool. Practice Question 4 (1 of 2)

27 Explanation of Question 4: The correct answer is d. Google Scholar can be useful for grey literature searches. Practice Question 4 (2 of 2)

28  This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center.  The module is based on Chapter 4, Effective Search Strategies for Systematic Reviews of Medical Tests. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June Authors

29  Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Review. Rockville, MD: Agency for Healthcare Research and Quality; April AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at  Astin MP, Brazzelli MG, Fraser CM, et al. Developing a sensitive search strategy in MEDLINE to retrieve studies on assessment of the diagnostic performance of imaging techniques. Radiology May;247(2): PMID:  Bachmann LM, Coray R, Estermann P, et al. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. J Am Med Inform Assoc Nov-Dec;9(6): PMID:  Bachmann LM, Estermann P, Kornenberg C, et al. Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc Jul;91(3): PMID: References (1 of 6)

30  Balion C, Santaguida PL, Hill S, et al. Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure. Evidence Report/Technology Assessment No. 142 (Prepared by the McMaster University Evidence-based Practice Center under Contract No ). Rockville, MD: Agency for Healthcare Research and Quality; September AHRQ Publication No. 06- E014. Available at  Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol Jan 28;5(1):6. PMID:  Bruening W, Launders J, Pinkney N, et al. Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities. Comparative Effectiveness Review No. 2 (Prepared by the ECRI Evidence-based Practice Center under Contract No ). Rockville, MD: Agency for Healthcare Research and Quality; February AHRQ Publication No. 06-EHC005-EF. References (2 of 6)

31  Conn VS, Isaramalai SA, Rath S, et al. Beyond MEDLINE for literature searches. J Nurs Scholarsh. 2003;35(2): PMID:  Deville WL, Bezemer PD, Bouter LM. Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol Jan;53(1):65-9. PMID:  Doust JA, Pietrzak E, Sanders S, et al. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol May;58(5): PMID:  Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. MBJ May 1;328(7447):1040. PMID:  Haynes RB, Wilczynski N, McKibbon KA, et al. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc Nov-Dec;1(6): PMID: References (3 of 6)

32  Honest H, Bachmann LM, Khan K. Electronic searching of the literature for systematic reviews of screening and diagnostic tests for preterm birth. Eur J Obstet Gynecol Reprod Biol Mar 26;107(1): PMID:  Leeflang MM, Scholten RJ, Rutjes AW, et al. Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies. J Clin Epidemiol Mar;59(3): PMID:  Relevo R. Effective search strategies for systematic reviews of medical tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June p AHRQ Publication No. 12-EHC017.  Ritchie G, Glanville J, Lefebvre C. Do published search filters to identify diagnostic test accuracy studies perform adequately? Health Info Libr J Sep;24(3): PMID: References (4 of 6)

33  Sampson M, Barrowman NJ, Moher D, et al. Should meta-analysts search EMBASE in addition to MEDLINE? J Clin Epidemiol Oct; 56(10): PMID:  Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med Dec;12(4): PMID:  Suarez-Almazor ME, Belseck E, Homik J, et al. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials Oct;21(5): PMID:  van der Weijden T, Ijzermans CJ, Dinant GJ, et al. Identifying relevant diagnostic studies in MEDLINE. The diagnostic value of the erythrocyte sedimentation rate (ESR) and dipstick as an example. Fam Pract Jun;14(3): PMID: References (5 of 6)

34  Vincent S, Greenley S, Beaven O. Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Info Libr J Sep;20(3): PMID:  Whiting P, Westwood M, Beynon R, et al. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clin Epidemiol Jun;64(6): PMID:  Whiting P, Westwood M, Burke M, et al. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol Apr;61(4): PMID:  Wilczynski NL, Haynes RB. EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC. Med 2005 Mar;3:7. PMID:  Zheng MH, Zhang X, Ye Q, et al. Searching additional databases except PubMed are necessary for a systematic review. Stroke Aug;39(8):e139. PMID: References (6 of 6)


Download ppt "Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules."

Similar presentations


Ads by Google