Presentation is loading. Please wait.

Presentation is loading. Please wait.

Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules.

Similar presentations


Presentation on theme: "Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules."— Presentation transcript:

1 Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2 Overview of a Medical Test Review Prepare Topic Develop the Topic and Structure the Review Choose the Important Outcomes Research Sources Search for and Select Studies for Inclusion Search for Studies Extract Data From Studies Analyze and Synthesize Studies Assess Risk of Bias as a Domain of Quality Assess Applicability Grade the Body of Evidence Meta-analysis of Test Performance Evidence With a “Gold Standard” — or — Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard Decision Modeling Report Medical Test Review Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3  Recognize the importance of a multiple- approach method of searching  Understand the current state of indexing and reporting and its effect on the most pragmatic methods of conducting searches Learning Objectives Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4  The overall goal is to locate all published studies relevant to key questions.  General guidance on searching for systematic reviews also applies to reviews of medical tests.  A balance must be achieved between recall and precision:  Recall = how much of the relevant literature is retrieved.  Precision = how much of the retrieved literature is relevant.  An optimal balance depends on the context. For comparative effectiveness reviews, the goal is to conduct a comprehensive and exhaustive search while minimizing the resources needed to review retrieved citations. General Search Guidance (1 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

5  Develop the search strategy for the study with the research team according to the analytic framework and the inclusion and exclusion criteria.  Search electronic databases for applicable studies:  MEDLINE  and the Cochrane Central Register of Controlled Trials should always be included in the search strategy.  Other useful databases include EMBASE , CINAHL , and PsycINFO .  Tailor the search to each database.  Use both controlled (i.e., MeSH  subject headings) and uncontrolled vocabulary.  Use delimiters such as language only if a specific case can be made. General Search Guidance (2 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

6  Identifying additional studies:  References from studies included for review  Citation databases (e.g., SciVerse  Scopus or Web of Science  )  Unpublished or “grey” literature, such as:  Clinical trial registries (e.g., ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform)  Conference proceedings/abstracts (e.g., ProQuest COS Conference Papers Index) General Search Guidance (3 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

7  Searches are required on each relevant test strategy being considered.  One of two approaches is used to conduct the search:  Search all possible tests used to evaluate the disease. This requires knowledge of all possible test strategies available.  Search on the disease or condition, and then filter for the medical test.  Searching is often an iterative process, requiring refinements to search criteria. Challenges Specific to Searches for Studies of Medical Tests (1 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

8 The first approach for identifying studies of medical tests is to search for specific named tests.  Benefits:  It is straightforward; test names are used to locate studies.  A specific search for the concept of diagnosis/screening/ prognosis may not be necessary.  Drawbacks:  New approaches may be missed because of evolving test strategies.  Using the test name alone may be insufficient, especially for gene product tests that are associated with multiple diseases. Challenges Specific to Searching for Studies of Medical Tests (2 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

9  The second approach for identifying studies of medical tests is to search for a specific disease or condition.  Benefits:  This approach is preferred if specific medical tests are unknown.  It can be used in combination with searches for known test names.  Drawback: Searches are broad and increase the amount of work needed to filter down to the relevant studies.  This approach is used more often by reviewers. Challenges Specific to Searching for Studies of Medical Tests (3 of 3) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

10 1.Do not rely on search filters alone. 2.Do not rely on controlled vocabulary (subject headings) alone. 3.Search in multiple locations. Principles of Searching for Studies of Medical Tests Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

11  Search filters (or “hedges”):  Are pre-prepared and tested searches.  Can be combined with searches on a particular disease or condition.  Most search filters have been developed for MEDLINE .  Additional search filters have been developed specifically for diagnostic imaging and EMBASE .  Currently, search filters are inappropriate for systematic reviews because they:  May miss relevant studies.  Do not perform as well for studies published before 1990 because of nonstandardized reporting and indexing of medical test studies, which may improve in the future. Search Principle 1: Do Not Rely on Search Filters Alone (1 of 2) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12 Category Optimization of Search Sensitivity/Specificity of SearchPubMed  Search String DiagnosisBroad Sensitivity/breadth (of “Diagnosis” search) 98%/74%(sensitiv*[Title/Abstract] OR sensitivity and specificity[MeSH Terms] OR diagnos*[Title/Abstract] OR diagnosis[MeSH:noexp] OR diagnostic* [MeSH:noexp] OR diagnosis,differential[MeSH:noexp] OR diagnosis[Subheading:noexp]) Narrow Specificity/narrowness (of “Diagnosis” search) 64%/98%(specificity[Title/Abstract]) Search Principle 1: Do Not Rely on Search Filters Alone (2 of 2)  Example: Filter used by PubMed  Clinical Queries  Category selected: “Diagnosis”  Two choices of scope to optimize search: “Broad” or “Narrow” Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

13  It is important to use all known variants of a test name, as in the examples below that refer to hematocrit:  Abbreviations (Hct, Crit, PCV)  Generic names (hematocrit, packed cell volume)  Proprietary names (e.g., LighTouch ® HCT)  International terms/spellings (haematocrit)  Controlled vocabulary may not contain all variants.  Controlled vocabulary is unreliable due to nonstandardized reporting and indexing.  Textwords help identify tests not yet indexed or indexed improperly.  Filters may suggest appropriate textwords. Search Principle 2: Do Not Rely on Controlled Vocabulary (Subject Headings) Alone Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14  Searching more than one database and tailoring searches to each is always advisable. Otherwise, there is a risk of bias due to little overlap between the many databases.  Until reporting/indexing is standardized, the best method is to combine highly sensitive searches and manual article screening.  Articles can still miss relevant articles, so additional sources of information are necessary.  Other sources of information:  Additional bibliographic databases  Citation tracking (i.e., reading through references of relevant articles)  Regulatory documents  Search for diagnostic documents at Devices@FDA, a U.S. Food and Drug Administration Web site. Search Principle 3: Search in Multiple Locations (1 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

15 Open-Access Databases DatabaseURLTopic Coverage IBIDS (International Bibliographic Information on Dietary Supplements) www.ods.od.nih.gov/Health_ Information/IBIDS.aspx Dietary supplements PubMed  www.ncbi.nlm.nih.gov/pubmed Biology Health sciences TOXNET (Toxicology Data Network) www.toxnet.nlm.nih.gov Toxicology Environmental health adverse effects World Health Organization Global Health Library www.who.int/ghl/medicus/en/ International biomedical topics Global Index Medicus Examples of specialized databases that have free access: Search Principle 3: Search in Multiple Locations (2 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

16 Subscription Databases DatabaseURLTopic Coverage CINAHL  (Cumulative Index to Nursing and Allied Health) www.ebscohost.com/cinahl Nursing Allied health EMBASE  www.embase.com Biomedical with emphases on drugs and pharmaceuticals More non – U.S. coverage than MEDLINE  IPA (International Pharmaceutical Abstracts) www.csa.com/factsheets/ipa-set-c.php Drugs and pharmaceuticals MANTIS (Manual Alternative and Natural Therapy Index System) www.healthindex.com/MANTIS.aspx Osteopathy Chiropractic Alternative medicine PsycINFO  www.apa.org/pubs/databases/psycinfo/ index.aspx Psychological literature Examples of specialized databases that require a subscription: Search Principle 3: Search in Multiple Locations (3 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

17 Citation-Tracking Databases DatabaseURLAccess Google Scholarwww.scholar.google.comFree PubFocuswww.pubfocus.comFree PubReMinerbioinfo.amc.uva.nl/human-genetics/pubreminerFree SciVerse  Scopuswww.info.scopus.comSubscription required Web of Science  thomsonreuters.com/products_services/science/ science_products/a-z/web_of_science Subscription required Examples of citation-tracking databases: Search Principle 3: Search in Multiple Locations (4 of 4) Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

18  AHRQ Evidence Report/Technology Assessment No. 142: Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure  All medical tests were known.  The search consisted of all possible variations on test names.  Using a search string to capture the diagnostic testing concept was not necessary.  AHRQ Comparative Effectiveness Review No. 2: Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities  All medical tests were not known.  The PubMed  search string relied on textwords used to capture the diagnostic testing concept: “diagnosis OR diagnose OR diagnostic OR di[sh] OR “gold standard” OR “ROC” OR “receiver operating characteristic” OR sensitivity and specificity[mh] OR likelihood OR “false positive” OR “false negative” OR “true positive” OR “true negative” OR “predictive value” OR accuracy OR precision” Examples of Search Strategies Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

19  Diagnostic search filters cannot be depended on exclusively.  When the full range of tests is known, searching for a specific test using all variant names may be sufficient.  Combining highly sensitive database searches (using textwords), hand searching, and searching lists of cited references is currently the best way to identify relevant studies.  Do not rely on controlled vocabulary alone.  Search multiple locations and databases.  Be sure to check the Devices@FDA Web site, which is a useful source. Key Messages Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

20 1.Generally, when designing a literature search, higher precision is desirable relative to recall. a.True b.False Practice Question 1 (1 of 2)

21 Explanation of Question 1: The statement is false. Maintaining balance between recall and precision is usually desirable. Practice Question 1 (2 of 2)

22 2.Search filters are inappropriate for systematic reviews. a.True b.False Practice Question 2 (1 of 2)

23 Explanation of Question 2: The statement is true. Search filters may miss relevant studies and do not perform as well for studies published before 1990 because of nonstandardized reporting and indexing. Practice Question 2 (2 of 2)

24 3.Which of the following are true statements regarding controlled vocabulary: i.A controlled vocabulary may not contain all variants. ii.A controlled vocabulary is reliable thanks to standardized reporting and indexing. iii.Textwords help identify tests not yet indexed or indexed improperly. Choices: a.a and b b.a and c c.b and c d.All three statements are true. Practice Question 3 (1 of 2)

25 Explanation of Question 3: The correct answer is b. Controlled vocabulary is actually unreliable for identifying variants of a test name because of nonstandardized reporting and indexing. Practice Question 3 (2 of 2)

26 4.Which of the following is not a principle of searching for studies outlined in this module: a.Do not rely on search filters alone. b.Do not rely on controlled vocabulary alone. c.Search in multiple locations. d.Avoid using Google Scholar as a search tool. Practice Question 4 (1 of 2)

27 Explanation of Question 4: The correct answer is d. Google Scholar can be useful for grey literature searches. Practice Question 4 (2 of 2)

28  This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center.  The module is based on Chapter 4, Effective Search Strategies for Systematic Reviews of Medical Tests. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm Authors

29  Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Review. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.  Astin MP, Brazzelli MG, Fraser CM, et al. Developing a sensitive search strategy in MEDLINE to retrieve studies on assessment of the diagnostic performance of imaging techniques. Radiology. 2008 May;247(2):365-73. PMID: 18372447.  Bachmann LM, Coray R, Estermann P, et al. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. J Am Med Inform Assoc. 2002 Nov-Dec;9(6):653-8. PMID: 12386115.  Bachmann LM, Estermann P, Kornenberg C, et al. Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc. 2003 Jul;91(3):341-6. PMID: 12883560. References (1 of 6)

30  Balion C, Santaguida PL, Hill S, et al. Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure. Evidence Report/Technology Assessment No. 142 (Prepared by the McMaster University Evidence-based Practice Center under Contract No. 290-02-0020). Rockville, MD: Agency for Healthcare Research and Quality; September 2006. AHRQ Publication No. 06- E014. Available at www.ncbi.nlm.nih.gov/books/NBK38136.  Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005 Jan 28;5(1):6. PMID: 15679886.  Bruening W, Launders J, Pinkney N, et al. Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities. Comparative Effectiveness Review No. 2 (Prepared by the ECRI Evidence-based Practice Center under Contract No. 290-02-0019). Rockville, MD: Agency for Healthcare Research and Quality; February 2006. AHRQ Publication No. 06-EHC005-EF. www.effectivehealthcare.ahrq.gov/repFiles/BrCADx%20Final%20Report.pdf. References (2 of 6)

31  Conn VS, Isaramalai SA, Rath S, et al. Beyond MEDLINE for literature searches. J Nurs Scholarsh. 2003;35(2):177-82. PMID: 12854300.  Deville WL, Bezemer PD, Bouter LM. Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol. 2000 Jan;53(1):65-9. PMID: 10693905.  Doust JA, Pietrzak E, Sanders S, et al. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol. 2005 May;58(5):444-9. PMID: 15845330.  Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. MBJ. 2004 May 1;328(7447):1040. PMID: 15073027.  Haynes RB, Wilczynski N, McKibbon KA, et al. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994 Nov-Dec;1(6):447-58. PMID: 7850570. References (3 of 6)

32  Honest H, Bachmann LM, Khan K. Electronic searching of the literature for systematic reviews of screening and diagnostic tests for preterm birth. Eur J Obstet Gynecol Reprod Biol. 2003 Mar 26;107(1):19-23. PMID: 12593888.  Leeflang MM, Scholten RJ, Rutjes AW, et al. Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies. J Clin Epidemiol. 2006 Mar;59(3):234-40. PMID: 16488353.  Relevo R. Effective search strategies for systematic reviews of medical tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 4.1-4.9. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.  Ritchie G, Glanville J, Lefebvre C. Do published search filters to identify diagnostic test accuracy studies perform adequately? Health Info Libr J. 2007 Sep;24(3):188-92. PMID: 17714173. References (4 of 6)

33  Sampson M, Barrowman NJ, Moher D, et al. Should meta-analysts search EMBASE in addition to MEDLINE? J Clin Epidemiol. 2003 Oct; 56(10):943-55. PMID: 14568625.  Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004 Dec;12(4):228-32. PMID: 15649836.  Suarez-Almazor ME, Belseck E, Homik J, et al. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000 Oct;21(5):476-87. PMID: 11018564.  van der Weijden T, Ijzermans CJ, Dinant GJ, et al. Identifying relevant diagnostic studies in MEDLINE. The diagnostic value of the erythrocyte sedimentation rate (ESR) and dipstick as an example. Fam Pract. 1997 Jun;14(3):204-8. PMID: 9201493. References (5 of 6)

34  Vincent S, Greenley S, Beaven O. Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Info Libr J. 2003 Sep;20(3):150-9. PMID: 12919278.  Whiting P, Westwood M, Beynon R, et al. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clin Epidemiol. 2011 Jun;64(6):602-7. PMID: 21075596.  Whiting P, Westwood M, Burke M, et al. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008 Apr;61(4):357-64. PMID: 18313560.  Wilczynski NL, Haynes RB. EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC. Med 2005 Mar;3:7. PMID: 15796772.  Zheng MH, Zhang X, Ye Q, et al. Searching additional databases except PubMed are necessary for a systematic review. Stroke. 2008 Aug;39(8):e139. PMID: 18583557. References (6 of 6)


Download ppt "Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules."

Similar presentations


Ads by Google