1. Introduction to the Methods Guide for Medical Test Reviews Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2.  Describe the purpose of the Methods Guide for Medical Test Reviews.  Understand the unique features of medical test reviews.  Review the structure of the Methods Guide for Medical Test Reviews. Learning Objectives Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3.  Systematic reviews of medical tests help determine which tests can be relied on to improve health outcomes and the circumstances required for the tests to achieve this.  Systematic reviews developed under the auspices of the Evidence-based Practice Center (EPC) Program:  Need to be technically comprehensive and practical  Need to be completed with limited time and resources Medical Test Reviews (1 of 4) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4.  The Methods Guide for Medical Test Reviews:  Provides practical guidance for those who prepare and use systematic reviews.  Promotes consistency and rigor in how specific issues are addressed.  Complements the Methods Guide for Effectiveness and Comparative Effectiveness Reviews developed by the Agency for Healthcare Research and Quality. Medical Test Reviews (2 of 4) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Health Care Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

5.  Medical tests are defined here as any test used in a health care context, for example, screening, diagnosis, or prognosis.  Using medical tests depends on the context.  Pre-existing conditions  Results of other tests  Skill and knowledge of providers  Availability of therapeutic resources  And others Medical Test Reviews (3 of 4) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

6.  Researchers have tended to focus on the ability of medical tests to:  Conform to technical specifications  Classify patients into diagnostic or prognostic categories  Influence thought or actions by clinicians and patients  Rarely are medical tests evaluated in randomized controlled trials with representative patient populations and comprehensive measures of patient- relevant outcomes. Medical Test Reviews (4 of 4) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

7.  Creation of the Methods Guide for Medical Test Reviews was guided by two tenets: 1.Evaluation of a medical test must be linked to the context of its use 2.Medical tests are about improving patient outcomes Development of the Methods Guide for Medical Test Reviews (1 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Health Care Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

8.  The first tenet is to consider the context.  Medical tests are not neutral reporters of reality. The context often affects sensitivity, specificity, and the clinical implications of a particular test result.  Therefore, systematic reviews must specify the context in which the test is being used.  The second tenet is to focus on patient outcomes.  The ultimate purpose of medical tests is to improve patient outcomes.  Improvement occurs by guiding the judgments of clinicians concerning disease state or treatment response. Development of the Methods Guide for Medical Test Reviews (2 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

9.  Most of the published literature:  Fails to address the clinical impact of tests.  Focuses instead on test development and performance characteristics.  The first step in a medical test review must be to establish a link between the use of a test and patient outcomes. Development of the Methods Guide for Medical Test Reviews (3 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Causal Chain Diagram Test  Result  Categorization*  Decision  Patient Outcome *For example: high risk, disease present, disease progression

10.  Typical uses of medical tests are to:  Screen for the likelihood of a disorder.  Diagnose the presence of disease.  Assess response to treatment.  Assess probability of desirable/undesirable consequences. Medical Tests Overview (1 of 2) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

11.  Typical types of medical tests:  Laboratory tests  Radiological imaging  Patient history  Physical examinations  Scored questionnaires  For screening  For determining likely prognosis  For assessing the potential response to therapy Medical Tests Overview (2 of 2) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12.  Assessing the impact of medical tests can be challenging.  Medical tests have several steps between conducting the test and determining the outcome of clinical importance.  Conversely, most treatments lead directly to intended results or adverse effects.  Consequences result from the less direct relationship between medical tests and patient outcomes.  Medical tests are often evaluated in isolation for their ability to detect a substance or a particular anatomic condition.  Medical tests are not evaluated in terms of their impact on overall health outcomes. Challenges Specific to Medical Tests (1 of 2) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Siebert U. Eur J Health Econ 2003;4(3):143-50 Tatsioni A, Zarin DA, Aronson N, et al. Ann Intern Med 2005 Jun 21;142(12 Pt 2):1048-55. PMID: 15968029.

13.  Randomized controlled trials, generally considered the strongest form of evidence, are rarely performed for medical tests.  Trials are difficult to carry out because of the numerous steps and the variables between the test and patient outcomes. Challenges Specific to Medical Tests (2 of 2) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14.  Medical tests can be likened to an electronic signal detector for the presence or absence of a clinical condition.  Signal detection theory (e.g., sensitivity, specificity) can be used to calculate disease probabilities for positive or negative test results. Using Medical Tests (1 of 2) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Green DM, Swets JA. Signal detection theory and psychophysics. New York, NY: Wiley; 1966. Reprinted 1988. Ledley RS, Lusted LB. Science 1959 Jul 3;130(3366):9-21. PMID: 13668531. Yerushalmy J. Public Health Rep 1947 Oct 3;62(40):1432-49. PMID: 20340527.

15. Medical test evaluation studies fall along a continuum related to the study objectives. Using Medical Tests Study ObjectiveTerms UsedExamples Test conforming to technical specifications Technical efficacy Technical quality of a radiological image Analytic validity Accuracy of a chemical assay for the target analyte Concordance of a commercial genetic test with the true genotype Test classifying a patient into a disease/phenotype or prognosis category Diagnostic accuracy efficacy Clinical validity Test accuracy Test performance Performance characteristics Operating characteristics Sensitivity and specificity Positive and negative likelihood ratios Positive and negative predictive value Test yield Receiver operating characteristic curve Test directing clinical management and improving patient outcomes Diagnostic thinking efficacy Therapeutic efficacy Patient outcome efficacy Clinical utility Impact on mortality or morbidity Impact on clinician judgment about diagnosis/ prognosis Impact on choice of management Test benefiting society as a whole Societal efficacy Incremental cost-effectiveness Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Ɩ Fryback DG, Thornbury JR. Med Decis Making 1991 Apr-Jun;11(2):88-94. PMID: 1907710.

16.  Reviewers need an explicit strategy for summarizing complex literature in a logical way to respond to key questions.  Frameworks can be used to:  Maintain an orderly process.  Clarify questions.  Organize evidence. Analytic Frameworks (1 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Bravata EP, McDonald KM, Shojania KG, et al. Ann Intern Med 2005 Jun 21;142(12 Pt 2): 1056-65. PMID: 15968030. Mulrow C, Langhorne P, Grimshaw J. Ann Intern Med 1997 Dec 1;127(11):989-95. PMID: 9412305. Whitlock EP, Orleans CT, Pender N, et al. Am J Prev Med 2002 May;22(4):267-84. PMID: 11988383.

17.  Analytic frameworks:  Depict graphically the linkages that relate interventions and outcomes.  Identify questions to help structure the literature review.  Provide an evidence map after the review for identifying gaps and weakness in the evidence. Analytic Frameworks (2 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. U.S. Preventive Services Task Force Procedure Manual. Available at www.uspreventiveservicestaskforce.org/uspstf08/methods/procmanual.htm.

18.  An analytic framework has two key components:  A typology for describing the context in which the test is used:  Fits with the PICOTS (population, intervention, comparator, outcomes, timing, and setting) typology (the current standard approach).  A visual representation of the relationship between the application of the test/treatment and the outcomes of importance to decisionmaking. Analytic Frameworks (3 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

19.  PICOTS stands for population, intervention, comparator, outcome, timing, and setting.  This typology was developed to describe the context in which medical interventions might be used.  It is important for defining the key questions of a review.  It is important for assessing whether or not a given study is applicable.  Variations in the typology exist, but the standard unchanging elements are PICO. PICOTS Typology (1 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Chalmers I, Hedges LV, Cooper H. Eval Health Prof 2002 Mar;25(1):12-37. PMID: 11868442.

20. PICOTS Typology (2 of 3) PICOTS As Applied to Medical TestsComments PopulationPatient population; includes results of other/previous tests Condition(s), disease severity and stage, comorbidities, and patient demographics InterventionIndex test; includes clinical role of the index strategy in relation to comparator and the test-and-treat strategy in relation to clinical outcomes Description of index test; includes administrator training, technology specifications, specific application issues Three main clinical roles in relation to the comparator: replacement, add-on, and triage Description of index test performance and interpretation; how the results of index test lead to management decisions/actions ComparatorComparator test-and-treat strategy Description of comparator test performance and interpretation; how the results of the comparator test lead to management decisions/actions Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

21. PICOTS Typology (3 of 3) PICOTS As Applied to Medical TestsComments OutcomesRelevant clinical outcomes; includes any intermediate outcomes of interest Patient health outcomes; includes morbidity (which also includes the adverse effects of the test and treatment), mortality, quality of life Intermediate outcomes; includes technical specifications, accuracy, decisional, therapeutic impact TimingTiming of outcome assessment Duration of followup; single or multiple followup assessments SettingSetting of test assessmentAmbulatory settings (including primary, specialty care) and inpatient settings Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

22.  Module 1 – Introduction  Module 2 – Topic Development and Structure  Module 3 – Choosing Outcomes  Module 4 – Search Strategies  Module 5 – Risk of Bias  Module 6 – Applicability  Module 7 – Grading Strength of Evidence  Module 8 – Meta-analysis With a “Gold Standard”  Module 9 – Meta-analysis Without a “Gold Standard”  Module 10 – Decision Modeling  Module 11 – Genetic Tests  Module 12 – Prognostic Tests Organization of the Methods Guide for Medical Test Reviews Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

23. Overview of the Flow of Modules Prepare Topic 2. Develop the Topic and Structure the Review 3. Choose the Important Outcomes Research Sources Search for and Select Studies for Inclusion 4. Search for Studies Extract Data From Studies Analyze and Synthesize Studies 5. Assess Risk of Bias as a Domain of Quality 6. Assess Applicability 7. Grade the Body of Evidence 8. Meta-analysis of Test Performance Evidence With a “Gold Standard” — or — 9. Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard 10. Decision Modeling Report Medical Test Review Encompassing the Entire Process: 11. Genetic Tests 12. Prognostic Tests Encompassing the Entire Process: 11. Genetic Tests 12. Prognostic Tests Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

24.  Evaluation of medical tests presents challenges:  There are several steps between a medical test and outcomes of interest (indirect impact).  Context has great importance.  Few randomized controlled trials are aimed at comparing clinical outcomes of different tests and test strategies.  Available guidance provides some suggestions:  Use the PICOTS (population, intervention, comparator, outcomes, timing, and setting) topology to clarify the context.  Use an organizing framework for classifying the types of studies and their relationship to key questions. Summary Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

25. 1.How are analytic frameworks used in the context of this module? a.To graphically depict the specific questions that need to be answered by the literature review. b.To list the challenges associated with a review of a medical test. c.To demonstrate recurrent themes from the literature. d.To display all areas of potential bias. Practice Question 1 (1 of 2)

26. Explanation for Question 1: The correct answer is a. The analytic framework should be used to clearly present the questions that need to be answered in the literature review. The questions are shown by linkages between interventions and outcomes. Practice Question 1 (2 of 2)

27. 2.Test accuracy is included in which part of the PICOTS typology? a.P for patient population b.I for intervention c.C for comparator d.O for outcomes Practice Question 2 (1 of 2)

28. Explanation for Question 2: The correct answer is d. The “O” in the PICOTS typology stands for outcomes, which include patient health outcomes and intermediate outcomes such as test accuracy. Practice Question 2 (2 of 2)

29. 3.Most of the published literature on medical tests addresses the clinical impact of tests. a.True b.False Practice Question 3 (1 of 2)

30. Explanation for Question 3: The statement is false. Most of the published literature on medical tests focuses on test development and test characteristics and not on the clinical impact of tests. This poses a challenge when reviewing medical tests. Practice Question 3 (2 of 2)

31. 4.A causal chain diagram is used to: a.Show possible sources of bias b.Show the links between the use of a test and the patient outcomes c.Help with decisionmaking based on a positive or negative test result d.Display the method used for the literature search Practice Question 4 (1 of 2)

32. Explanation for Question 4: The correct answer is b. A causal chain diagram is used to show visually the links between the use of a test and patient outcomes. Practice Question 4 (2 of 2)

33.  This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center.  The module is based on Chapter 1, Introduction to the Methods Guide for Medical Test Reviews. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Authors

34.  Agency for Healthcare Research and Quality. U.S. Preventive Services Task Force Procedure Manual. Rockville, MD: Agency for Healthcare Research and Quality; July 2008. 22-4. AHRQ Publication No. 08-05118-EF. www.uspreventive servicestaskforce.org/uspstf08/methods/procmanual.htm.  Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methods guide.cfm.  Bravata DM, McDonald KM, Shojania KG, et al. Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med. 2005 Jun 21;142(12 Pt 2):1056-65. PMID: 15968030. References (1 of 4)

35.  Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof 2002 Mar;25(1):12-37. PMID: 11868442.  Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991 Apr-Jun;11(2):88-94. PMID: 1907710.  Green DM, Swets JA. Signal detection theory and psychophysics. New York, NY: Wiley; 1966. Reprinted in 1988 with corrections and an updated topical bibliography (Los Altos, CA: Peninsula Publishing).  Ledley RS, Lusted LB. Reasoning foundations of medical diagnosis; symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science 1959 Jul 3;130(3366):9-21. PMID: 13668531. References (2 of 4)

36.  Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 1.1-1.11. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/ medtestsguide.cfm.  Mulrow C, Langhorne P, Grimshaw J. Integrating heterogeneous pieces of evidence in systematic reviews. Ann Intern Med. 1997 Dec 1;127(11):989-95. PMID: 9412305.  Siebert U. When should decision analytic modeling be used in the economic evaluation of health care? Eur J Health Econ. 2003;4(3):143-50.  Tatsioni A, Zarin DA, Aronson N, et al. Challenges in systematic reviews of diagnostic technologies. Ann Intern Med. 2005 Jun 21;142(12 Pt 2):1048-55. PMID: 15968029. References (3 of 4)

37.  Whitlock EP, Orleans CT, Pender N, et al. Evaluating primary care behavioral counseling interventions: an evidence-based approach. Am J Prev Med. 2002 May;22(4):267-84. PMID: 11988383.  Yerushalmy J. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep. 1947 Oct 3;62(40):1432-49. PMID: 20340527. References (4 of 4)