Systematic Review Module 11: Grading Strength of Evidence Interactive Quiz Kathleen N. Lohr, PhD Distinguished Fellow RTI International

Interactive Case Quiz: Instructions Open this presentation as a slideshow. This will activate the hyperlinks. When you come to a decision slide, choose the red box corresponding to the correct choice. If you are correct, you will be directed forward in the case. If you are incorrect, you will be directed back to the decision slide to choose again. – – Click on forward hyperlinks (Red Boxes) to follow through case based on your responses – – Click on home hyperlink (Blue House Icon) to go back to the last correct step in the series

Case Study Quiz: Grading Strength of Evidence (SOE) You are near the end of the process of developing a CER of the benefits and harms of a therapeutic intervention. This step entails grading the strength of one or more bodies of evidence that pertain to the key questions in your CER.

Participation Quiz: Question 1 Grading SOE is the same as rating the quality of studies. They are the same thing. You can grade SOE at the same time as you rate the quality of individual studies. You can grade SOE only after you have rated the quality of individual studies.

They Are the Same Thing This is incorrect. Individual studies are rated for their quality (i.e., internal validity or risk of bias). Grading pertains to entire bodies of evidence about important key questions, outcomes, or comparisons, and the quality of individual studies is just one element in grading SOE. [Click on Blue Box to Go Back]

You Can Grade SOE and Rate Quality at the Same Time This is incorrect. Grading pertains to entire bodies of evidence about important key questions, outcomes, or comparisons. Because domains other than the quality of individual studies must be used in grading SOE, evaluating study quality and SOE together is insufficient. [Click on Blue Box to Go Back]

You Grade SOE Only After Rating Study Quality Yes, you are correct. Grading SOE is a late step in the review process. It requires judgments about several key domains, only one of which is quality of individual studies. [SELECT RED BOX]

Participation Quiz: Question 2 Grading SOE specifies required domains. What are they? Risk of bias, consistency, precision, and directness Risk of bias, consistency, precision, directness, and publication bias Risk of bias, precision, strength of association, and directness

Four Required Domains Yes, You are correct! The four required domains are – – Risk of bias: quality ratings for individual studies – – Consistency: degree of similarity in the effect sizes of different studies within an evidence base – – Precision: degree of certainty for estimate of effect with respect to a specific outcome – – Directness: whether evidence reflects a single, direct link between the interventions of interest and the ultimate health outcome under consideration, or for comparisons, whether the evidence base has head-to- head studies [SELECT RED BOX]

Four Required Domains and Publication Bias This is incorrect. The four required domains are – – Risk of bias – – Consistency – – Precision – – Directness Publication bias is an additional domain that can and should be used only when relevant. [Click on Blue Box to Go Back]

Three Required Domains and Strength of Association This is incorrect. The four required domains are – – Risk of bias – – Consistency, which is the missing required domain in this answer – – Precision – – Directness Strength of association is an additional domain referring to magnitude of effect that can be used when relevant. [Click on Blue Box to Go Back]

Participation Quiz: Question 3 Strength of evidence is graded according to the following levels or grades: High, moderate, low, very low Very high, high, moderate, low, very low High, moderate, low, insufficient

High, Moderate, Low, Very Low This is incorrect. Very low is not an option. Use four SOE grades: – – High confidence that the evidence reflects the true effect – – Moderate confidence that the evidence reflects the true effect – – Low confidence that the evidence reflects the true effect – – Insufficient: Evidence either is unavailable or does not permit a conclusion [Click on Blue Box to Go Back]

High, Moderate Low, Insufficient Yes, this is correct. Strength of evidence can have three “strength” grades relating to the confidence one has that evidence reflects the true effect and that future research might confirm or overturn the results. Insufficient implies either that one has no relevant evidence or that the evidence is so mixed and inadequate that one cannot definitely label it as high, moderate, or low. [SELECT RED BOX]

Very High, High, Moderate, Low, Very Low This is incorrect. Very high and very low are not used. The four correct options for scores are – – High – – Moderate – – Low – – Insufficient [Click on Blue Box to Go Back]

Participation Quiz: Question 4 In combining scores for domains into an overall grade for strength of evidence, you can use the following approaches: Only the GRADE algorithm Only your own weighting system Only your own qualitative approach Any of the above

Only One Approach is Allowed This is incorrect. Any of the three approaches can be used. The critical element in selecting any option is careful documentation of the methods. [Click on Blue Box to Go Back]

GRADE, Own Weighting, or Own Qualitative Approach Yes, this is correct. Any of the three approaches can be used. The critical element in selecting any option is careful documentation of the methods. [SELECT RED BOX]

Participation Quiz: Question 5 In scoring required and additional domains and in arriving at an overall grade for strength of evidence, you Use only one senior reviewer to do these tasks and report this score. Use two or more reviewers, resolve differences by consensus or adjudication by a third party, and report all scores. Use two or more reviewers, resolve differences by consensus or adjudication by a third party, and report a consensus score.

Only One Reviewer This is incorrect. You should use at least two reviewers with appropriate clinical and methodological expertise. [Click on Blue Box to Go Back]

Two or More Reviewers and Report All Score This is incorrect. You should use at least two reviewers with appropriate clinical and methodological expertise. But Differences should be mediated or adjudicated by a third party. You should report only the consensus grade. [Click on Blue Box to Go Back]

Two or More Reviewers and Consensus Score Yes, this is correct. You should – – Use at least two reviewers with appropriate clinical and methodological expertise and – – Mediate or adjudicate differences by a third party But You should report only the consensus grade.