1. Selecting Evidence for Comparative Effectiveness Reviews Melissa McPheeters, PhD., MPH Associate Director, Vanderbilt University Evidence-based Practice Center

2. Overview Why use selection criteria? Why use selection criteria? What do they look like in the context of a systematic review? What do they look like in the context of a systematic review? When should they be set? When should they be set? How do they affect the conduct and interpretation of the review? How do they affect the conduct and interpretation of the review?

3. CER Process Overview

4. Study Selection Criteria Function the same in systematic reviews as in primary research Function the same in systematic reviews as in primary research Should reflect the analytic framework and key questions Should reflect the analytic framework and key questions Are powerful tools for widening or narrowing the scope of a review Are powerful tools for widening or narrowing the scope of a review Provide information to determine whether reviews can be compared or combined Provide information to determine whether reviews can be compared or combined

5. Types of Criteria Population descriptors Population descriptors Intervention descriptors Intervention descriptors Outcome descriptors Outcome descriptors Study design Study design Study size Study size Source of literature (including grey lit) Source of literature (including grey lit)

6. Some Example Criteria Adult, community dwelling females Adult, community dwelling females Study of a screening tool for depression Study of a screening tool for depression United States only United States only Hospital-based studies only Hospital-based studies only N>200 N>200 Randomized controlled trials Randomized controlled trials Articles published in English or German Articles published in English or German

7. Selecting Criteria Review study goals Review study goals Assess analytic framework and key questions Assess analytic framework and key questions Tie criteria to PICO(TS) Tie criteria to PICO(TS) Set criteria before beginning abstract review Set criteria before beginning abstract review

8. Using broad criteria Can be as broad as all related studies Can be as broad as all related studies Helpful for exploring “what is known” Helpful for exploring “what is known” May result in too much literature to feasibly review, or disparate literature that can’t be compared. May result in too much literature to feasibly review, or disparate literature that can’t be compared.

9. Using narrow criteria May return too little literature May return too little literature Can result in inability to answer the intended question Can result in inability to answer the intended question Helpful in culling homogenous literature Helpful in culling homogenous literature Can reduce size of the literature to a manageable scope Can reduce size of the literature to a manageable scope

10. Bias in this context Distortion of the estimate of effect that comes from how studies are selected to be included Distortion of the estimate of effect that comes from how studies are selected to be included Affects the applicability or “external validity” of the review itself Affects the applicability or “external validity” of the review itself

11. For example, included studies may not have been conducted in the patient population whose care the review is intended to affect For example, included studies may not have been conducted in the patient population whose care the review is intended to affect – e.g., the use of studies of twin pregnancies in a review of preterm labor management for low risk women Or – selection criteria may be set to include more of a certain study type that either over or under estimates effectiveness Or – selection criteria may be set to include more of a certain study type that either over or under estimates effectiveness

12. What is the relevant population? What is the relevant population? What is the intervention of interest? What is the intervention of interest? To what exposure is the intervention being compared? To what exposure is the intervention being compared? What outcomes are relevant? What outcomes are relevant? Should time to outcome be limited? Should time to outcome be limited? In what setting should the results be applicable? In what setting should the results be applicable? Basic Questions

13. Considerations Grey literature: New technologies may have substantial grey literature, including govt. reports and trade journals Grey literature: New technologies may have substantial grey literature, including govt. reports and trade journals Non-English articles may be useful for certain types of interventions (e.g. CAM) Non-English articles may be useful for certain types of interventions (e.g. CAM) Study size limitations may depend on what is available Study size limitations may depend on what is available – Easier to set for observational studies – Large bodies of efficacy/effectiveness lit may be made up of small trials

14. Exercise 1 What would you do … What would you do … – If you were asked to review the literature on transition support for adolescents with autism entering adulthood? – Before seeing the key questions, consider the categories of criteria that we will want to apply.

15. Apply PICOTS Population – condition, disease severity and stage, co-morbidities, patient demographics Population – condition, disease severity and stage, co-morbidities, patient demographics Intervention – dosage, frequency, and method of administration Intervention – dosage, frequency, and method of administration Comparator – placebo, usual care, or active control Comparator – placebo, usual care, or active control Outcome – health outcomes, morbidity, mortality, QoL Outcome – health outcomes, morbidity, mortality, QoL Timing – Duration of follow-up Timing – Duration of follow-up Setting - Primary, specialty, in-patient, co- interventions Setting - Primary, specialty, in-patient, co- interventions

16. PICOTS Population Intervention Comparators Outcomes: Timing Setting What constitutes an adolescent? What constitutes a diagnosis of ASD? How is transition support defined? Do we compare to no transition support or directly compare types of support? What are the goals for adolescents with ASD as they transition to adulthood? Should they be individually focused? How quickly should the outcomes be apparent? Is transition support provided in multiple settings such as schools, clinics and the community?

17. What would you do with … A paper that was about “individuals over age 10?” A paper that was about “individuals over age 10?” A paper that was about an intervention for individuals with a range of developmental disabilities? A paper that was about an intervention for individuals with a range of developmental disabilities? Or conversely, about children specifically with Asperger syndrome but not other ASDs? Or conversely, about children specifically with Asperger syndrome but not other ASDs?

18. Example of a narrow scope What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth? What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?

19. Selection Criteria Efficacy: RCTs only Efficacy: RCTs only Low risk: No prior preterm birth Low risk: No prior preterm birth No. of studies: 11 studies No. of studies: 11 studies

20. OAB Study Management of Overactive Bladder among Women Management of Overactive Bladder among Women Considerations: Considerations: – OAB is a fairly difficult condition to define – Treatments include pharmacologic, behavioral, CAM and procedural – and each area includes very different types of studies – Its study is often combined with other types of urological conditions such as stress incontinence or prostate issues

21. Exercise 2 Set two criteria and consider the potential implications: Set two criteria and consider the potential implications: – – Minimum study size – – Gender of study participants

22. Study Size 50 at study start 50 at study start Implications: Implications: – Excluded for size only:79 – Excluded for N < 20:36 – Excluded for N 20-2923 – Excluded for N 30-398 – Excluded for N 40-4912

23. Gender Studies had to include at least 75% women Studies had to include at least 75% women This decision was based on expert opinion and the size and scope of the literature This decision was based on expert opinion and the size and scope of the literature 40 studies were excluded with less than 75% women 40 studies were excluded with less than 75% women 27 additional studies would have been excluded had the review been limited to studies of only women 27 additional studies would have been excluded had the review been limited to studies of only women

24. Other Considerations What study designs should be included? What study designs should be included? Include foreign studies? Other languages? Studies conducted in specific parts of the world? Include foreign studies? Other languages? Studies conducted in specific parts of the world? Is the “grey” or “fugitive” literature included? Is the “grey” or “fugitive” literature included?

25. Limit to RCTs? Limit to RCTs? Include observational studies? Include observational studies? If so, which kinds? If so, which kinds? What is the value of case series? What is the value of case series? What counts as a case series? What counts as a case series? Types of Studies?Types of Studies?

26. Example 3 Maternal-fetal Surgery Technical Brief Maternal-fetal Surgery Technical Brief – Included case series with N>=2 – Only 3 of 169 studies were RCTs and 122 were case series – Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups

27. Observational Studies Types Types – Cohorts (with comparisons) – Case controls – Case series – Registries/databases

28. Observational Studies Well-done observational studies can address issues of applicability and need for longer-term outcomes if they: Well-done observational studies can address issues of applicability and need for longer-term outcomes if they: – Include more representative patient populations – Have relevant comparators – Report more meaningful clinical outcomes over longer timeframes – May be a better source of information about harms

29. Foreign Literature Positive findings may be more likely to be published in high-profile journals published in English Positive findings may be more likely to be published in high-profile journals published in English Therefore, to include only English may overestimate the positive effect of an intervention Therefore, to include only English may overestimate the positive effect of an intervention Empirically, the bias associated with limiting one’s review to English only has been shown to be small Empirically, the bias associated with limiting one’s review to English only has been shown to be small Some content areas may be more heavily published in the foreign literature (e.g., some CAM interventions) Some content areas may be more heavily published in the foreign literature (e.g., some CAM interventions)

30. Cesarean Delivery NIH-OMAR State-of-the-Science conference on outcomes of cesarean delivery on maternal request NIH-OMAR State-of-the-Science conference on outcomes of cesarean delivery on maternal request

31. Exercise 3 Define the appropriate population group and comparator. Define the appropriate population group and comparator. What other limitations would you put on included literature? What other limitations would you put on included literature?

32. The challenge No evidence on outcomes of CDMR vs other modes of delivery No evidence on outcomes of CDMR vs other modes of delivery Urgent need for actionable evidence Urgent need for actionable evidence Need to recognize and account for confounders Need to recognize and account for confounders

33. Solution Expand search to include proxies, weight rungs of evidence to account for confounding Expand search to include proxies, weight rungs of evidence to account for confounding – Highest rung: Trials of breech delivery, but only for maternal outcomes – Next rung: Planned cesarean versus planned vaginal – Lowest rung: Comparisons of maternal and neonatal outcomes from actual modes of delivery

34. Summary Selection criteria (aka inclusion/exclusion) are an essential tool for setting the scope of the review Selection criteria (aka inclusion/exclusion) are an essential tool for setting the scope of the review They should be tied to the analytic framework, key questions and PICOTS They should be tied to the analytic framework, key questions and PICOTS When properly applied, selection criteria can reduce bias and support the applicability of the review When properly applied, selection criteria can reduce bias and support the applicability of the review