Recommendation Methods Advisory Committee on Heritable Disorders and Genetic Diseases of Newborns and Children Ned Calonge, M.D., M.P.H.

Workgroup Members l Denise Dougherty, Piero Rinaldo, Coleen Boyle, Michael Watson, Tracy Trotter, Sharon Terry l Liaison from External Workgroup: Nancy Green, MD, Columbia University l Liaison/Committee Staff: Michele Puryear

Process for creating recommendations based on Systematic Evidence Review l Anticipate not having direct evidence of screening efficacy l Create chain of evidence, evaluating »Analytic validity »Clinical validity »Clinical utility l Base recommendation on certainty of net benefit

Steps in process l Define the question regarding testing for the specific question within an analytic framework that includes an overarching key question (direct evidence) and a chain of related key questions (indirect evidence)

Figure 1. Analytic Framework General population of newborns Testing for condition 3 Mortality, morbidity, and other outcomes Harms of testing/identification Harms of treatment/other interventions 2 Identification of condition Treatment of Condition 6

Key question 1 l Is there direct evidence that screening for the condition at birth leads to improved health outcomes? (overarching question) l Best evidence would be randomized trials involving screen-detected infants l For many conditions considered by the Advisory Committee, it is unlikely that there will be direct evidence

Key question 2 l What is known about the condition? »Is the condition well-defined and important? »What is the incidence of the condition in the U.S. population? »What is the spectrum of disease for the condition? »What is the natural history of the condition, including the impact of recognition and treatment?

Key question 3 l Is there a test for the condition with sufficient analytic utility and validity? l Refers to the laboratory performance of the test »analytic reliability »assay robustness

Key question 4 l Does the test accurately and reliably detect the condition and clinical disease? Clinical validity: »Sensitivity »Specificity »Positive predictive value »False positive rate l Measures must relate to clinical/ symptomatic disease (phenotype)

Key question 5 l Are there available treatments for the condition that improve important health outcomes? l Does treatment of the condition detected through NBS improve important health outcomes when compared with waiting until clinical detection? l Are there subsets of affected children more likely to benefit from treatment that can be identified through testing or clinical findings? l Are the treatments for affected children standardized, widely available, and if appropriate, FDA approved?

Key question 5 l The Advisory Committee will need to determine which outcomes should be considered as important health outcomes »Patient outcome impacts (mortality/morbidity) »Therapeutic/management decisions »Diagnostic thinking/health information impact »Familial and Societal impact Not necessarily of equal weight

Key questions 6 & 7 l Are there harms or risks identified for the identification? Harms or risks for treatment of affected children? »Harms of screening, including ELSI »Harms of diagnostic workup for screen positives »Harms of treatment (especially if no benefit, or if provided to false positives)

Key question 8 l What is the estimated cost-effectiveness of testing for the condition? »It is unlikely there will be empiric data »May be addressed through decision modeling, which can provide estimates that the Advisory Committee will take into consideration when considering a recommendation

Translating evidence into recommendations l Judgment regarding the magnitude of net benefit (benefits minus harms) l Judgment of the adequacy of evidence in answering the key questions l Judgment of the certainty of net benefit

Magnitude of net benefit l Significant: benefits clearly outweigh harms l Zero/net harm (more harm than benefit) l Small net benefit »Must carefully consider level of certainty, other issues such as cost effectiveness

Adequacy of evidence l Evidence should be classified as adequate or inadequate l Inadequate evidence for a key question represents a “break” in the evidence chain that would lead to a finding of insufficient certainty of net benefit l Adequacy should be determined by applying a set of critical appraisal questions to each key question

Critical appraisal questions 1. Do the studies have the appropriate research design to answer the key question? 2. To what extent are the studies of high quality (internal validity)? 3. To what extent are the studies generalizable to the U.S. population (external validity)? 4. How many studies and how large have been done to answer the key question (precision of the evidence)? 5. How consistent are the studies? 6. Are there additional factors supporting conclusions?

Judge the certainty of net benefit l Based on the evidence, estimate the magnitude of benefit or potential benefit l Based on the evidence, estimate the magnitude of harm or potential harm l Estimate net benefit (benefits minus harms) l Base judgment of certainty of net benefit through applying critical appraisal questions across the chain of evidence

Certainty l Sufficient: evidence is sufficient to determine effects on health outcomes with an acceptable risk or level of comfort of “being wrong” and thus a low susceptibility to being overturned or otherwise altered by additional research l Insufficient: evidence is insufficient to assess effects on health outcomes; additional information from future studies may allow for assessment

Insufficient certainty, but compelling contextual issues l There may be conditions where the evidence is inadequate to reach a conclusion, but contextual issues support a recommendation to add the condition, with a commitment to fill in the gaps in evidence going forward l Contextual issues might include: »Known benefits associated with testing (and intervention) for similar conditions »High incidence that would translate to potential substantial net benefit »Availability of promising but yet unproven new therapies »Indirect evidence of perhaps less important health outcomes, but with evidence of low potential harms

Recommendation matrix RECOMMENDATIONLEVEL OF CERTAINTYMAGNITUDE OF NET BENEFIT Recommend adding the condition to the core set SufficientSignificant (note—special considerations in net benefit if small) Recommend not adding the condition to the core set SufficientZero or net harm (more harm than benefit) Recommend adding the condition with “provisional status” Insufficient, but potential net benefit is compelling, add and re-evaluate Potentially significant, supported by contextual issues Recommend not adding the condition now, but pilot studies Insufficient, additional information is needed to support a recommendation Potentially significant or unknown

Acknowledgements l Michele Puryear, Nancy Green, Piero Rinaldo and the other members of the decision process work group l The U.S. Preventive Services Task Force, especially former member Russ Harris l The Evaluating Genomic Applications in Practice and Prevention Work Group (EGAPP), especially Steve Teutsch and Glen Palomaki