Download presentation

Presentation is loading. Please wait.

Published byJuan Medley Modified over 2 years ago

1
Systematic Review of Prognostic Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2
Develop the topic and structure the systematic review of a prognostic test. Describe the similarities and differences between the evaluation of diagnostic and prognostic tests for systematic reviews. Explain the time-dependent characteristics of prognostic tests. Perform appropriate statistical analyses based on outcome probabilities. Learning Objectives Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3
1.Develop the review topic and framework. 2.Search for studies. 3.Select studies and assess quality. 4.Extract statistics to evaluate test performance. 5.Conduct meta-analyses of estimates of outcome probabilities. Steps Involved in Conducting a Systematic Review of a Prognostic Tests Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4
The review topic, analytic framework, and Key Questions can be fundamentally different for diagnostic and prognostic tests Diagnostic tests determine whether a patient has a disease at the time of the test. A “gold standard” (i.e., the best available) reference test is often used to determine true disease presence or absence. Prognostic tests predict a patient’s likelihood of developing a disease or experiencing a future medical event. The “reference test” is the proportion of study subjects who actually develop the condition predicted by the test. Step 1: Developing the Review Topic and Analytic Framework (1 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

5
It is often useful to group test results by implications for decisionmaking when structuring reviews Example 1: Structuring based on prognostic test categories (low/intermediate/high risk) when different treatments for each category are evaluated Example 2: Structuring based on precision and accuracy of outcome probabilities of categories when a decision model is used as a framework Step 1: Developing the Review Topic and Analytical Framework (2 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

6
Considerations for the review summarized in the general PICOTS framework: Step 1: Developing the Review Topic and Framework (3 of 4) Population Clinical spectrum and other characteristics of the prognostic groups including the observed probabilities of the outcome being predicted Intervention The prognostic test or assessment including all components; exactly what the test measures and how it is done; how clinical specimens are obtained, processed, and stored for testing; exactly what is being predicted and how the test results are to be interpreted and used by test operators ComparatorStandard prognostic tests or assessments for predicting the same outcome. Outcomes Time-dependent probabilities (time-to-event curves) of what is being predicted, changes or differences in predicted outcome probabilities or reclassification of patients into different prognostic groups, changes in patient care, the net effect of using the prognostic test on patient outcomes, and cost-effectiveness. Timing The stage in the natural history of outcome development the prognostic test to be used; the followup time covered by the prognostic test cover; the percentage of patients who experience the outcome usually increases with time, thereby changing the performance characteristics of prognostic tests. Setting Who will use the prognostic test; how the test will be used; what the applicable testing scenario is. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

7
In some contexts, it is informative to: Categorize subjects as those who did/did not experience the predicted outcome during a specified time interval. Consider what followup times are informative to patients, clinicians, or policymakers. Look back to categorize results of the test. This step permits assessment of accuracy by calculating sensitivity, specificity, and predictive values. Prognostic tests can be specifically used to predict response to a treatment. They predict beneficial/adverse responses to treatment. They are also referred to as predictive tests Step 1: Developing the Review Topic and Framework (4 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

8
Studies can relate to one or more categories: 1.Proof of concept: Is the test result associated with a clinically important outcome? 2.Prospective clinical validation: How accurately does the test predict outcomes in different patient cohorts, clinical practices, and prognostic groups? 3.Incremental predictive value: How much does the new test change predictive probabilities and increase the discrimination of patients who did/did not experience the outcome of interest within a specific time period? 4.Clinical utility: Does the new assessment change predicted probabilities enough to reclassify many patients into different prognostic groups that would be managed differently? 5.Clinical outcomes: Would use of the prognostic test improve patient outcomes? 6.Cost-effectiveness: Do the improvements in patient outcomes justify the additional costs of testing and subsequent medical care? Step 2: Searching for Studies (1 of 5) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

9
The first four categories are most readily addressed by large cohort studies and secondary analyses of clinical trials. For the last two categories, randomized control trials (RCTs) are preferred. RCTs are rare because of barriers including costs and the time involved. In the event that no RCTs are found, use a decision model to focus on providing best estimates of outcome probabilities. Step 2: Searching for Studies (2 of 5) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

10
There are no reliable, validated methods for searching the literature for prognostic test information. Some strategies use key words and index terms in studies meeting selection criteria. Others use search terms such as “incidence” and word roots such as “prognos*”. Terms describing the prognostic test and condition/event to be predicted should be included in the search. “Find similar” or “related article” functions can be useful. A manual search of references will be needed. The records of the regulatory agency for submitted tests can be useful. The Web site of test producer may be informative. Step 2: Searching for Studies (3 of 5) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

11
Unlike diagnostic tests, many prognostic tests are incorporated into multivariable regression models/ algorithms for prediction. Many reports support only an independent association of a variable with the patient outcome. It is difficult to find reports where the test variable did not add significantly to a multivariable regression model. This introduces potential bias by failing to uncover the lack of association/predictive value. All studies including the variable should be sought out. Step 2: Searching for Studies (4 of 5) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12
When prognostic groups are defined by predicted outcome probabilities: Search for decision analyses, guidelines, or expert opinions that support outcome probability thresholds used to define clinically meaningful prognostic groups (i.e., groups that would be treated differently in practice because of their predicted outcome) Ideally use randomized controlled trials of interventions in patients selected on the basis of the prognostic test. This helps establish the rationale for using the test to classify patients. It is not always sufficient to evaluate this use of the test. Step 2: Searching for Studies (5 of 5) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

13
Prognostic indicators vary substantially in: Study design Subject inclusion criteria Methods of measuring key variables Methods of analysis (including definition of prognostic groups) Adjustment for covariates Presentation of results Reviewer access to patient-level data would allow more uniform analyses to overcome these difficulties. When these data are lacking, following certain questions can help. Step 3: Selecting Studies and Assessing Their Quality (1 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14
1.Was the study designed to evaluate the new prognostic test, or was it a secondary analysis of data collected for other purposes? 2.Were the subjects somehow referred or selected for testing? What was the testing scenario? 3.Was the clinical population clearly described including the sampling plan, inclusion and exclusion criteria, subject participation, and the spectrum of test results? Did the sample represent patients who would be tested in clinical practice? 4.Did everyone in the samples have a common starting point for followup with respect to the outcome of interest, including any treatments that could affect the outcome being predicted? Questions for Judging the Quality of Individual Studies of Prognostic Tests (1 of 3) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

15
5.Was the prognostic test clearly described and conducted using a standardized, reliable, and valid method? a.Was the test used and interpreted the same way by all sites/studies, including any indeterminate test results? b.Were the results ascertained without knowledge of the outcome? c.Were investigators blinded to the test results? d.How were previously established prognostic indicators or other prognostic assessments included in the study and analyses? 6.Was the outcome being predicted clearly defined and ascertained using a standardized, reliable, and valid method? a.How complete was the followup of subjects, and were losses to followup related to the test results or the outcome being predicted? b.Was the duration of followup adequate? Questions for Judging the Quality of Individual Studies of Prognostic Tests (2 of 3) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

16
7.Were the data used to develop the prognostic test? a.Were the prognostic groups predefined based on clinically meaningful decision thresholds for predicted outcome probabilities? b.Were the results externally validated using an independent sample or internally validated using bootstrap or cross-validation methods? c.Were previously established prognostic tests that were used as comparators fitted to the sample data in the same manner as the potential new prognostic test? d.Were outcome predictions adjusted for any other factors? Which ones? How? Questions for Judging the Quality of Individual Studies of Prognostic Tests (3 of 3) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

17
Reviewers should explicitly state inclusion/exclusion criteria Test comparisons should use data from the same cohort of subjects to minimize confounding Within a study, the tests being compared should be conducted at the same time. This ensures a common starting point with respect to the patient outcome being predicted. Reviewers should note the starting point of each study. Step 3: Selecting Studies and Assessing Their Quality (2 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

18
Prognostic test results and their interpretation should be ascertained without knowledge of outcomes to avoid ascertainment bias. Investigators should be blinded to the results of a test to avoid selective changes in treatment that could affect the outcome being predicted. Be aware of any previously established prognostic indicators that should be included in the comparative analysis of potential new tests. Note any adjustments for covariates that could make studies more or less comparable. Step 3: Selecting Studies and Assessing Their Quality (3 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

19
Fitting a new prognostic test to the sample data (test development sample) by using these data to define cutoffs or model relationships to outcomes and estimate regression coefficients may: Be overly optimistic for estimated predicted performance Bias the comparison to an established prognostic method that was not fitted to the sample data Step 3: Selecting Studies and Assessing Their Quality (4 of 4) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

20
Summary statistics must be appropriate for the review’s Key Questions. Example: Hazard ratios from Cox regression analyses or odds ratios from logistic regression analyses for associations between tests and outcomes Address only the early phases in test development Often do not discriminate between subjects who eventually do or do not experience the outcome of interest Statistically significant associations (odds/hazard ratios, relative risks) merely indicate that more definitive evaluation of a new predictor is warranted. Due to these concerns, the questions that a systematic review can answer by summarizing a test’s association with an outcome are limited. Step 4: Extracting Statistics To Evaluate Test Performance (1 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

21
Discrimination statistics Indices of discrimination Estimates of sensitivity, specificity, and area under the receiver operating characteristic curve are calculated at one particular time. They can be calculated retrospectively and compared when: A new prognostic indicator is added to a predictive model A prognostic test is compared to predictions made by other methods Retrospective indices of discrimination do not: Summarize predicted outcome probabilities Directly address questions about the predictions on the basis of a new prognostic test or its impact on patient outcomes Step 4: Extracting Statistics To Evaluate Test Performance (2 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

22
Reclassification tables are better suited for assessments of the clinical impact of prognostic tests. They are not as common or reported as often as discrimination statistics. When using discrimination statistics, recognize that they change over time as more patients develop the outcome being predicted. Time-dependent measures of discrimination statistics have been developed (e.g., Harrell’s C-statistic); however, examples of reviews using these measures were not found Step 4: Extracting Statistics To Evaluate Test Performance (3 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

23
Reclassification tables For example: Patients are placed into prognostic groups based on their Framingham cardiovascular risk scores. Reclassification tables are then used to determine how adding a prognostic test reclassifies the patients into prognostic groups. Ideally, the classification of outcome probabilities into prognostic groups should be based on outcome probabilities that will lead to different courses of action. If not, then reclassifications might not have clinical utility; that is, they may not make a difference in patient care. Step 4: Extracting Statistics To Evaluate Test Performance (4 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

24
Sample Reclassification Table Based on Predicted Outcome Probabilities Step 4: Extracting Statistics To Evaluate Test Performance (5 of 12) Adding the new test reclassified 10% of the 1000 people originally in the lower risk group and 25% of the 400 people in the higher risk group Grouped mortality probabilities estimated by the first prognostic test Grouped mortality probabilities estimated by the first prognostic test + a new prognostic test 0 to 0.10> 0.10 Total 0 to 0.10 Patients in prognostic group Mortality predictions using 1 st test Mortality prediction using both tests Observed mortality 900 4.0% 3.8% 3.9% 100 (10%) 8.0% 11.0% 12.0% 1000 4.4% - 4.7% > 0.10 Patients in prognostic group Mortality predictions using 1 st test Mortality prediction using both tests Observed mortality 100 (25%) 15.0% 9.0% 10.0% 300 17.0% 19.0% 400 16.5% - 16.8% Total Patients in prognostic group Mortality prediction using both tests Observed mortality 1000 4.3% 4.5% 400 17.0% 17.2% 1400 - 8.2% Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

25
Reclassification tables provide information about: Observed outcome probabilities in each prognostic group Predicted probabilities Drawbacks Information is often limited to a single follow-up time. The precision of estimates may not be reported. Differences between estimated probabilities and observed outcomes for each prognostic group may be analyzed by using the chi-square goodness-of-fit test. The results will not help determine if differences in predicted and observed probabilities are better when a new prognostic test is added. Step 4: Extracting Statistics To Evaluate Test Performance (6 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

26
The net reclassification improvement statistic is a summary statistic of separate reclassification tables. One table is for those who did experience the outcome event within a particular time period. The other table is for those who did not experience the outcome event. In a group experiencing the outcome of interest (e.g., those who died), net improvement = the proportion of patients reclassified into a higher probability group minus the proportion reclassified into a lower probability group. In a 2 x 2 table, this is an estimated change in test sensitivity. Step 4: Extracting Statistics To Evaluate Test Performance (7 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

27
For those who do not experience an outcome (e.g., those who survived), the inverse is used: net improvement = the proportion reclassified into a lower-risk group minus the proportion reclassified into the higher-risk group. In a 2x2 table, this is the estimated change in test specificity. The net reclassification improvement (NRI) statistic is the sum of net improvement in classification in patients who did or did not experience the outcome. The integrated discrimination index (IDI) uses mean changes in individual predicted probabilities instead of net improvement in the calculations above. Step 4: Extracting Statistics To Evaluate Test Performance (8 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

28
The “continuous formulation” version of the net reclassification improvement (NRI) statistic calculates the: Probability of predicted event among those who have an increase in predicted probability after new test Probability of predicted event among those who have a decrease in predicted probability after new test Event probability in the overall sample With this version of the NRI statistic: The NRI statistic can be estimated by time-to-event analysis, but the three probabilities still represent only a single point of followup. The NRI statistic does not require clinically meaningful prognostic categories, since not all increases/decreases in probability prompt a change in patient management. The NRI statistic focuses instead on subjects with a higher or lower predicted outcome probability when a new test is used. Step 4: Extracting Statistics to Evaluate Test Performance (9 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

29
When pooling net reclassification improvement (NRI) or integrated discrimination index (IDI) estimates from different studies, the following should not differ substantially: Characteristics of prognostic groups Definition of outcome event Overall probability of event Followup time Step 4: Extracting Statistics To Evaluate Test Performance (10 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

30
Predictive values Treatment decisions based on outcome probabilities are often dichotomous. For example, “treat those at high risk” and “do not treat those at low risk” If patients would be treated because a test indicates they are at “high risk,” then the observed time-dependent percentages of patients developing the outcome (without treatment) are positive predictive values. That is, the proportion of those who have a “positive” prognostic test result who end up having the event. If patients would not be treated because they are at “low risk,” then: 1 – (observed outcome probabilities) = negative predictive values. That is, the proportion of those who have a “negative” prognostic test result who do not end up having the event. Step 4: Extracting Statistics To Evaluate Test Performance (11 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

31
Predictive values (continued) For a single point of followup, positive and negative predictive values can be compared via methods used for diagnostic tests. Ratios of positive and negative predictive values of two prognostic tests are often summarized along with confidence intervals. The regression model of Leisenring et al. (2000) may be used to determine how patient characteristics relate to relative predictive values. Time-to-event curves comparing predictive values of two prognostic tests are available if necessary. Step 4: Extracting Statistics To Evaluate Test Performance (12 of 12) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Leisenring W, Alonzo T, Pepe MS. Biometrics 2000 Jun;56(2):345-51. PMID: 10877288.

32
Randomized controlled trials (RCTs) designed to demonstrate net improvement in patient outcomes and cost-effectiveness are the most definitive level of evidence. Many prognostic test studies do not provide this level. Systematic reviews can provide estimates of outcome probabilities for decision models instead. Estimates from either RCTs or observational studies can be used, provided that prognostic groups are well characterized and similar. Meta-analysis can provide more precise estimates. Additionally, meta-analysis of estimates of outcome probabilities in a prognostic group extracted from several studies can provide insight into: Stability of the estimates Whether variation is related to prognostic group characteristics Step 5: Meta-analysis of Estimates of Outcome Probabilities (1 of 2) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

33
Methods have been developed to combine estimates of outcome probabilities from different studies. Dear’s method (1994) Uses a fixed-effects regression model Arend’s method (2008) Is similar to a DerSimonian-Laird random-effects model Is used when there is only one common followup time for all studies/prognostic groups Step 5: Meta-Analysis of Estimates of Outcome Probabilities (2 of 2) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Arends LR, Hunink MG, Stijnen T. Stat Med 2008 Sep 30;27(22):4381-96. PMID: 18465839. Dear KB. Biometrics 1994 Dec;50(4):989-1002. PMID: 7787011. DerSimonian R, Laird N. Control Clin Trials 1986 Sep;7(3):177-88. PMID: 3802833.

34
Methods used to conduct systematic reviews of prognostic tests are not well established. The intended use of the prognostic test should be specified; predicted probabilities need to be classified into clinically meaningful groups that are described in detail (including outcome probabilities). Many published reports focus on associations between prognostic indicators and patient outcomes (the first stage of development of prognostic tests); such studies have limited clinical value for a review. Criteria for evaluating the quality of studies of prognostic tests have not been firmly established. Reviewers can adapt criteria developed for diagnostic tests. Key Messages (1 of 2) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

35
Given differences between diagnostic tests (current state of disease) and prognostic tests (future state of disease), common evaluation statistics used with diagnostic tests are not as informative for prognostic tests. These include point estimates of sensitivity/specificity and area under the receiver operating characteristic (ROC) curves. The most pertinent summary statistics for prognostic tests are: The time-dependent observed outcome probabilities The closeness of the prognostic group’s predicted probabilities to observed outcomes How the use of a new test reclassifies patients into different groups and improves predictive accuracy and overall outcomes Methods for comparing and summarizing predictive performance of prognostic tests need further development and widespread use. Key Messages (2 of 2) Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

36
1.What is the relationship between predictive and prognostic tests? a.The terms are essentially synonymous. b.Predictive tests are a category of prognostic tests. c.Prognostic tests are a category of predictive tests. Practice Question 1 (1 of 2)

37
Explanation for Question 1: The correct answer is b. Predictive tests, which are a category of prognostic tests, can be used to predict beneficial or adverse responses to treatment. Practice Question 1 (2 of 2)

38
2.Which of the following four categories of prognostic tests is NOT most readily addressed by large cohort studies and secondary analyses of clinical trials? a.Clinical outcomes b.Clinical utility c.Prospective clinical validation d.Proof of concept e.None of the above Practice Question 2 (1 of 2)

39
Explanation for Question 2: The correct answer is a. Questions that pertain to clinical outcomes or cost-effectiveness are generally better answered by randomized controlled trials. The other three categories listed may often be addressed by less costly cohort studies or secondary analyses of clinical trials. Practice Question 2 (2 of 2)

40
3.There are relatively few reports where the test variable did not add significantly to a multivariable regression model. a.True b.False Practice Question 3 (1 of 2)

41
Explanation of Question 3: The correct answer is true. The relative paucity of published findings in which a test variable did not add significantly to a multivariable regression model suggests that there may be potential bias by failing to publish lack of association or predictive value. Practice Question 3 (2 of 2)

42
4.Prognostic indicators vary substantially in which of the following: a.Study design b.Subject inclusion criteria c.Methods of measuring key variables d.Adjustment for covariates e.Presentation of results f.All of the above Practice Question 4 (1 of 2)

43
Explanation of Question 4: The correct answer is f. Wide variability in one or more of these factors presents a challenge to selecting studies and assessing quality. Reviewer access to patient-level data would allow uniform analyses to overcome these difficulties. When lacking the ability to access patient-level data, suggestions for judging the quality of individual studies of prognostic tests can be followed. Practice Question 4 (2 of 2)

44
5.Examples of indices of discrimination do not include: a.95-percent confidence intervals b.Estimates of sensitivity and specificity c.Area under the receiver operating characteristic (ROC) curve d.All of the above Practice Question 5 (1 of 2)

45
Explanation of Question 5: The correct answer is a. 95-percent confidence intervals provide a measure of the precision of a given statistic, but are not, in themselves, a discrimination statistic. Practice Question 5 (2 of 2)

46
This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center. The module is based on Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 12.1-13. AHRQ Publication No. 12- EHC017. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Authors

47
Arends LR, Hunink MG, Stijnen T. Meta-analysis of summary survival curve data. Stat Med 2008 Sep 30;27(22):4381-96. PMID: 18465839. Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ 2001 Jul 28;323(7306):224-8. PMID: 11473921. Altman DG, Riley RD. Primer: an evidence-based approach to prognostic markers. Nat Clin Pract Oncol 2005 Sep;2(9):466-72. PMID: 16265015. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007 Feb 20;115(7):928-35. PMID: 17309939. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008 Jan;54(1):17-23. PMID:18024533. Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med 2009 Jun 2;150(11):795-802. PMID: 19487714. References (1 of 10)

48
Dear KB. Iterative generalized least squares for meta-analysis of survival data at multiple times. Biometrics 1994 Dec;50(4):989-1002. PMID: 7787011. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986 Sep;7(3):177-88. PMID: 3802833. Earle CC, Pham B, Wells GA. An assessment of methods to combine published survival curves. Med Decis Making 2000 Jan-Mar;20(1):104- 11. PMID: 10638543. Feng Z, Prentice R, Srivastava S. Research issues and strategies for genomic and proteomic biomarker discovery and validation: a statistical perspective. Pharmacogenomics 2004 Sep;5(6):709-19. PMID: 15335291. Fibrinogen Studies Collaboration. Measures to assess the prognostic ability of the stratified Cox proportional hazards model. Stat Med 2009 Feb 1;28(3):389-411. PMID: 18833567. References (2 of 10)

49
Fowkes FG, Murray GD, Butcher I, et al. Ankle brachial index combined with Framingham Risk Score to predict cardiovascular events and mortality: a meta-analysis. JAMA 2008 Jul 9;300(2):197- 208. PMID: 18612117. Groenveld HF, Januzzi JL, Damman K, et al. Anemia and mortality in heart failure patients a systematic review and meta-analysis. J Am Coll Cardiol 2008 Sep 2;52(10):818-27. PMID: 18755344. Grundy SM, Cleeman JI, Merz CN, et al; National Heart, Lung, and Blood Institute; American College of Cardiology Foundation; American Heart Association. Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III guidelines. Circulation 2004 Jul 13;110(2):227-39. PMID: 15249516. Hall PA, Going JJ. Predicting the future: a critical appraisal of cancer prognosis studies. Histopathology 1999 Dec;35(6):489-94. PMID: 10583572. References (3 of 10)

50
Hlatky MA, Greenland P, Arnett DK, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009 May 5;119(17):2408-16. PMID: 19364974. Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 2006 Mar 21;144(6):427-37. PMID: 16549855. Ingelsson E, Schaefer EJ, Contois JH, et al. Clinical utility of different lipid measures for prediction of coronary heart disease in men and women. JAMA 2007 Aug 15;298(7):776-85. PMID: 17699011. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc 2001 Jul-Aug;8(4):391-7. PMID: 11418546. References (4 of 10)

51
Janes H, Pepe MS. Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting. Am J Epidemiol 2008 Jul 1;168(1):89-97. PMID: 18477651. Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med 2008 Nov 18;149(10):751-60. PMID: 19017593. Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer 2007 Nov;43(17):2559-79. PMID: 17981458. Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst 2005 Jul 20;97(14):1043-55. PMID: 16030302. Leisenring W, Alonzo T, Pepe MS. Comparisons of predictive values of binary medical diagnostic tests for paired designs. Biometrics 2000 Jun;56(2):345-51. PMID: 10877288. References (5 of 10)

52
Mackillop WJ, Quirt CF. Measuring the accuracy of prognostic judgments in oncology. J Clin Epidemiol 1997 Jan;50(1):21-9. PMID: 9048687. McShane LM, Altman DG, Sauerbrei W, et al; Statistics Subcommittee of the NCI-EORTC Working Group on Cancer Diagnostics. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 2005 Aug 17;97(16):1180-4. PMID: 16106022. Meigs JB, Shrader P, Sullivan LM, et al. Genotype score in addition to common risk factors for prediction of type 2 diabetes. N Engl J Med 2008 Nov 20;359(21):2208-19. PMID: 19020323. Pencina MJ, D'Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med 2004 Jul 15;23(13):2109-23. PMID: 15211606. References (6 of 10)

53
Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008 Jan 30;27(2):157-72, discussion 207-12. PMID: 17569110. Pepe MS. Section 9.2: Incorporating the time dimension. In: The statistical evaluation of medical tests for classification and prediction. New York, NY: Oxford University Press USA; 2004. p. 259- 67. Pepe MS, Feng Z, Janes H, et al. Pivotal evaluation of the accuracy of a biomarker used for classification or prediction: standards for study design. J Natl Cancer Inst 2008 Oct 15;100(20):1432-8. PMID: 18840817. Pepe MS, Janes HE. Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer. J Natl Cancer Inst 2008 Jul 16;100(14):978-9. PMID: 18612128. References (7 of 10)

54
Pepe MS, Janes H, Longton G, et al. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 2004 May 1;159(9):882-90. PMID: 15105181. Pepe MS, Zheng Y, Jin Y, et al. Evaluating the ROC performance of markers for future events. Lifetime Data Anal 2008 Mar;14(1):86- 113. PMID: 18064569. Poses RM, Cebul RD, Collins M, et al. The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis. Ann Intern Med 1986 Oct;105(4):586-91. PMID: 3530079. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 12.1-13. AHRQ Publication No. 12-EHC017. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. References (8 of 10)

55
Ridker PM, Buring JE, Rifai N, et al. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA 2007 Feb 14;297(6):611-9. PMID: 17299196. Ridker PM, Paynter NP, Rifai N, et al. C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men. Circulation 2008 Nov 25;118(22):2243- 51, 4p following 2251. PMID: 18997194. Riesterer O, Milas L, Ang KK. Use of molecular biomarkers for predicting the response to radiotherapy with or without chemotherapy. J Clin Oncol 2007 Sep 10;25(26):4075-83. PMID: 17827456. References (9 of 10)

56
Riley RD, Abrams KR, Sutton AJ, et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer 2003 Apr 22;88(8):1191-8. PMID: 12698183. Sinuff T, Adhikari NK, Cook DJ, et al. Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med 2006 Mar;34(3):878-85. PMID: 16505667. Speight PM, Palmer S, Moles DR, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess 2006 Apr;10(14):1-144, iii-iv. PMID: 16707071. Ware JH. The limitations of risk factors as prognostic tools. N Engl J Med 2006 Dec 21;355(25):2615-7. PMID: 17182986. References (10 of 10)

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google