1. 1 Risk Assessment Tests Marina Kondratovich, Ph.D. OIVD/CDRH/FDA March 9, 2011 Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests

2. 2 1. Introduction 2. Basic concepts: risks, relative risks, likelihood ratios, odds ratios 3. Description of a typical DTC risk assessment test 4. Clinical validation (discrimination and calibration) Overview

3. 3 1. Introduction  Susceptibility/Pre-dispositional tests (DTC Risk Assessment tests): tests that estimate the risk (relative or absolute) that an individual will develop a condition during the lifetime Examples: test for Alzheimer’s disease, test for prostate cancer, test for type 2 diabetes  Possible Intended Use Claim: “…to estimate the likelihood that an individual will develop during the lifetime…”

4. 4 Typical DTC Risk Assessment Test Individual CovariatesMarkers RaceGenderSNP 1 SNP 2 SNP 3 SNP 4 Pre-test risk (average risk) Race and gender specific Relative riskAbsolute riskRisk category (as “Low”, “Average”, “High”) 20% Race=European Gender=Male 1.5 30% (1.5 x 20%) “High”

5. 5 2. Basic Concepts  Absolute risks, relative risks  Likelihood ratios, odds ratios  Test with more than two outcomes

6. 6 Consider Test with Two Outcomes (Pos., Neg.) D+D-Total T+70160230 -30240270 Total100400500 Clinical Performance of the Test Sensitivity70.0% (70/100) Specificity60.0% (240/400) Let us have 500 subjects who are representative subjects from intended use population (target population). Each subject has results of the Test (Pos., Neg.) and a Clinical Reference Standard (“Gold Standard”) (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

7. 7 Risks (Absolute Risks) Clinical Performance of the Test R 1 =Risk of D+ for T+ (PPV)*30.4% (70/230) R 0 =Risk of D+ for T- (1-NPV)*11.1% (30/270) π = Pre-test risk of D+ (baseline risk, prevalence, average risk (averaged over risk factors)) 20.0% (100/500) D+D-Total T+70160230 -30240270 Total100400500 * Post-test risk for T Pos, post-test risk for T Neg.

8. 8 Relative Risks  R 1 /π = 1.52 : For a subject with T+, the risk increases by 1.52 times with regard to pre-test risk (=30.4/20.0);  R 0 /π = 0.56 : For a subject with T-, the risk increases by 0.56 times (decreased by 1.80 (1/0.56) times) with regard to pre-test risk (=11.1/20.0);  R 1 /R 0 = 2.74 : For a subject with T+, the risk increases by 2.74 times with regard to the subjects with T- (=30.4/11.1) Clinical Performance of the Test R 1 =Risk of D+ for T+ (PPV)30.4% (70/230) R 0 =Risk of D+ for T- (1-NPV)11.1% (30/270) π = Pre-test risk of D+20.0% (100/500)

9. 9 Absolute risks and relative risk depend on the sensitivity, specificity and also on the pre-test risk. D+D- Total T+70160230 -30240270 Total 100400500 Se Sp R1R1 R0R0 D+D- Total T+140120260 -60180240 Total 200300500 Se = 70.0% (70/100) Sp = 60.0% (240/400) Pre-test risk =20% (100/500) R 1 =30.4% (70/230) R 0 =11.1% (30/270) R 1 /π=1.52; R 0 /π=0.56; R 1 /R 0 = 2.74 Se = 70.0% (140/200) Sp = 60.0% (180/300) Pre-test risk =40% (200/500) R 1 =53.8% (140/260) R 0 =25.0% (60/240) R 1 /π=1.35; R 0 /π=0.63; R 1 /R 0 = 2.15

10. 10 “Odds” are the ratio of the probability of one outcome to the probability of its opposite outcome. Example: Single fair coin with outcomes {Head, Tail}: odds =1 because Pr (Head)=0.5 and Pr (Tail)=1-0.5=0.5 => odds=1 (0.5/0.5=1). Likelihood Ratios (LR) are another way to describe the performance of a test. Likelihood Ratios, Odds Ratios

11. 11 Subject from the IU population with pre-test risk π, two outcomes (D+, D-); Pr(D+) = π. Likelihood Ratios, Odds Ratios (Continued) After the test is performed (with knowledge of the test results): Is there a relationship between post-test odds and pre-test odds?

12. 12 Post-test odds = Likelihood Ratio x Pre-test odds Likelihood Ratios, Odds Ratios (Continued)

13. 13 Post-test odds = Likelihood Ratio x Pre-test odds Likelihood Ratios, Odds Ratios (Continued) Likelihood Ratios do not depend on the pre-test risk. Odds Ratio does not depend on the pre-test risk.

14. 14 Consider Test with More than Two Outcomes. Let us have 500 subjects who are representative subjects from the intended use population (target population). Each subject has results of the Test and a Clinical Reference Standard (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population. In the hypothetical example, the test measures four markers: each marker has three possible results (aa, Aa, AA) Then the test has 81 possible results (=3 x 3 x 3 x 3). For the sake of simplicity, consider test with three outcomes: as (Result 1, Result 2 and Result 3 ).

15. 15 Test with Three Outcomes: as (Result 1 ), (Result 2 ) and (Result 3 ). D+D-TotalRisk T Result 3 24729625.0% Result 2 5621627220.6% Result 1 2011213215.2% 10040050020.0%  Pre-test odds: 0.200/(1-0.200) = 0.250  Post-test odds(Result 3 ): 0.250/(1-0.250) = 0.333 Post-test odds(Result 2 ): 0.206/(1-0.206) = 0.259 Post-test odds(Result 1 ): 0.152/(1-0.152) = 0.179 Is there a relationship between post-test odds and pre-test odds?

16. 16 Post-test odds (Result i ) = LR(Result i ) x Pre-test odds Likelihood Ratios, Odds Ratios LR is a way of quantifying how much given test result changes the pre-test (baseline) risk of the target condition. D+D-TotalRiskD+D-LR T Result 3 24729625.0%24.0%18.0%1.33 Result 2 5621627220.6%56.0%54.0%1.04 Result 1 2011213215.2%20.0%33.0%0.61 10040050020.0%100%

17. 17 Likelihood Ratios, Odds Ratios (Continued)  ORs are usually considered with regard to the Result with the lowest risk (normalized to the lowest risk): OR i =LR i /LR 1.  LRs are related to the pre-test risk (average risk). D+D-D+D-LROR T Result 3 24729624.0%18.0%1.332.18 Result 2 5621627256.0%54.0%1.041.70 Result 1 2011213220.0%33.0%0.611.00 100400500100%

18. 18 Summary Risks and relative risks  Risks and relative risks depend on corresponding likelihood ratios and pre-test (baseline) risk.  Because risks (and relative risks) depend on pre-test risk, in some study designs, they cannot be estimated (as in case-control studies).  Risks and relative risks measure probabilities of events in a way that is interpretable and consistent with how people think.

19. 19 Likelihood Ratios (LR) and Odds Ratio (OR)  LRs and ORs do not depend on the pre-test risk.  Because they do not depend on the pre-test risk, LRs and ORs can be calculated even in the case-control studies.  It is easy to adjust an OR for other variables (logistic regression)  LRs and ORs are more difficult for interpretation because they are related to pre-test and post-test odds, which are not intuitive. Summary (Continued)

20. 20 3. Description of Typical DTC Risk Assessment Test For the sake of simplicity, consider a test which measures four markers: each marker has three possible results: (aa, Aa, AA). Then the test has 81 possible results (=3 x 3 x 3 x 3). Post-test Odds = Likelihood Ratio x Pre-test Odds Consider that an individual has test result (A i, B j, C k, D l ). Basic idea of calculation of the risk for this individual is

21. 21 Note 1 For a given race/ethnicity, information from case-control studies in published literature is used (independent confirmations of GWAS) 1)Even for the same set of published papers related to the target condition (disease), different markers (SNPs) can be included in the test (different approaches for selection of SNPs are used). 2) Even for the same set of published papers and for the same SNP included in the test, different OR estimates can be used in the calculation of the LR for the test result (different approaches are used). For example, estimates of OR are: 1.2 in paper 1; 1.4 in paper 2; 1.1 in paper 3 (study with largest sample size? meta-analysis? …) 3) Information about OR in the case-control studies is used for calculation of LR. Different assumptions are considered. For example, an assumption that “Controls” are not subjects without disease but a random sample from population.

22. 22 Note 2 Consider the test for the target condition with 4 markers (SNPs); ORs for individual markers are obtained from literature. SNP 1 SNP 2 SNP 3 SNP 4 LR Result 3 1.27 BB 1.55 DD Result 2 1.05 Aa Result 1 0.77 cc Multiplicative Model: an assumption that all four SNPs are independent (no interactions). This assumption may be not correct.

23. 23 Note 3 Pre-Test Risk, π Absolute risk R i,j,k,l is calculated based on corresponding LR and the pre-test risk (average risk).  Pre-test risk is provided based on the publicly available information about race- and gender- specific lifetime risks (for example, Surveillance Epidemiology and End Results (SEER) Cancer Statistics Review).  Pre-test risk (average risk) is gender- and race- specific (very limited number of factors). The average risks present risks averaged over other important risk factors (such as family history, smoking, environmental factors and so on). => An individual pre-test risk taking into account other important factors can be very different from the average risk.

24. 24 Note 3 Pre-Test Risk, π (Continued) Race and gender specific risk averaged over other important factors 5% 20% 35% Pre-Test Risk5%20%35% LR1.5 Post-Test Risk*7.3%27.3%44.7% Increase in Risk2.3%7.3%9.7% Relative risk1.461.361.28 Subjects of the same gender and race and with low risk factors Subjects of the same gender and race and with high risk factors * Hypothetical Example

25. 25  Absolute values of the post-test risks are considerably affected by the pre-test risk. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), post-test risks were from 7.3% to 44.7%.  Relative risks are also affected by the pre-test risks but to much lesser degree. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), relative risks were from 1.28 to 1.46.  If the pre-test risk is very low, then relative risk ≈ LR In hypothetical example of LR=1.5 and pre-test average risk =3% (range 1%-5%), relative risks were from 1.46 to 1.49. If there is an assumption that “Controls” in the case-control study are not subjects without disease but a random sample from the intended use population (this assumption may be not correct), then relative risk = LR. Note 3 Pre-Test Risk, π (Continued)

26. 26 Note 4 Risk Categories “Low”, “Average”, “High” Various approaches (based on either relative risks, or likelihood ratios, or absolute risks) and different cutoffs can be used for defining these three categories. Pre-test risk 20% LR Low Average High 0.80 1.00 1.20 RR Low Average High Risk Low Average High 0.833 1.00 1.154 16.67% 20.0% 23.08% Hypothetical example: The same person can be classified into different risk categories.

27. 27  DTC risk assessment tests report an absolute risk for an individual.  Note that, with regard to the study designs, the absolute risks cannot be evaluated in the case-control studies.  For the absolute risks, a clinical validation includes two aspects: discrimination and calibration. 4. Clinical Validation

28. 28 4. Clinical Validation (Continued) Discrimination  Under discrimination, we understand the ability of the test to discriminate between subjects who have target condition and subjects who do not. We would like that the subjects with target condition have higher values of the absolute risk compare to the absolute risks of the subjects without target condition.  For assessing discrimination, a receiver operating characteristic (ROC) analysis is used (ROC curve and area under ROC curve along with 95% confidence interval). As a general rule, the larger is the area under ROC curve, the better is the discriminatory capability of the test.

29. 29 ROC Analysis Test Variable: absolute risk values for Diseased group (Y); absolute risk values for Non-Diseased group (X) AUC = P{Y>X}, probability that an absolute risk of a randomly selected Diseased subject is larger than an absolute risk of a randomly selected Non-Diseased subject. X Y Consider, for example, a wrong pre-test risk and correct pre-test risk (all other calculations are the same): ROC curves are the same.

30. 30 Calibration  Absolute risks should be well-calibrated. If one has 100 subjects and the test is telling that that their risk is 12%, then one can anticipate that among these 100 subjects, approximately 12 subjects will have the target condition in reality.  Calibration evaluates the degree of correspondence between the risk of the target condition provided by the test (Expected according to the absolute risk by the test) and the actual risk of the target condition (Observed). Calibration of the test which provides absolute risks cannot be evaluated in the case-control studies. 4. Clinical Validation (Continued)

31. 31 Summary 1. Absolute and relative risks provided by the DTC risk assessment tests are calculated based on different approaches that can lead to inconsistencies in the results. 2. Absolute risks depend considerably on the pre-test risks. 3. Absolute risks in the DTC risk assessment tests do not include important risk factors other than markers measured by the DTC risk assessment tests and some limited number of factors (as race, gender, sometimes age).

32. 32 Thank You! Marina.Kondratovich@fda.hhs.gov