FDA Evaluation of Prescription Genetic Tests

FDA Evaluation of Prescription Genetic Tests
Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests FDA Evaluation of Prescription Genetic Tests Reena Philip, Ph.D. OIVD/CDRH/FDA March 9, 2011

IVD Device Regulation Safety Effectiveness
Are there reasonable assurances, based on valid scientific evidence that the probable benefits to health from use of the device outweigh any probable risks? [21 CFR 860.7(d)(1)] Effectiveness Is there reasonable assurance based on valid scientific evidence that the use of the device in the target population will provide clinically significant results? [21 CFR 860.7(e)(1)]

Genetic Tests: Categories
Single analyte tests Genotyping Multiple analyte genetic tests Multiplex

FDA Cleared Prescription Genetic Tests: Examples
Single analyte genetic tests Factor II / Factor V / MTHFR (aid in diagnosis claim) Multiple analyte genetic tests CFTR (carrier testing, newborn screening, and confirmatory diagnosis claim) CYP2D6 genotyping (drug metabolism claim)

What Does FDA Review for Prescription Genetic Tests?
Intended use/indications for use Device description (platform, software) Pre-analytical Analytical validation Clinical validation Instrumentation, software validation (if applicable) Labeling (package insert)

Intended Use/Indications for Use
Intended use specifies What the test measures Why (the clinical indication for use) In what population it is intended to be used Setting in which the device is meant to be used

Example: Intended Use The –---- Cystic Fibrosis Kit is a device used to simultaneously detect and identify a panel of mutations and variants in the cystic fibrosis transmembrane conductance regulator (CFTR) gene in human blood specimens. The panel includes mutations and variants currently recommended by the American College of Medical Genetics and American College of Obstetricians and Gynecologists (ACMG/ACOG), plus some of the worlds most common and North American-prevalent mutations. The Cystic Fibrosis Kit is a qualitative genotyping test which provides information intended to be used for carrier testing in adults of reproductive age, as an aid in newborn screening, and in confirmatory diagnostic testing in newborns and children. The kit is not indicated for use in fetal diagnostic or pre-implantation testing. This kit is also not indicated for stand-alone diagnostic purposes. For Prescription use only.

Intended Use/Indications for Use
Device needs to have a clinical indication The types of validation studies that are needed depend on the claims that are made in the intended use

Pre-analytical Sample collection/transport/storage
Sample preparation/conditions Nucleic acid isolation Stability of the analyte in the patient specimen

Analytical Performance
Does my test measure the analyte I think it does? Correctly? Reliably?

Analytical Performance
Accuracy Precision (repeatability, reproducibility) Performance around the cut-off Limit of Detection Interference, cross-reactivity Sample type/matrix Potential for carryover, cross-hybridization Effect of excess/limiting sample

Analytical Performance: Accuracy
Closeness of the agreement between the result of a test and result of reference method.

Analytical Performance: Accuracy
Compare to ………. Comparison to a reference method e.g., bi-directional DNA sequencing Comparison to a clinical truth Real clinical samples Multiple clinical samples per allele Cover every claimed allele/result, genotypes, subtypes/classes

Analytical Performance: Accuracy (Continued)
An example of why we ask accuracy data of every individual allele the test claims to detect: Alleles % Agreement Total 98.40 Alleles 100.00 Allele 21 87.50 Allele 22 66.67 Allele 23 96.55

Analytical Performance: Precision
Studies should demonstrate that the intended users can get reliable results All sources of variability should be identified and assessed for its impact on assay precision Should use clinical samples where possible Adequate coverage of all genotypes/tumor types In limited cases (i.e., very rare alleles) may use contrived samples Samples should mimic the molecular composition and concentration of real clinical samples All analytical steps of the assay should be included

Test Performance: Evaluation
Analytical performance - does my test measure the analyte I think it does? Correctly? How reliably? Clinical performance - does my test result correlate with target condition of interest in a clinically significant way?

Clinical Performance: Genetic Tests
When there is sufficient information that establishes well-known association between genetic variants and medical condition – For each claimed allele: Peer-reviewed articles Genotype – Phenotype

Clinical Performance: Genetic Tests (Continued)
When there is not enough information that establishes well-known association between genetic variants and medical condition – May require clinical studies

Clinical Performance: Examples
CFTR mutation panel ACOG/ACMG recommendation Published literature Mutations in a novel gene to predict risk of developing cancer Most likely needs clinical studies

Clinical Effectiveness
Established Markers (Medical literature) New Markers (Should meet FDA standard for effectiveness) (based on valid scientific evidence that the use of the device in the target population will provide clinically significant results [21 CFR 860.7(e)(1)])

Labeling and Reporting Results
21 CFR part 809 (subpart B) Tests provide results, limited interpretation required

Most Frequent Issues… FDA Evaluation of Genetic Tests
Lack of clinical samples covering all genotypes Lack of literature to support validity One or two references may not be sufficient Genotype and Phenotype not indicated Pre-analytical issues Lack of specimens from start to end (e.g., whole blood  assay result) Sample matrix issues

Summary: What Does FDA Review for Prescription Genetic Tests?
Safety and effectiveness generally determined based on: Satisfactory analytical performance Clinical performance in the context of use Labeling that is compliant with the labeling regulations for IVDs (21 CFR 809 Subpart B) And other factors such as ability to repeatedly manufacture the device to specifications

Relevant Guidance Documents
Guidance on Pharmacogenetic Tests and Genetic Tests for Heritable Markers Class II Special Controls Guidance Document: CFTR Gene Mutation Detection Systems Class II Special Controls Guidance Document: Drug Metabolizing Enzyme Genotyping System - Guidance for Industry and FDA Staff Class II Special Controls Guidance Document: Instrumentation for Clinical Multiplex Test Systems - Guidance for Industry and FDA Staff

Performance of FDA Approved/Cleared Genetic Tests are Publicly Available
Decision summaries of 510(k)s Summary of Safety & Effectiveness of PMAs

Thank You!

Carol C. Benson OIVD/CDRH/FDA March 9, 2011
Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests Principles of FDA Regulation for In Vitro Diagnostic Tests for Home Use Carol C. Benson OIVD/CDRH/FDA March 9, 2011

Overview Examples of home use test Benefits Risks Interpretation of results Device performance Labeling Human factors

Examples of Home Use Tests FDA Regulates
Collect sample Perform test Interpret results Glucose Yes Pregnancy Drugs of abuse Breath alcohol Ovulation and Menopause

Examples of Home Use Tests FDA Regulates (Continued)
Collect sample Perform test Interpret results Collection Hep. C kit Yes No Collection HbA1c kit Collection HIV-1 kit And others …but “No” genetic tests

What are the Benefits for Home Use Tests?
Condition/disease that needs to be monitored at home? Diabetes – home glucose meters monitor the management of diabetes home user under care of a physician Not for diagnosis – no performance

Condition/disease that can be identified to allow for early detection at home? Pregnancy – urine hCG tests users retest, go to HCP

Condition/disease that can be screened for at home? Drug detection – Home DOA urine test Not a definitive test Mitigation - send sample for confirmation testing

What are the Risks of Home Use Tests?
Is the device robust? Simple to use Works correctly every time Not affected by environmental conditions or different operators Can a home user read instructions and Collect the sample correctly Perform the test Get accurate results and interpret results

Interpretation of the Results to Ensure Safe and Effective Use
Does the home user know what to do with the results? Test again on another day Collect another sample and retest Contact HCP – seek treatment Not seek treatment Not suspect the test may be wrong

What are the Risks of False or Inaccurate Results at Home?
Failure to seek treatment Delay in seeking treatment Improper self management/treatment No follow up with health care provider Unnecessary worry False sense of security

Do the benefits outweigh the risks? If yes, then…

Evaluate Performance of the Test in the Hands of the Intended User: Home User Study
Compare home user results to laboratory method How well the test should work at home depends upon benefit and mitigation of risks Likelihood of incorrect results

Labeling Does the labeling provide adequate information so home user can perform the test and interpret the results for safe and effective use?

How Does FDA Review Labeling for Home Use Tests?
Written at 8th grade reading level Simple instructions Pictures and diagrams on how to get sample and perform test Clear instructions on how to interpret the results (what to do with the results – call HCP – retest) Users know when device did not work User know what to do if device does not work Telephone number to call for assistance

How Do Human Factors Play a Role in Home Tests?
People have different abilities to follow directions Home users are not trained users so no “good laboratory practice” standard for them Fail to get adequate or appropriate sample Can perform test incorrectly Can interpret results incorrectly

Summary – FDA Principles for Regulation of Home Use Tests
FDA regulates home use tests Benefits vs. risks Mitigation of risks Interpretation of results by home user Performance of the device by home user Labeling Human factors

Website for Database Search for Home Use Tests:

Thank You!

Risk Assessment Tests Marina Kondratovich, Ph.D. OIVD/CDRH/FDA
Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests Risk Assessment Tests Marina Kondratovich, Ph.D. OIVD/CDRH/FDA March 9, 2011

Overview Introduction Basic concepts: risks, relative risks, likelihood ratios, odds ratios 3. Description of a typical DTC risk assessment test 4. Clinical validation (discrimination and calibration)

1. Introduction Susceptibility/Pre-dispositional tests
(Risk Assessment tests): tests that estimate the lifetime risk (relative or absolute) that an individual will develop a condition. Examples: test for Alzheimer’s disease, test for prostate cancer, test for type 2 diabetes Possible Intended Use Claim: “…to estimate the likelihood that an individual will develop <target condition> during the lifetime…”

Typical DTC Risk Assessment Test
Individual Covariates Markers Race Gender SNP1 SNP2 SNP3 SNP4 Pre-test risk (baseline risk) Race and gender specific Relative risk Absolute risk Risk category (as “Low”, “Average”, “High”) 20% Race=European Gender=Male 1.5 30% (1.5 x 20%) “High”

2. Basic Concepts Absolute risks, relative risks
Likelihood ratios, odds ratios Test with more than two outcomes

Clinical Performance of the Test
Consider Test with Two Outcomes (Pos., Neg.) Let us have 500 subjects who are representative subjects from intended use population (target population). Each subject has results of the Test (Pos., Neg.) and a Clinical Reference Standard (D+, D-). D+ D- Total T + 70 160 230 - 30 240 270 100 400 500 Prevalence of 20% (100/500) reflects prevalence in the IU population. Clinical Performance of the Test Sensitivity 70.0% (70/100) Specificity 60.0% (240/400)

Risks (Absolute Risks) D+ D- Total T + 70 160 230 - 30 240 270 100 400 500 Clinical Performance of the Test R1=Risk of D+ for T+ (PPV)* 30.4% (70/230) R0=Risk of D+ for T- (1-NPV)* 11.1% (30/270) π= Pre-test risk (baseline risk, prevalence, average risk (averaged over other risk factors)) 20.0% (100/500) *Post-test risk for T Pos, post-test risk for T Neg.

Relative Risks Clinical Performance of the Test R1=Risk of D+ for T+ (PPV) 30.4% (70/230) R0=Risk of D+ for T- (1-NPV) 11.1% (30/270) π= Pre-test risk 20.0% (100/500) R1/π = 1.52 : For a subject with T+, the risk increases by 1.52 times with regard to pre-test risk (=30.4/20.0); R0/π = 0.56 : For a subject with T-, the risk increases by 0.56 times (decreased by 1.80 (1/0.56) times) with regard to pre-test risk (=11.1/20.0); R1/R0 = 2.74 : For a subject with T+, the risk increases by 2.74 times with regard to the subjects with T- (=30.4/11.1)

Absolute risks and relative risk depend on the sensitivity, specificity and also on the pre-test risk. Se Sp D+ D- Total T + 70 160 230 - 30 240 270 100 400 500 R1 R0

Likelihood Ratios, Odds Ratios
Likelihood Ratios (LR) are another way to describe the performance of a test. “Odds” are the ratio of the probability of one outcome to the probability of its opposite outcome. Example: Single fair coin with outcomes {Head, Tail}: odds =1 because Pr (Head)=0.5 and Pr (Tail)=1-0.5=0.5 => odds=1 (0.5/0.5=1).

Likelihood Ratios, Odds Ratios (Continued)
Subject from the IU population with pre-test risk π, two outcomes (D+, D-); Pr(D+) = π and Pr(D-)=1-π. After the test is performed (with knowledge of the test results): Is there a relationship between post-test odds and pre-test odds?

Post-test odds = Likelihood Ratio x Pre-test odds

Post-test odds = Likelihood Ratio x Pre-test odds Likelihood Ratios do not depend on the pre-test risk. Odds Ratio does not depend on the pre-test risk.

Consider Test with More than Two Outcomes.
In the hypothetical example, the test examines four markers: each marker has three possible results (aa, Aa, AA) Then the test has 81 possible results (=3 x 3 x 3 x 3). For the sake of simplicity, consider test with three outcomes: as (Result1, Result2 and Result3). Let us have 500 subjects who are representative subjects from the intended use population (target population). Each subject has results of the Test and a Clinical Reference Standard (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

Test with Three Outcomes: as (Result1), (Result2) and (Result3).
Total Risk T Result3 24 72 96 25.0% Result2 56 216 272 20.6% Result1 20 112 132 15.2% 100 400 500 20.0% Pre-test odds: /( ) = 0.250 Post-test odds(Result3): 0.250/( ) = 0.333 Post-test odds(Result2): 0.206/( ) = 0.259 Post-test odds(Result1): 0.152/( ) = 0.179 Is there a relationship between post-test odds and pre-test odds?

Likelihood Ratios, Odds Ratios
Post-test odds (Resulti) = LR(Resulti) x Pre-test odds D+ D- Risk LR T Result3 24 72 96 25.0% 24.0% 18.0% 1.33 Result2 56 216 272 20.6% 56.0% 54.0% 1.04 Result1 20 112 132 15.2% 20.0% 33.0% 0.61 100 400 500 100% LR is a way of quantifying how much given test result changes the pre-test (baseline) risk of the target condition.

LR OR T Result3 24 72 96 24.0% 18.0% 1.33 2.18 Result2 56 216 272 56.0% 54.0% 1.04 1.70 Result1 20 112 132 20.0% 33.0% 0.61 1.00 100 400 500 100% ORs are usually considered with regard to the Result with the lowest risk (normalized to the lowest risk): ORi=LRi/LR1. LRs are related to the pre-test risk (average risk).

Summary Risks and relative risks
Risks and relative risks depend on corresponding likelihood ratios and pre-test (baseline) risk. Because risks (and relative risks) depend on pre-test risk, in some study designs, they cannot be estimated (as in case-control studies). Risks and relative risks measure probabilities of events in a way that is interpretable and consistent with how the people think.

Summary (Continued) Likelihood Ratios (LR) and Odds Ratio (OR)
LRs and ORs do not depend on the pre-test risk. Because they do not depend on the pre-test risk, LRs and ORs can be calculated even in the case-control studies. It is easy to adjust an ORs for other variables (logistic regression) LRs and ORs are more difficult for interpretation because they are related to pre-test and post-test odds, which are not intuitive.

3. Description of Typical DTC Risk
Assessment Test For the sake of simplicity, consider a test which measures four markers: each marker has three possible results: (aa, Aa, AA). Then the test has 81 possible results (=3 x 3 x 3 x 3). Consider that an individual has test result (Ai, Bj, Ck, Dl). Basic idea of calculation of the risk for this individual is Post-test Odds = Likelihood Ratio x Pre-test Odds

Note 1 For a given race/ethnicity, information from case-control studies in published literature is used (independent confirmations of GWAS) Even for the same set of published papers related to the target condition (disease), different markers (SNPs) can be included in the test (different approaches for selection of SNPs are used). 2) Even for the same set of published papers and for the same SNP included in the test, different OR estimates can be used in the calculation of the LR for the test result (different approaches are used). For example, estimates of OR are: 1.2 in paper 1; 1.4 in paper 2; 1.1 in paper 3 (study with largest sample size? meta-analysis? …) 3) Information about OR in the case-control studies is used for calculation of LR (OR with regard to the average risk). Different assumptions are considered. For example, an assumption that “Controls” are not subjects without disease but a random sample from population.

Note 2 Consider the test for the target condition with 4 markers (SNPs) for a given race/ethnicity; ORs for individual markers are obtained from literature. SNP1 SNP2 SNP3 SNP4 LR Result3 1.27 1.55 Result2 1.05 Result1 0.77 Multiplicative Model: an assumption that all four SNPs are independent (no interactions). This assumption may be not correct.

Note 3 Pre-Test Risk, π Absolute risk Ri,j,k,l is calculated based on corresponding LR and the pre-test risk (average risk). Pre-test risk is provided based on the publicly available information about race- and gender- specific lifetime risks (for example, Surveillance Epidemiology and End Results (SEER) Cancer Statistics Review). Pre-test risk (average risk) is gender- and race- specific (very limited number of factors). The average risks present risks averaged over other important risk factors (such as family history, smoking, environmental factors and so on). => An individual pre-test risk taking into account other important factors can be very different from the average risk.

Note 3 Pre-Test Risk, π (Continued)
Hypothetical Example Race and gender specific risk averaged over other important factors Subjects of the same gender and race and with low risk factors Subjects of the same gender and race and with high risk factors 5% % % Pre-Test Risk 5% 20% 35% LR 1.5 Post-Test Risk* 7.3% 27.3% 44.7% Increase in Risk 2.3% 9.7% Relative risk 1.46 1.36 1.28 *

Note 3 Pre-Test Risk, π (Continued)
Absolute values of the post-test risks are considerably affected by the pre-test risk. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), post-test risks were from 7.3% to 44.7%. Relative risks are also affected by the pre-test risks but to much lesser degree. (range 5%-35%), relative risks were from 1.28 to 1.46. If the pre-test risk is very low, then relative risk ≈ LR In hypothetical example of LR=1.5 and pre-test average risk =3% (range 1%-5%), relative risks were from 1.46 to 1.49. If there is an assumption that “Controls” in the case-control study are not subjects without disease but a random sample from the intended use population (this assumption may be not correct), then relative risk = LR.

Note 4 Risk Categories “Low”, “Average”, “High”
Various approaches (based on either relative risks, or likelihood ratios, or absolute risks) and different cutoffs can be used for defining these three categories. Hypothetical example: Pre-test risk 20% LR Low Average High RR Low Average High Risk Low Average High 16.67% % % The same person can be classified into different risk categories.

4. Clinical Validation Risk assessment tests report an absolute risk for an individual. Note that, with regard to the study designs, the absolute risks cannot be evaluated in the case-control studies. For the absolute risks, a clinical validation includes two aspects: discrimination and calibration.

4. Clinical Validation (Continued)
Discrimination Under discrimination, we understand the ability of the test to discriminate between subjects who have target condition and subjects who do not have one. We would like that the subjects with target condition have higher values of the absolute risk compare to the absolute risks of the subjects without target condition. For assessing discrimination, a receiver operating characteristic (ROC) analysis is used (ROC curve and area under ROC curve along with 95% confidence interval). As a general rule, the larger is the area under ROC curve, the better is the discriminatory capability of the test.

Test Variable: absolute risk values for Diseased group (Y);
ROC Analysis Test Variable: absolute risk values for Diseased group (Y); absolute risk values for Non-Diseased group (X) X Y AUC = P{Y>X}, probability that an absolute risk of a randomly selected Diseased subject is larger than an absolute risk of a randomly selected Non-Diseased subject. Consider, for example, a wrong pre-test risk and correct pre-test risk (all other calculations are the same): ROC curves are the same.

4. Clinical Validation (Continued)
Calibration Absolute risks should be well-calibrated. If one has 100 subjects and the test is telling that that their risk is 12%, then one can anticipate that among these 100 subjects, approximately 12 subjects will have the target condition in reality. Calibration evaluates the degree of correspondence between the risk of the target condition provided by the test (Expected according to the absolute risk by the test) and the actual risk of the target condition (Observed). Calibration of the test which provides absolute risks cannot be evaluated in the case-control studies.

Summary 2. Absolute risks depend considerably on the pre-test risks.
1. Absolute and relative risks provided by the DTC risk assessment tests are calculated based on different approaches what can lead to inconsistencies in the results. 2. Absolute risks depend considerably on the pre-test risks. 3. Absolute risks in the DTC risk assessment tests do not include important risk factors other than markers measured by the DTC risk assessment tests and some limited number of factors (as race, gender, sometimes age).

Thank You!

FDA Evaluation of Prescription Genetic Tests

Similar presentations

Presentation on theme: "FDA Evaluation of Prescription Genetic Tests"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

FDA Evaluation of Prescription Genetic Tests

Similar presentations

Presentation on theme: "FDA Evaluation of Prescription Genetic Tests"— Presentation transcript:

Similar presentations

About project

Feedback