1. 1 Using Biostatistics to Evaluate Vaccines and Medical Tests Holly Janes Fred Hutchinson Cancer Research Center

2. 2 Two projects Evaluating a candidate HIV vaccine: The Step Study Statistical methods for evaluating medical tests: PSA screening test for prostate cancer

3. 3 The Step Study To evaluate a candidate HIV vaccine aimed at: Preventing HIV infection Delaying disease progression in those who become HIV infected 2004 to 2007 North America, South America, Caribbean, Australia 3000 HIV negative participants randomized to vaccine or placebo Tested approximately every 6 months for HIV infection

4. 4 Vaccine was ineffective at preventing infection Estimated annual rate of HIV acquisition:  3.1% (2.1 to 4.3%) for placebos  4.6% (3.4 to 6.1%) for vaccinees

5. 5 Evaluating vaccine effects on disease progression In the subset of participants who became HIV infected  As of October, 2007: 81 male infections  Not enough female infections to study Did the vaccine recipients who became infected have slower disease progression than the placebos who became infected?

6. 6 Measures of HIV disease progression Time to initiation of antiretroviral therapy (ART) HIV viral load: repeated measures over time CD4 cell count: repeated measures over time

7. Demographic Characteristics of HIV Infected Participants Vaccinees (n = 49)Placebos (n = 32) Country (%)US7181 Peru2213 Canada46 Haiti20 Ethnicity (%)White4956 Mestizo2213 Black1016 Hispanic163 Multi-race03 Other29 Age (mean (SD))31 (7)30 (7)

8. 8 No Vaccine Effect on Time to ART Initiation

9. 9 Vaccine effects on viral load and CD4 cell count Repeated measures over time on each subject Set values to “missing” after ART initiation Lots of missing data, due to:  ART initiation  Patient dropout  Missed visits Missing values are informative!!

10. 10 Sample Individual Viral Load Trajectories

11. 11 Population Trends in Viral Load

12. 12 Analysis of Viral Load and CD4 Cell Count Statistical methods:  Longitudinal data methods allow for repeated measures over time on the same subjects  Missing data methods incorporate information about missing data Imputation Inverse probability weighting Findings:  No evidence that vaccine and placebo groups have different levels or trends in viral load or CD4 cell count

13. 13 Evaluating Medical Tests

14. 14 Cancer Screening Tests Aimed at finding disease before it causes symptoms  Early-stage disease usually easier to treat Commonly used screening tests:  Mammography, for breast cancer  Pap test, for cervical cancer  PSA test, for prostate cancer

15. 15 Evaluating cancer screening tests How accurate is the test?  How often is cancer found? (true positive rate)  How often are healthy individuals told they have cancer? (false-positive rate) Screening tests must have very low false positive rates  The test is applied in the general population  The vast majority of subjects do not have cancer  A positive test result leads to invasive follow-up procedures (eg biopsy), unnecessary cost and stress  If false positive rate is 5%, 5,000 unnecessary biopsies for every 100,000 people screened

16. 16 PSA test for prostate cancer Commonly used screening test for prostate cancer in men over 50 Utility is hotly debated Test measures amount of prostate-specific antigen (PSA) in the blood “High” value suggests cancer  What is “high”? Positive test result prompts biopsy

17. 17 Quantifying test accuracy The true positive rate (TPR)  Proportion of subjects with cancer who test positive The false positive rate (FPR)  Proportion of healthy subjects who test positive How to define “test positive” for a quantitative test?

18. 18 How to define “test positive”?

19. 19 TPR = 0.98 FPR = 0.75

20. 20 TPR = 0.75 FPR = 0.25

21. 21 TPR = 0.25 FPR = 0.02

22. 22 The ROC Curve TPR vs. FPR as the “test-positive” threshold is varied

23. 23 Quantifying the accuracy of the PSA test The age of the man matters:  PSA increases with age, in the absence of cancer  Age is a strong risk factor for cancer If we ignore age, PSA performance will look artificially high:  Men with cancer are older on average  Older men tend to have higher PSA  “Confounding”

24. 24 An “Age-Adjusted” ROC Curve TPR vs. FPR among men of the same age This allows the “test-positive” threshold to depend on age

25. 25 The Age-Adjusted ROC Curve for PSA When FPR = 0.025, TPR = 0.17 (0.13 to 0.21) When FPR = 0.05, TPR = 0.27 (0.21 to 0.33)

26. 26 Summary Evaluating the efficacy of a candidate HIV vaccine  The Step trial  Vaccine effects on time to ART, viral load, CD4  Statistical methods that accommodate longitudinal data, missing data Statistical methods for evaluating medical tests  Eg PSA for prostate cancer screening  The tradeoff between TPR and FPR  Statistical method to adjust for covariates