1. Observational Study Designs and Studies of Medical Tests
Michael A. Kohn, MD, MPP
25 August 2009

2. Outline Single Sentence Study Description
REVIEW Observational study designs
Cohort, Double Cohort
Case-Control
Cross-sectional
Studies of Medical Tests
Diagnostic Test Accuracy
Prognostic Test Accuracy
Examples of observational designs (“Name that Design”)

3. Single-Sentence Study Description (Unless Studying a Medical Test)
“The [cute acronym] study is a [DESIGN] study of the association* between [predictor] and [outcome] in [study population].”
“The SCOTCH Study is a cohort study of the association* between HPV infection and development of cutaneous squamous cell carcinoma in renal transplant recipients.”
* Interested in causal association.

4. Single-Sentence Description If Studying a Test
“The [cute acronym] study is a [DESIGN] study of [test] as a [diagnostic/prognostic] test for [disease/outcome] in [study population].”
“The 3D-ERUS Study is a cross-sectional study of the accuracy of endorectal ultrasound in re-staging rectal cancer relative to the gold standard of surgical pathology after neoadjuvant chemoradiation in patients with locally invasive rectal cancer.”

5. Single-Sentence Study Description
Exercise for section today: Present your study with a sentence like this.

6. Study Design Not just a matter of semantics
Weaknesses and strengths associated with each study design
Different measures of disease association
Worth getting right or at least thinking about

7. Study Designs Experimental -- Randomized controlled trial
Observational (today’s topic)
-- Cohort
-- Double Cohort (exposed-unexposed)
-- Case-control
-- Cross-sectional

8. Predictor Type and Experimental vs. Observational Design
Predictor = treatment or screening program
-- experiment (randomized controlled trial)*
-- observational study of a treatment or program
Predictor = exposure or risk factor
-- observational study of an exposure or risk factor
Predictor = test result
-- observational study of a test
*Not all treatments or screening programs require RCTs to prove effectiveness.

9. OBSERVATIONAL STUDIES
Only option if predictor is a potentially harmful exposure, risk factor, or test.
Even if the predictor is an intervention, RCT may not be feasible
Confounding is an issue*
More intellectually interesting than RCTs?
* Except in studies of tests, then the issue isn’t confounding, but how much the test adds to information that is already available.

10. Note on Figures
Following schematics of observational study designs assume:
Predictor = Risk Factor
Outcome = Disease

11. Cohort Study

12. Cohort Studies
1)Determine predictor status on a sample from a single population (defined by something other than the predictor).
2)Exclude any potential subjects who already have the outcome.
3)Follow sample over time and attempt to determine outcome on all subjects.

13. Cohort Studies Can identify individuals lost to follow up
Can estimate overall incidence of outcome in the population (e.g., cases/person-year)
Measure of disease association is the relative risk (RR) or relative hazard (RH)

14. Double Cohort Study

15. Double Cohort (Exposed-Unexposed) Studies
Sample study subjects based on predictor status.
Exclude potential subjects in whom outcome has already occurred.
3) Attempt to determine outcome in all subjects over time.

16. Double Cohort (Exposed-Unexposed) Studies
Can identify individuals lost to follow up
Cannot estimate overall incidence of outcome in the population (e.g., cases/person-year)
Measure of disease association is the relative risk (RR) or relative hazard (RH)

17. Cohort Studies: Sampling Frame vs. Time Frame
Time Frame: All cohort studies are longitudinal (follow patients over time).
Sampling Frame:
Double cohort study -- samples on predictor status
Cohort study -- starts with a cross-sectional sample

18. Cohort Studies: Prospective vs. Retrospective
Prospective – Predictor status collected as part of this study
Retrospective – Predictor status collected by someone else in the past (another study, medical records, etc.)
Don’t worry too much about retrospective vs. prospective!

19. Case-Control Study

20. Case-Control Study
1) Separately sample subjects with the outcome (cases) and without the outcome (controls)
2) Attempt to determine predictor status on all subjects in both outcome groups

21. Case-Control Study
Cannot identify individuals lost to follow up (no such thing as “lost to follow up”, since by definition outcome status is known)
Cannot calculate prevalence (or incidence) of outcome
Measure of disease association is the Odds Ratio (OR)
Trying to replicate a nested case control study in which the cases and controls come from the same cohort.

22. Nested Case-Control Study

23. Cross-Sectional Study

24. Cross-Sectional Study
Attempt to determine predictor and outcome status on all patients in a single population (defined by something other than predictor and outcome).

25. Cross-Sectional Study
Cannot identify individuals lost to follow up (no such thing as “lost to follow up”)
Can calculate prevalence but not incidence
Measure of disease association is the Relative Prevalence (RP).
Time frame is the same as for a case-control study; both discussed in DCR3, Chapter 8

26. Cohort Studies Start with a Cross-Sectional Study
Eliminate subjects who already have disease

27. Causal Association Between Predictor and Outcome
Most observational studies: Does predictor cause outcome?
Studies of diagnostic/prognostic test accuracy: Test result does not cause outcome.

28. Studies of Medical Tests
Causality irrelevant.
Not enough to show that test result is associated with disease status or outcome*.
Need to estimate parameters (e.g., sensitivity and specificity) describing test performance.
*Although if it isn’t, you can stop.

29. Studies of Diagnostic Test Accuracy for Prevalent Disease
Predictor = Test Result
Outcome = Disease status as determined by Gold Standard
Designs:
Case-control (sample separately from disease positive and disease negative groups)
Cross-sectional (sample from the whole population of interest)

30. Dichotomous Tests Sensitivity = a/(a + c) Specificity = d/(b + d)
Disease +
Disease -
Test + a
True Positives b
False Positives
Test - c
False Negatives d
True Negatives
Total
a + c
Total With Disease
b + d
Total Without
Disease
Sensitivity = a/(a + c)
Specificity = d/(b + d)

31. Sensitivity and Specificity
PID = “Positive In Disease”
Proportion of D+ patients with “+” test result
Specificity
NIH = “Negative in Health”
Proportion of D- patients with “–” test result

32. Studies of Dx Tests Importance of Sampling Scheme
If sampling separately from Disease+ and Disease– groups (case-control sampling), cannot calculate prevalence, positive predictive value, or negative predictive value.

33. Dx Test:Case-Control Sampling
Disease +
Sampled Separately
Disease –
Test + a
True Positives b
False Positives
Test - c
False Negatives d
True Negatives
Total
a + c
Total With Disease
b + d
Total Without
Disease
Sensitivity = a/(a + c)
Specificity = d/(b + d)

34. Dx Test: Cross-sectional Sampling
Disease +
Disease -
Total
Test + a
True Positives b
False Positives
a + b
Total Positives
Test - c
False Negatives d
True Negatives
c + d
Total Negatives
a + c
Total With Disease
b + d
Total Without
Disease
a + b + c + d
Total N
Prevalence = (a + c)/N
Positive Predictive Value = a/(a + b)
Negative Predictive Value = d/(c + d)

35. Immunohistochemical Test for ARVC*
Yes No
Immuno-histochemical
Test
Positive 10 2 12
Negative 1 9 11
Sensitivity =
10/11 = 91%
Specificity =
9/11 = 82%
*N Engl J Med Mar 12;360(11):

36. Immunohistochemical Test for ARVC*
Yes No
Immuno-histochemical
Test
Positive 10 2 12
Negative 1 9 11
PPV =
10/12 = 83%
NPV =
9/10 = 90%
Your patient has a negative result on this test. Does the NPV of 90% mean he still has a 10% chance of ARVC?
*N Engl J Med Mar 12;360(11):

37. Sample Size Calculations for Studies of Diagnostic Test Accuracy
Sensitivity and Specificity are descriptive proportions.*
Choose N with disease to estimate sensitivity with the desired precision.
Choose N without disease to estimate specificity with the desired precision.
*Table 6E, page 91 DCR3

38. LR(result) = P(result|D+)/P(result|D-)
Likelihood Ratio
P(Result) in patient WITH disease
P(Result) in patients WITHOUT disease
LR(result) = P(result|D+)/P(result|D-)
See DCR3, Chapter 12, page 191

39. Sample Size Calculations for Studies of Diagnostic Test Accuracy
Size the sample to estimate a likelihood ratio with the desired precision.
See DCR3, Chapter 12, page 191

40. Studies of Prognostic Tests for Incident Outcomes
Predictor = Test Result
Development of outcome or time to development of outcome.
Design:
Cohort study

41. Studies of Prognostic Tests for Incident Outcomes
Prognostic test “result” is often a probability of having the outcome by a future time point (e.g. risk of death or recurrence by 5 years).
Need to assess both calibration and discrimination.

42. Comparing Predictions
Evidence-Based Diagnosis, Chapter 7
Jan. 30, 2008 Issue of Statistics in Medicine*
*Pencina et al. Stat Med Jan 30;27(2):157-72;

43. Examples
Name that observational study design

44. JIFee
Babies born at Kaiser with neonatal hyperbilirubinemia (Bili > 25) are compared with randomly selected “controls” from the same birth cohort.
Outcome measure is IQ and neurologic status at age 5 years.
No difference in IQ or fraction with neurologic disability between the “case” and “control” groups.
Newman, T. B., P. Liljestrand, et al. (2006). N Engl J Med 354(18):

45. JIFee
Design?
(Be Careful)

46. RRISK (Reproductive Risk Factors for Incontinence at Kaiser)
Random sample of 2100 women aged yo
Interview, self report, diaries to determine whether they have the outcome, urinary incontinence.
Chart abstraction of L&D/surgical records to establish predictor status

47. RRISK
Design?

48. HIV Tropism and Rapid Progression*
Is HIV CXCR4 (as opposed to CCR5) tropism a predictor of rapid progression in acutely infected HIV patients?
Molecular tropism assay is “high end” and labor-intensive. Have funding to perform a total of 80 assays.
UCSF OPTIONS cohort follows patients acutely infected with HIV. Has banked serum from near time of acute infection.
* Vivek Jain’s Project

49. HIV Tropism and Rapid Progression (continued)
Identify the 40 patients with the most rapid progression (Group 1) and randomly select 40 others from the UCSF Options cohort (Group 2).
Run the tropism assay on banked serum for these 80 patients and compare results between Group 1 and Group 2.

50. HIV Tropism and Rapid Progression
Design?

51. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
Subjects: Patients presenting to the SFGH ED with RLQ pain in 1998 and 2003
Predictor: Year of presentation
Outcome: Receipt of parenteral analgesia
Neighbor ML, Baird C*, Kohn MA. Changing Opioid Use for Right Lower Quadrant Abdominal Pain in the ED. Acad Emerg Med 2005; 12(12):
. *UCSF MSIII

52. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
Yes No
Total
Prevalence
2003 72 65
137
53%
1998 43
144
187
23%
Relative Prevalence = 2.3

53. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
Design?

54. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
1. Patients who get abdominal CTs are much more likely to get analgesia (59% vs. 22%)
2. CTs were much more common in 2003 than in 1998 (56% vs 20%)
Is the increase in analgesia rates between 1998 and 2003 wholly explained by increased CT scanning?

55. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
CT YES
CT NO
Year
Analg
No Analg
Total
Prev
Prev.
2003 47 30 77
61% 25 35 60
42%
1998 21 17 38
55% 22
127
149
15%
Prev. Ratio
1.1
2.8

56. Year of Visit and Provision of Analgesia in ED Patients with RLQ Pain
Is the increase in analgesia rates between 1998 and 2003 wholly explained by increased CT scanning?
NO. In the group that did not receive CTs, analgesia rate was almost 3x higher in 2003 than in 1998.

57. Enhancing Causal Inference
Stratification
DCR 3rd Ed. Chapter 9 pp. 137.
Also, Appendix 9A
Smoking as a confounder of the relationship between coffee drinking and MI.

58. Causal Inference and Confounding in Observational Studies
(Next Week)

59. B-hCG Example if Time

60. B-hCG and Ectopic Pregnancy
Subjects: All women with non-zero serum B-hCGs presenting to the SFGH ED for abdominal pain or vaginal bleeding between 9/1/96 and 6/30/99.
Predictor: B-hCG Level
Outcome: Pregnancy type (ectopic, spontaneous ab, normal IUP) determined on medical record review by trained abstractors using explicit criteria
Kohn MA, et al.. Acad Emerg Med 2003;10(2):

61. B-hCG and Ectopic Pregnancy
Results: 845 patients, pregnancy type (ectopic vs. intrauterine) could not be established in 115, leaving 730 for analysis
HCG EP
IUP
Total
Risk
<1500 40
118
158
25%
> 1500 56
516
572
10% 96
634
730
Risk Ratio
2.59

62. B-hCG and Ectopic Pregnancy
Design?

63. B-hCG and Ectopic PregnancyEP
IUP
Total
Risk
<1500 40
118
158
25%
> 1500 56
516
572
10% 96
634
730
13%
Risk Ratio
2.59
Your pregnant patient with abdominal pain has an HCG < Is her risk of ectopic pregnancy 25%?

64. B-hCG and Ectopic Pregnancy
β-hCG Distribution of Ectopic Pregnancy (EP) and Intrauterine Pregnancy (IUP)
β-hCG
Pregnancy Type
(mIU/mL) EP
IUP
Likelihood Ratio
< 1500 40
42%
118
19%
2.24
( ) 55
57%
313
49%
1.16
( )
> 50000* 1 1%
203
32%
0.03
( )
Total 96
100%
634
*Sensitivity and specificity of B-hCG < 50,000 mIU/mL for EP were 0.99 (95% CI: ) and 0.32 ( )