1. Observational Study Designs and Studies of Medical Tests
Tom Newman
August 17, 2010
Thanks to Michael Kohn

2. Outline Conceptual overview Review common observational study designs
Cohort, Double Cohort
Case-Control
Cross-sectional
Studies of Medical Tests
Diagnostic Test Accuracy
Prognostic Test Accuracy
Examples
“Name that study”

3. Caveats Nomenclature is confusing and used inconsistently
“Cross-sectional” can refer to timing or sampling
“Retrospective” does not always mean retrospective
Getting the name right is helpful, but it is more important to be able to explain what you want to do and have it make sense for your RQ
If you can’t name your study it’s worth making sure it makes sense

4. Key elements of study design
Timing of the study
Timing of variable occurrence and measurement
How the subjects will be sampled

5. Timing of the study
Prospective: investigator enrolls subjects and makes measurements in the present and future
Historical: investigator relates predictor variables that have already been measured to outcomes that have already occurred
Retrospective: can mean historical, but best reserved for case-control studies

6. Prospective studies
Control over subject selection and variable measurements
Have to wait for outcomes to occur
Take longer
More expensive

7. Historical studies
Less control over subject selection and variable measurements
Outcomes have already occurred
Done sooner
Less expensive

8. Timing of measurements
Longitudinal: measurements in subjects made at more than one time
Cross-sectional: predictor and outcome measured at the same time

9. Longitudinal timing of measurements
Predictor variable precedes outcome
Better for causality (reduces likelihood of “effect-cause”)
Measurement of predictor precedes measurement of outcome
No need for blinding of measurement of predictor variable
Needed to measure incidence = new cases/population at risk/time
Risk of getting the disease

10. Cross-sectional timing of measurements
Measurement of predictor and outcome at about the same time
Causality may be more difficult to infer
No loss to follow-up
Can only measure prevalence = existing cases at one point in time/population at risk
Prevalence = incidence x duration
Risk of having the disease
Not as good for causality

11. Example: “Incidence-Prevalence Bias”
In asymptomatic adults, prevalence of coronary calcium is lower in blacks than in whites*
Does this mean blacks get less heart disease?
No, incidence is greater, but duration is shorter**
*Doherty TM et al J Am Coll Cardiol. 1999;34:787–794
**Nieto FJ, Blumenthal RS. J Am Coll Cardiol, 2000; 36:

12. Sampling of subjects Usually best By predictor variable
By outcome variable
By other (e.g., demographic) factors that define the population of interest
Sometimes called “cross-sectional” sampling
Usually best

13. Study designs Descriptive Analytical Many studies of medical tests
Hint variables must VARY
If either the predictor or outcome variable does not vary in your study (e.g., because one value is an inclusion criterion) your study is descriptive
Analytical

15. Analytical study designs
Experimental
-- Randomized trial
Observational (today’s topic)
-- Cohort
-- Double Cohort (exposed-unexposed)
-- Case-control
-- Cross-sectional

16. Observational analytic studies
Causality is important
May be only ethical option for studying risk factors for disease
Often more efficient
Populations may be more representative
More intellectually interesting than RCTs?

17. Note on Figures
Following schematics of observational study designs assume:
Predictor = Risk Factor
Outcome = Disease
Both dichotomous

18. Cohort Study

19. Prospective Cohort Study19

20. Historical Cohort Study
THE PAST 20

21. Cohort Studies
1) Measure predictor variables on a sample from a population (defined by something other than the variables you are studying).
2) Exclude any subjects who already have the outcome.
3) Follow the subjects over time and attempt to determine outcome on all subjects.

22. Cohort Studies are longitudinal
Can identify individuals lost to follow up
Can estimate the incidence of the outcome in the population (e.g., cases/person-year)
Measure of disease association is the relative risk (RR) or relative hazard (RH)

23. Double Cohort Study

24. Double Cohort (Exposed-Unexposed) Studies
Sample study subjects separately based on predictor variable
Exclude potential subjects in whom outcome has already occurred.
Attempt to determine outcome in all subjects in both samples over time.

25. Double Cohort (Exposed-Unexposed) Studies
Can identify individuals lost to follow up
Can measure incidence in each cohort, but not overall incidence in the population*
Measure of disease association is the relative risk (RR) or relative hazard (RH)
*Unless one of the cohorts is a sample of everyone not in the other cohort

26. Cohort Studies: Summary
Timing of the STUDY
Prospective
Historical
Timing of the MEASUREMENTS:
All cohort studies are longitudinal (follow patients over time)
SAMPLING
Cohort study – sample based on other (e.g., demographic) characteristics
Double cohort study -- sample on predictor variable

27. Case-Control Study

28. Case-Control Study
1) Separately sample subjects with the outcome (cases) and without the outcome (controls)
2) Attempt to determine predictor status on all subjects in both outcome groups

29. Case-Control Study
Cannot identify individuals lost to follow up (no such thing as “lost to follow up”, since by definition outcome status is known)
Cannot calculate prevalence (or incidence) of outcome
Measure of disease association is the Odds Ratio (OR)
Try to replicate a nested case control study in which the cases and controls arise from the same cohort.

30. Nested Case-Control Study

31. Cross-Sectional Study

32. Cross-Sectional Study
Attempt to determine predictor and outcome status on all patients in a single population (defined by something other than predictor or outcome).

33. Cross-Sectional Study
No loss to follow-up
Can calculate prevalence but not incidence
Measure of disease association is the Relative Prevalence (RP).
Can be prospective or historical

34. Cohort Studies Start with a Cross-Sectional Study
Eliminate subjects who already have disease

35. Studies of Medical Tests
Causality often irrelevant.
Not enough to show that test result is associated with disease status or outcome*.
Need to estimate parameters (e.g., sensitivity and specificity) describing test performance.
*Although if it isn’t, you can stop.

36. Studies of Diagnostic Test Accuracy for Prevalent Disease
Predictor = Test Result
Outcome = Disease status as determined by Gold Standard
Designs:
Case-control (sample separately from disease positive and disease negative groups)
Cross-sectional (sample from the whole population of interest)
Double-cohort-like sampling (sample separately from test-positive and test-negative groups)

37. Dichotomous Tests Sensitivity = a/(a + c) Specificity = d/(b + d)
Disease +
Disease -
Test + a
True Positives b
False Positives
Test - c
False Negatives d
True Negatives
Total
a + c
Total With Disease
b + d
Total Without
Disease
Too small
Sensitivity = a/(a + c)
Specificity = d/(b + d)

38. Studies of Dx Tests Importance of Sampling Scheme
If sampling separately from Disease+ and Disease– groups (case-control sampling), cannot calculate prevalence, positive predictive value, or negative predictive value.

39. Dx Test: Case-Control Sampling
Disease +
Sampled Separately
Disease –
Test + a
True Positives b
False Positives
Test - c
False Negatives d
True Negatives
Total
a + c
Total With Disease
b + d
Total Without
Disease
Sensitivity = a/(a + c) Specificity = d/(b + d)

40. Dx Test: Cross-sectional Sampling
Disease +
Disease -
Total
Test + a
True Positives b
False Positives
a + b
Total Positives
Test - c
False Negatives d
True Negatives
c + d
Total Negatives
a + c
Total With Disease
b + d
Total Without
Disease
a + b + c + d
Total N
PPV =
a/(a + b)
NPV =
d/(c + d)
Prevalence =
(a + c)/N

41. Studies of Prognostic Tests for Incident Outcomes
Predictor = Test Result
Development of outcome or time to development of outcome.
Design: Cohort study

42. Examples
Name that observational study design

43. Outcomes: (blinded) IQ test and neurologic examination at age 5 years.
Babies born at Kaiser with severe neonatal hyperbilirubinemia (Bili  25) were compared with randomly selected “controls” from the same birth cohort.
Outcomes: (blinded) IQ test and neurologic examination at age 5 years.
Results: No difference in IQ or fraction with neurologic disability between the “case” and “control” groups.
Newman, T. B., P. Liljestrand, et al. (2006). N Engl J Med 354(18):

44. Jaundice and Infant Feeding Study
Design?
(Be Careful)

45. Double Cohort (Exposed-Unexposed) Study*
JIFee
Double Cohort (Exposed-Unexposed) Study*
The subjects are divided by predictor (Bili 25+), not outcome (neurologic disability). The “cases” are actually the exposed group and the “controls” are actually the unexposed group
*Actually a nested triple cohort study, since “cases” and “controls” came from the same birth cohort and we also studied dehydration. See Hulley page 104.

46. HIV Tropism and Rapid Progression*
Is HIV CXCR4 (as opposed to CCR5) tropism a predictor of rapid progression in acutely infected HIV patients?
Molecular tropism assay is expensive. Have funding to perform a total of 80 assays.
UCSF OPTIONS cohort follows patients acutely infected with HIV. Has banked serum from near time of acute infection.
* Vivek Jain’s Project

47. HIV Tropism and Rapid Progression (continued)
Identify the 40 patients with the most rapid progression (Group 1) and randomly select 40 others from the UCSF Options cohort (Group 2).
Run the tropism assay on banked serum for these 80 patients and compare results between Group 1 and Group 2.

48. HIV Tropism and Rapid Progression
Design?

49. HIV Tropism and Rapid Progression
Nested Case-Control Study

50. RRISK (Reproductive Risk Factors for Incontinence at Kaiser)
Random sample of 2100 women aged years old
Interview, self report, diaries to determine whether they have the outcome, urinary incontinence.
Chart abstraction of obstetrical and surgical records to establish predictor status

51. RRISK
Design?

52. RRISK Funded with an R01 by the NIDDK as a retrospective cohort study
Longitudinal, but can’t tell loss to follow-up, incidence of incontinence, or relative risk from this design
Michael calls it a cross-sectional study
Tells us prevalence of incontinence
But not all measurements made at the same time
It’s a lot like a nested case control study
But did not employ “case-control” sampling
Nested cross-sectional study?

53. Steroid treatment in the ED and among children hospitalized for asthma
Research Question: what are the frequency and predictors of delayed receipt of steroids in the ED among children admitted for asthma?
Subjects: children admitted for asthma
Predictors: age, time of arrival, etc.
Outcome: Time to steroid administration
what are the frequency and predictors of delayed receipt of steroids in the ED among children admitted for asthma?

54. Steroid treatment in the ED and among children hospitalized for asthma -2
This study is hard to name
Time to event data makes this sound like a cohort study (even if follow-up time is very short)
Define a group at risk of the outcome
Measure predictors
Follow for outcome occurrence
But there is a problem:
You can’t define a cohort based on variables not present at baseline

55. Steroid treatment in the ED and among children hospitalized for asthma -3
Possible changes
Make it a descriptive study of hospitalized patients
Make it a cohort study of ED patients
Could study predictors of time to steroids
Time to steroids could be a predictor of hospitalization

56. Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children*
Did pH probe, barium swallow, and endoscopy on 115 children with chronic respiratory tract disorders to determine GE reflux
Group 1: 74 children with GER
Group 2: 41 children with no GER
Bronchoscopy and bronchial lavage to determine LLAM
LLAM were present in 63/74 (85%) with GER
LLAM in 8/41 (19%) children without GER
P <
Design?
*J Pediatr 1987;110:190‑4

57. Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children*
Design: Cross-sectional study of diagnostic test accuracy (with cross-sectional sampling)
*J Pediatr 1987;110:190‑4

58. Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -3
Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.”
What is wrong?
J Pediatr 1987;110:190‑4

59. Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -4
Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.”
Study design does not permit this conclusion
Can’t estimate risk of developing lung disease from a
Cross-sectional sample that
Includes only patients with lung disease
J Pediatr 1987;110:190‑4