Observational Study Designs and Studies of Medical Tests Tom Newman August 17, 2010 Thanks to Michael Kohn

Outline Conceptual overview Review common observational study designs Cohort, Double Cohort Case-Control Cross-sectional Studies of Medical Tests Diagnostic Test Accuracy Prognostic Test Accuracy Examples “Name that study”

Caveats Nomenclature is confusing and used inconsistently “Cross-sectional” can refer to timing or sampling “Retrospective” does not always mean retrospective Getting the name right is helpful, but it is more important to be able to explain what you want to do and have it make sense for your RQ If you can’t name your study it’s worth making sure it makes sense

Key elements of study design Timing of the study Timing of variable occurrence and measurement How the subjects will be sampled

Timing of the study Prospective: investigator enrolls subjects and makes measurements in the present and future Historical: investigator relates predictor variables that have already been measured to outcomes that have already occurred Retrospective: can mean historical, but best reserved for case-control studies

Prospective studies Control over subject selection and variable measurements Have to wait for outcomes to occur Take longer More expensive

Historical studies Less control over subject selection and variable measurements Outcomes have already occurred Done sooner Less expensive

Timing of measurements Longitudinal: measurements in subjects made at more than one time Cross-sectional: predictor and outcome measured at the same time

Longitudinal timing of measurements Predictor variable precedes outcome Better for causality (reduces likelihood of “effect-cause”) Measurement of predictor precedes measurement of outcome No need for blinding of measurement of predictor variable Needed to measure incidence = new cases/population at risk/time Risk of getting the disease

Cross-sectional timing of measurements Measurement of predictor and outcome at about the same time Causality may be more difficult to infer No loss to follow-up Can only measure prevalence = existing cases at one point in time/population at risk Prevalence = incidence x duration Risk of having the disease Not as good for causality

Example: “Incidence-Prevalence Bias” In asymptomatic adults, prevalence of coronary calcium is lower in blacks than in whites* Does this mean blacks get less heart disease? No, incidence is greater, but duration is shorter** *Doherty TM et al J Am Coll Cardiol. 1999;34:787–794 **Nieto FJ, Blumenthal RS. J Am Coll Cardiol, 2000; 36:308-309

Sampling of subjects Usually best By predictor variable By outcome variable By other (e.g., demographic) factors that define the population of interest Sometimes called “cross-sectional” sampling Usually best

Study designs Descriptive Analytical Many studies of medical tests Hint variables must VARY If either the predictor or outcome variable does not vary in your study (e.g., because one value is an inclusion criterion) your study is descriptive Analytical

Analytical study designs Experimental -- Randomized trial Observational (today’s topic) -- Cohort -- Double Cohort (exposed-unexposed) -- Case-control -- Cross-sectional

Observational analytic studies Causality is important May be only ethical option for studying risk factors for disease Often more efficient Populations may be more representative More intellectually interesting than RCTs?

Note on Figures Following schematics of observational study designs assume: Predictor = Risk Factor Outcome = Disease Both dichotomous

Cohort Study

Prospective Cohort Study 19

Historical Cohort Study THE PAST 20

Cohort Studies 1) Measure predictor variables on a sample from a population (defined by something other than the variables you are studying). 2) Exclude any subjects who already have the outcome. 3) Follow the subjects over time and attempt to determine outcome on all subjects.

Cohort Studies are longitudinal Can identify individuals lost to follow up Can estimate the incidence of the outcome in the population (e.g., cases/person-year) Measure of disease association is the relative risk (RR) or relative hazard (RH)

Double Cohort Study

Double Cohort (Exposed-Unexposed) Studies Sample study subjects separately based on predictor variable Exclude potential subjects in whom outcome has already occurred. Attempt to determine outcome in all subjects in both samples over time.

Double Cohort (Exposed-Unexposed) Studies Can identify individuals lost to follow up Can measure incidence in each cohort, but not overall incidence in the population* Measure of disease association is the relative risk (RR) or relative hazard (RH) *Unless one of the cohorts is a sample of everyone not in the other cohort

Cohort Studies: Summary Timing of the STUDY Prospective Historical Timing of the MEASUREMENTS: All cohort studies are longitudinal (follow patients over time) SAMPLING Cohort study – sample based on other (e.g., demographic) characteristics Double cohort study -- sample on predictor variable

Case-Control Study

Case-Control Study 1) Separately sample subjects with the outcome (cases) and without the outcome (controls) 2) Attempt to determine predictor status on all subjects in both outcome groups

Case-Control Study Cannot identify individuals lost to follow up (no such thing as “lost to follow up”, since by definition outcome status is known) Cannot calculate prevalence (or incidence) of outcome Measure of disease association is the Odds Ratio (OR) Try to replicate a nested case control study in which the cases and controls arise from the same cohort.

Nested Case-Control Study

Cross-Sectional Study

Cross-Sectional Study Attempt to determine predictor and outcome status on all patients in a single population (defined by something other than predictor or outcome).

Cross-Sectional Study No loss to follow-up Can calculate prevalence but not incidence Measure of disease association is the Relative Prevalence (RP). Can be prospective or historical

Cohort Studies Start with a Cross-Sectional Study Eliminate subjects who already have disease

Studies of Medical Tests Causality often irrelevant. Not enough to show that test result is associated with disease status or outcome*. Need to estimate parameters (e.g., sensitivity and specificity) describing test performance. *Although if it isn’t, you can stop.

Studies of Diagnostic Test Accuracy for Prevalent Disease Predictor = Test Result Outcome = Disease status as determined by Gold Standard Designs: Case-control (sample separately from disease positive and disease negative groups) Cross-sectional (sample from the whole population of interest) Double-cohort-like sampling (sample separately from test-positive and test-negative groups)

Dichotomous Tests Sensitivity = a/(a + c) Specificity = d/(b + d) Disease + Disease - Test + a True Positives b False Positives Test - c False Negatives d True Negatives Total a + c Total With Disease b + d Total Without Disease Too small Sensitivity = a/(a + c) Specificity = d/(b + d)

Studies of Dx Tests Importance of Sampling Scheme If sampling separately from Disease+ and Disease– groups (case-control sampling), cannot calculate prevalence, positive predictive value, or negative predictive value.

Dx Test: Case-Control Sampling Disease + Sampled Separately Disease – Test + a True Positives b False Positives Test - c False Negatives d True Negatives Total a + c Total With Disease b + d Total Without Disease Sensitivity = a/(a + c) Specificity = d/(b + d)

Dx Test: Cross-sectional Sampling Disease + Disease - Total Test + a True Positives b False Positives a + b Total Positives Test - c False Negatives d True Negatives c + d Total Negatives a + c Total With Disease b + d Total Without Disease a + b + c + d Total N PPV = a/(a + b) NPV = d/(c + d) Prevalence = (a + c)/N

Studies of Prognostic Tests for Incident Outcomes Predictor = Test Result Development of outcome or time to development of outcome. Design: Cohort study

Examples Name that observational study design

Outcomes: (blinded) IQ test and neurologic examination at age 5 years. Babies born at Kaiser with severe neonatal hyperbilirubinemia (Bili  25) were compared with randomly selected “controls” from the same birth cohort. Outcomes: (blinded) IQ test and neurologic examination at age 5 years. Results: No difference in IQ or fraction with neurologic disability between the “case” and “control” groups. Newman, T. B., P. Liljestrand, et al. (2006). N Engl J Med 354(18): 1889-900.

Jaundice and Infant Feeding Study Design? (Be Careful)

Double Cohort (Exposed-Unexposed) Study* JIFee Double Cohort (Exposed-Unexposed) Study* The subjects are divided by predictor (Bili 25+), not outcome (neurologic disability). The “cases” are actually the exposed group and the “controls” are actually the unexposed group *Actually a nested triple cohort study, since “cases” and “controls” came from the same birth cohort and we also studied dehydration. See Hulley page 104.

HIV Tropism and Rapid Progression* Is HIV CXCR4 (as opposed to CCR5) tropism a predictor of rapid progression in acutely infected HIV patients? Molecular tropism assay is expensive. Have funding to perform a total of 80 assays. UCSF OPTIONS cohort follows patients acutely infected with HIV. Has banked serum from near time of acute infection. * Vivek Jain’s Project

HIV Tropism and Rapid Progression (continued) Identify the 40 patients with the most rapid progression (Group 1) and randomly select 40 others from the UCSF Options cohort (Group 2). Run the tropism assay on banked serum for these 80 patients and compare results between Group 1 and Group 2.

HIV Tropism and Rapid Progression Design?

HIV Tropism and Rapid Progression Nested Case-Control Study

RRISK (Reproductive Risk Factors for Incontinence at Kaiser) Random sample of 2100 women aged 40-69 years old Interview, self report, diaries to determine whether they have the outcome, urinary incontinence. Chart abstraction of obstetrical and surgical records to establish predictor status

RRISK Design?

RRISK Funded with an R01 by the NIDDK as a retrospective cohort study Longitudinal, but can’t tell loss to follow-up, incidence of incontinence, or relative risk from this design Michael calls it a cross-sectional study Tells us prevalence of incontinence But not all measurements made at the same time It’s a lot like a nested case control study But did not employ “case-control” sampling Nested cross-sectional study?

Steroid treatment in the ED and among children hospitalized for asthma Research Question: what are the frequency and predictors of delayed receipt of steroids in the ED among children admitted for asthma? Subjects: children admitted for asthma Predictors: age, time of arrival, etc. Outcome: Time to steroid administration what are the frequency and predictors of delayed receipt of steroids in the ED among children admitted for asthma?

Steroid treatment in the ED and among children hospitalized for asthma -2 This study is hard to name Time to event data makes this sound like a cohort study (even if follow-up time is very short) Define a group at risk of the outcome Measure predictors Follow for outcome occurrence But there is a problem: You can’t define a cohort based on variables not present at baseline

Steroid treatment in the ED and among children hospitalized for asthma -3 Possible changes Make it a descriptive study of hospitalized patients Make it a cohort study of ED patients Could study predictors of time to steroids Time to steroids could be a predictor of hospitalization

Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children* Did pH probe, barium swallow, and endoscopy on 115 children with chronic respiratory tract disorders to determine GE reflux Group 1: 74 children with GER Group 2: 41 children with no GER Bronchoscopy and bronchial lavage to determine LLAM LLAM were present in 63/74 (85%) with GER LLAM in 8/41 (19%) children without GER P < 0.0001 Design? *J Pediatr 1987;110:190‑4

Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children* Design: Cross-sectional study of diagnostic test accuracy (with cross-sectional sampling) *J Pediatr 1987;110:190‑4

Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -3 Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.” What is wrong? J Pediatr 1987;110:190‑4

Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -4 Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.” Study design does not permit this conclusion Can’t estimate risk of developing lung disease from a Cross-sectional sample that Includes only patients with lung disease J Pediatr 1987;110:190‑4