1. Bayesian modeling of nonsampling error Alan M. Zaslavsky Harvard Medical School

2. General setup for nonsampling error Focus on measurement error problem –Item responses with error –Item or unit nonresponse as a special error response –…or nonresponse as part of error for aggregates Y = data measured with error Y* = latent “true” values (object of inference) –Might be observed for part of data (calibration) X = covariates –Assumed (for presentation) correct and complete –Include design information

3. Objective of inference Estimate statistics of “true” values f(Y*) Estimate parameters of models –From likelihood standpoint: inference from L(  | Y*,X) –(Specifically) from Bayesian standpoint, draw from P(  | Y*,X) Both possible if we have draws of Y* –Multiple imputation for valid inferences

4. Two ways to factorize distribution Predictive factorization: P(Y,Y* | X, ,  *) = P(Y* | Y,X,  *) P(Y | X,  ) –Direct prediction of Y* for imputation “Scientific” factorization: P(Y,Y* | X, ,  *) = P(Y | Y*,X,  ) P(Y* | X,  *) –First factor is observation (measurement error) model –Second factor is model for true relationships

5. More on “scientific” factorization Separates two distinct processes –Information might be from different sources –Possibility of more (or different) generalizability Models are more interpretable –Incorporate prior information for specification and parameters –Easier to assess “congeniality” of models? Compare model for P(Y* | X,  *) with model involving  –Simplifications? e.g. P(Y | Y*,X,  ) = P(Y | Y*,  )

6. Inference with “scientific” factorization Computations via Gibbs sampler –Imputation of Y* by Bayes’s theorem –Complete-data inferences for  * Inferences of scientific interest (  ) –Multiple imputation inference using Y* –Direct from model if  =  (  *)

7. Possible sources for measurement error model parameters (  ) Calibration study –Sample of (Y,Y*) pairs to identify the two parameters –For robustness, important to build in adequate flexibility to avoid identifying off unverified model assumptions about P(Y | X, ,  *) Prior studies (also used Bayesianly as prior) –Previous calibration model estimates, if measurement process is consistent –Synthesis of accumulated survey methodology

8. Example 1: Correction for underreporting in study of chemotherapy for colorectal cancer Provision of guideline-recommended adjuvant chemotherapy a critical issue in quality of care for cancer Cancer registries as a source of chemo data –Excellent population coverage –Underreporting of treatment

9. California study Cancer registry data –Statewide coverage –About 70,000 cases over 5 years in relevant stages (appropriate for chemotherapy) Calibration survey –Request medical record data from physicians –Limited in time (1+ year) and space (3 of 10 regions) –1956 cases in sample, 1449 (74%) respond

10. Reporting of adjuvant therapy Folllowup survey response rate higher … –at HMO-affiliated and high-volume hospitals –when chemo reported in original record 82% of adjuvant therapy was reported to Registry (among “respondents”) –Substantial underestimation if Registry alone used –More complete in teaching hospitals, HMO affiliates, high volume hospitals, younger and rectal cancer patients Cress et al., Medical Care 2003

11. Naïve estimation of administration of adjuvant chemotherapy Analysis based only on “gold standard” survey + Registry data in sample Strong variation by patient characteristics –Age (less if older), marital status –Race (less if Black, more if Hispanic, Asian) –Income (upward gradient with higher income) Substantial unexplained hospital-level variation Ayanian et al., J Clinical Oncology 2003

12. Limitations of standard analytic approaches Survey respondents alone: –Small portion of available California data (1449/70,000) –Single area of state –Unrepresentative due to survey nonresponse –Confounding of survey response, reporting, treatment variation (e.g. volume effects) Registry data alone: –Underreporting of chemotherapy –Reporting is nonuniform

13. Combining Registry and survey data Combine –power of large Registry data – correction for underreporting based on survey Simple correction based on: P(reported chemo) = P(chemo)  P(report | chemo) Therefore : P(chemo) = P(reported chemo) / P(report | chemo)

14. Registry plus simple correction In survey: P(reported chemo) = 59% P(report | chemo) = 82% P(chemo) = 59% / 82% ≈ 71% Outside survey (mostly rest of state): P(reported chemo) = 49% P(report | chemo) = 82% P(chemo) = 49% / 82%≈ 60%  Depends on assumption that reporting is similar in the two areas

15. Model-based methodology (Yucel and Zaslavsky) Disaggregated model –Take into account individual effects on both chemotherapy and reporting –Take into account hospital variation in both chemotherapy and reporting Imputation of chemo for individual cases –Allow fitting of any desired models –Multiple imputation to obtain proper measures of uncertainty with imputed data

16. Models for reporting and therapy Logit or Probit regression for therapy (outcome) –Patient p has characteristics x hp : age, sex, race/ethnicity, comorbidity score (Charlson), tumor stage/site, income category –Hospital h has characteristics z h : volume, ACOS-certified registry, teaching –Random effect  h for hospital h logit P(chemo hp ) =  x hp + z h +  h Similar model (with or without random effect) for reporting given therapy –Random effects for reporting & therapy could be correlated

17. Two versions of hierarchical model (a) single random effect (b) bivariate RE ←Parameters→ Latent “true” status Observed status ReportingOutcome Reporting Outcome

18. Fitting the model Full Bayesian specification –Diffuse priors for coefficients, (co)variances Fit via Gibbs sampling: alternately –Impute true chemo status for non-survey cases –Draw random hospital effects  –Draw “fixed” coefficients  and variance components 

19. Imputing chemo status (Bayes thrm) Example: consider individual (not in survey) for whom models give –Prior P(chemo)=70% –Prior P(reporting | chemo) = 80% If chemo reported, then true chemo = 1 If chemo not reported: –P(no chemo, no report) = 30% –P(chemo, no report) = 70%  20% = 14% –P(chemo | no report) = 14% / (14% + 30%) ≈ 32% –Impute chemo=1 with probability 32%

20. Computing: probit via latent variables Probit model:  (P(Y hp =1))=  x hp + z h +  h –Equivalently: Y hp =1 ↔ e hp <  x hp + z h +  h, where e hp ~N(0,1) is a normal latent variable (Albert & Chib 1993) –Equivalently, Y hp =1 ↔ u hp =  x hp + z h +  h −e hp >0 –Observing Y hp implies truncated normal posterior for u hp given higher-level parameters  h Given a draw of u hp, higher levels reduce to normal multilevel model with observation u hp and fixed variance=1 at bottom level (well-known problem) independent of the discrete data or imputed values direct generalization to correlated bivariate response

21. “Restricted” inference for robustness Two kinds of information involved in inference for “reporting” model –“Direct” in survey sample (1449 cases): Y | Y*, parameters, X –“Indirect” in remaining area (~74,000+ cases): Y | parameters, X (combines outcome & reporting models) –Possibly sensitive to model misspecification? Ad hoc solution: Restrict likelihood for reporting model to direct data from reporting survey cases –Throw away some information from others –Greater robustness to slight misspecification? –Reparametrize  as regression  (R) |  (O) & marginal  (O)

22. Direct interpretation of fitted model Effects broadly similar to those in naïve (sample only) analyses. –Volume effect on reporting but not on chemo –Lower chemo rate outside survey region Substantial hospital random effects in both reporting and therapy rates –Indication of substantial unexplained variation – a problem (from health services standpoint)! –Reporting completeness and therapy rates not (residually) correlated

23. Using imputations to estimate effect of chemotherapy on survival Re-fit model including 2-year survival as predictor of chemotherapy Using imputed corrected chemotherapy, fit model with chemotherapy (and other variables) as predictor of survival –Correct variances with multiple imputation –Missing info ≈70% for chemo, 1-4% for other variables Finds significant positive effect (OR=1.26) of chemo on survival –[Are the severity controls good enough?]

24. Modeling critical with missing data Several kinds of missing data: –Unreported chemotherapy –Nonresponse to followback (validation) survey –Areas excluded from followback survey Potential for confounding if unjustifiable MCAR (or insufficiently conditional MAR) assumptions are made –MCAR = Missing Completely at Random: missingness independent of everything –MAR = Missing at Random: missingness independent of unobserved, conditional on observed

25. Some countinterintuitive results!

26. Limitations and potential design improvements Major limitation: calibration survey is unrepresentative (in known ways) –Only covers some areas (trial implementation) –Differences by region in reporting are plausible –Can evaluate sensitivity to alternative assumptions Could improve design for ongoing studies –Sample across entire area –Quality improvement for both therapy and reporting

27. Example 2: Adjustment for measurement bias of 1990 Post Enumeration Survey Post-Enumeration Survey provides estimates of proportional error in Decennial Census estimates –Includes whole-household and within- household under- and overenumerations –Tabulated for poststrata of individuals defined by household-level (region, urbanicity) and individual-level (age, sex, race/ethnicity) variables

28. k = domain index c k = population share of domain k y* k = true census underenumeration rate = (biased) estimate of y* k from survey y k = E = expectation, b k = y k −y* k = bias = unbiased estimate of b k, E = b k Constraints:  c k y* k =  c k y k =  c k =  c k b k = 0 (sum of errors in shares is 0). Sampling variance of = Var | y = V y Notation for undercount estimation (Zaslavsky 1993, JASA)

29. Components and variance of Sources of bias estimates (total error model) –Small calibration studies to estimate process errors (matching, geocoding, fabrications) –Model-based estimates of correlation bias –Uncertainty about imputation model Var ( − b) = V b includes –Sampling variances from calibration studies, –Uncertainty across correlation bias models, –(Multiple) imputation variance and model uncertainty

30. A naïve approach and its problems Simple bias corrected estimate is –Unbiased estimator of y* –Variance is V y + V b and V b is likely to be large –Problem for non-Bayesian approaches: if we have very little data to estimate something, must we assume that it could be “anything”? Alternative (Bayesian) approach: introduce reasonable prior beliefs –Bias terms b k are a collection centered around 0 –Characterize variability by variance component –Similar argument for undercount terms y k

31. Hierarchical model for estimation and bias correction “Sampling” model: –Not exactly “sampling” since some model uncertainty is included in V b “Structural” (Level 2) model:

32. Hierarchical model for estimation and bias correction “ Structural” (Level 2) model: –Undercount and bias terms each drawn from common distribution –Proportional covariance structures for each and for correlation of the two –Matrix U based on a prior “similarity” of domains (number of common characteristics)

33. Priors and inference Fairly vague priors for variance components, correlation –These represent assessments of degree of variation in bias, undercount and how they relate across domains –Key to this inference is existence of collection of domains Inference via Gibbs sampler Extensive simulations –Compare to uniform shrinkage, hypothesis testing approaches, etc. –Suggested that full hierarchical Bayes model would outperform competitors

34. Analyses with 1992 data Data combined 3 sources –1990 census –Post-Enumeration Survey –Various sources of bias component estimates Estimates: –Substantial differential undercount, –Substantial differential bias,

35. Refinement: misaligned domains (Zaslavsky 1992, Proc. SRMS) Domains for bias estimates might differ from those for y –e.g. if they combine the main domains –Observation is Modifies the sampling model: Applied to 1992 data: –357 poststrata, 51 poststratum groups, but only 10 evaluation poststrata

36. Other potential applications Domain-level estimates –No gold standard data for individuals –No individual-level corrections Many applications where there are small evaluation samples for a measure –Welfare or food stamp payment error –Quality evaluations in medical care

37. Example 3: Imputation of households to correct for enumeration error Setting: Census (or survey) of households with errors of enumeration –Whole-household errors –Within-household errors –[Assumption (here) that all errors are omissions] Objective: To (multiply) impute corrected rosters. –Add person to households –Impute additional households

38. Bayesian imputation strategy (Zaslavsky 2004; Zaslavsky & Rubin 1989 Proc. ARC) Based on “scientific” factorization –Prevalence model: distribution of households by compositional type (roster of members by poststratum), P(Y* bk =t |  k )  k = (latent) parameter of block b –Observational model: probability of observed types (with error), P(Y bk =u | Y* bk =t,  )

39. Model specifics Prevalence models – x(t) summarizes characteristics of type t –Prevalence proportional to exp(x(t) ·  k ) · h(t) h(t) is (nonparametric) general prevalence of type t Observational model Loglinear model based on probabilities of omission of individuals Terms for dependence of omissions within household Could be based on (hypothetical) dataset … … and/or calibrated to match aggregate omission rate estimates by poststratum

40. Imputations Draw Y* bk by Bayes’s theorem –Possible values are those types that could “lose” one or more members yielding observed Y* bk –Draw from all possible values of t Special type for unobserved households –Count imputed using SOUP (unbiased) prior –True types imputed similar to others Gibbs sampler to estimate all parameters

41. General summary of examples All are “Bayesian” in drawing corrected values from posterior distributions –“Scientific” factorization for interpretability (Examples 1 and 3) –“Observations” might have simple (Ex. 1,2) or complex (Ex. 3) structure Bayesian also in –Incorporating prior information –Pooling across collections of units (“shrinkage”) –Hierarchical specification of complex models –Probability representation of model uncertainty (Ex. 2)

42. Program to move forward Systematic quantitative meta-analysis of information on nonresponse errors Models for various types of nonresponse error Think more about how to combine information from data and model uncertainty Standard algorithms and software Integrate with analyses of nonresponse, item missing data, etc.