Distinguishing between Treatment Effects and DIF in a Substance Abuse Outcome Measures Using Multiple Indicator Multiple Causes (MIMIC) Models Barth B. Riley, Michael L. Dennis Chestnut Health Systems Study supported by National Institute on Drug Abuse Grant (NIDA) No. R37 DA11323.

Overview Differential Item Functioning and Its Impact The Multiple Indicator Multiple Cause (MIMIC) Model Demographic differences in substance use and substance abuse treatment Present Study

Differential Item Functioning DIF: Two groups differ in their likelihood of endorsing an item after controlling for differences on the measured construct. Group differences in the likelihood of endorsing an item may be due to: Group differences on the latent trait Differential item functioning (DIF) Both DIF can also occur over time

Differential Item and Test Functioning The presence of DIF items can reduce the validity of a measure in making between group comparisons. If DIF is of sufficient magnitude to cause measurement bias against one group relative to another, efforts to interpret outcomes measures becomes complex.

Did the persons change or did the items in the instrument change?

Analysis of DIF Several approaches have been employed for the analysis of DIF: T tests comparing item parameters between two groups Mantel-Haenszel contingency tables Logistic regression IRT Likelihood ratio tests Most of these approaches are limited to comparisons of two groups on a single factor. Do not directly assess impact of DIF on person measures.

Multiple Indicator Multiple Causes (MIMIC) Models Combines aspects of confirmatory factor analysis and structural equation modeling. The basic MIMIC models consist of the following components: A latent variable—the construct being measured. A set of measured indicators—items Grouping variables such as race and gender

Basic IRT Model Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Indicators

MIMIC Model, No DIF Assumed Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Latent Variable Ethnicity Gender Indirect effects Indicators

MIMIC Model with DIF Effects Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Latent Variable Ethnicity Gender Effect of DIF is partialed out of the indirect effects Indicators Direct DIF effect

Study The purpose of this study was to examine the effect of DIF by time, gender race on the Global Appraisal of Individual Needs (GAIN) Substance Problem Scale Data were collected from 446 participants as part of a three-year substance abuse early re- intervention study.

Participants (N=446) Recruited from community-based substance abuse treatment in Chicago in Participants were randomly assigned to either outcome monitoring or recovery management checkups, designed to help relapsing participants to return to treatment. Followed quarterly for 3 years. Participants were predominantly Male (54.5%) African American (80.2%) Average age: 38.4 years (SD=8.3)

Primary Drug

Substance Problem Scale The Substance Problem Scale (SPS) measures problems with alcohol/drug use during the past month, including abuse, dependence and substance-abuse health problems. Consists of 16 dichotomous items Based on DSM-IV-TR criteria for substance abuse and substance dependence. Internal consistency:.9 Test-retest reliability:.73

Model In order to assess treatment effects over time, a multilevel framework was used: Level 1: Time: random effect Level 2: Person: fixed effects Treatment variables: Random assignment to recovery management Days in outpatient, intensive outpatient and residential treatment DIF factors: gender and ethnicity One and two parameter IRT models were compared.

MIMIC Model: Within Level SPS Time Tx Participation Tx Participation SPS 1 SPS 2 SPS 3 SPS n ………… Control for DIF

MIMIC Model: Between Level SPS Race Gender RMC SPS 1 SPS 2 SPS 3 SPS n ………… Control for DIF Opiates

Goodness of Fit ModelCFITLIRMSEA 1 Parameter IRT Basic IRT model MIMIC No DIF MIMIC DIF Parameter IRT Basic IRT model MIMIC No DIF MIMIC DIF N Cases = 400, N Observations = 5393

Time DIF

Gender DIF

Ethnicity DIF

Primary Drug DIF

Group Differences on DIF Factors Factor No DIF ModelDIF Model ZSig.Z Time Gender ns Ethnicity ns-0.845ns Primary Drug ns

Treatment Effects Treatment Variables No DIFDIF ZSig.Z Any outpatient tx Times in outpatient tx ns0.003ns Any intensive outpatient ns-0.324ns Days, intensive outpatient tx ns-0.007ns Any residential treatment Nights in residential tx Any methadone tx Days taking methadone ns-0.023ns Recovery Monitor. Checkups

Conclusions The MIMIC model is a promising tool for assessing the presence and impact of DIF on at the scale level (DTF). Controlling for DIF reduced differences in SPS measures as a function of gender and primary drug. Treatment effects as measured by the SPS were not affected by gender, ethnicity, primary drug or time DIF.

MIMIC: Strengths Assess DIF and DTF on multiple factors DIF factors can be discrete or continuous variables Distinguish between treatment and DIF effects Can be used in conjunction with longitudinal analysis methods (e.g., multilevel modeling).

MIMIC: Limitations/Caveats In order to specify the model, at least one item must be free of DIF (or have minimal DIF). Can not detect non-uniform DIF—DIF in the discrimination parameter Obtaining group specific item parameters is not straightforward Assumes consistent factor structure across groups

Useful References Muthén, B. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, Fleishman, L.A., & Lawrence, W.F. (2003). Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care, 41(7 Suppl.) III75-III86. MacIntosh, R. & Hashim, S. (2003). Converting MIMIC model parameters to IRT parameters in DIF analysis. Applied Psychological Measurement, 27, Finch, H. (2005). The MIMIC Model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4):

Contact Information A copy of this presentation will be at: For information on this method and a paper on it, please contact Barth Riley at