1. Distinguishing between Treatment Effects and DIF in a Substance Abuse Outcome Measures Using Multiple Indicator Multiple Causes (MIMIC) Models Barth B. Riley, Michael L. Dennis Chestnut Health Systems Study supported by National Institute on Drug Abuse Grant (NIDA) No. R37 DA11323.

2. Overview Differential Item Functioning and Its Impact The Multiple Indicator Multiple Cause (MIMIC) Model Demographic differences in substance use and substance abuse treatment Present Study

3. Differential Item Functioning DIF: Two groups differ in their likelihood of endorsing an item after controlling for differences on the measured construct. Group differences in the likelihood of endorsing an item may be due to: Group differences on the latent trait Differential item functioning (DIF) Both DIF can also occur over time

4. Differential Item and Test Functioning The presence of DIF items can reduce the validity of a measure in making between group comparisons. If DIF is of sufficient magnitude to cause measurement bias against one group relative to another, efforts to interpret outcomes measures becomes complex.

5. Did the persons change or did the items in the instrument change?

6. Analysis of DIF Several approaches have been employed for the analysis of DIF: T tests comparing item parameters between two groups Mantel-Haenszel contingency tables Logistic regression IRT Likelihood ratio tests Most of these approaches are limited to comparisons of two groups on a single factor. Do not directly assess impact of DIF on person measures.

7. Multiple Indicator Multiple Causes (MIMIC) Models Combines aspects of confirmatory factor analysis and structural equation modeling. The basic MIMIC models consist of the following components: A latent variable—the construct being measured. A set of measured indicators—items Grouping variables such as race and gender

8. Basic IRT Model Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Indicators

9. MIMIC Model, No DIF Assumed Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Latent Variable Ethnicity Gender Indirect effects Indicators

10. MIMIC Model with DIF Effects Latent Construct Latent Construct Item 1 Item 2 Item 3 Item n ………… Latent Variable Ethnicity Gender Effect of DIF is partialed out of the indirect effects Indicators Direct DIF effect

11. Study The purpose of this study was to examine the effect of DIF by time, gender race on the Global Appraisal of Individual Needs (GAIN) Substance Problem Scale Data were collected from 446 participants as part of a three-year substance abuse early re- intervention study.

12. Participants (N=446) Recruited from community-based substance abuse treatment in Chicago in 2004. Participants were randomly assigned to either outcome monitoring or recovery management checkups, designed to help relapsing participants to return to treatment. Followed quarterly for 3 years. Participants were predominantly Male (54.5%) African American (80.2%) Average age: 38.4 years (SD=8.3)

13. Primary Drug

14. Substance Problem Scale The Substance Problem Scale (SPS) measures problems with alcohol/drug use during the past month, including abuse, dependence and substance-abuse health problems. Consists of 16 dichotomous items Based on DSM-IV-TR criteria for substance abuse and substance dependence. Internal consistency:.9 Test-retest reliability:.73

15. Model In order to assess treatment effects over time, a multilevel framework was used: Level 1: Time: random effect Level 2: Person: fixed effects Treatment variables: Random assignment to recovery management Days in outpatient, intensive outpatient and residential treatment DIF factors: gender and ethnicity One and two parameter IRT models were compared.

16. MIMIC Model: Within Level SPS Time Tx Participation Tx Participation SPS 1 SPS 2 SPS 3 SPS n ………… Control for DIF

17. MIMIC Model: Between Level SPS Race Gender RMC SPS 1 SPS 2 SPS 3 SPS n ………… Control for DIF Opiates

18. Goodness of Fit ModelCFITLIRMSEA 1 Parameter IRT Basic IRT model.974.970.099 MIMIC No DIF.972.961.076 MIMIC DIF.974.970.075 2 Parameter IRT Basic IRT model.952.994.045 MIMIC No DIF.975.994.03 MIMIC DIF.976.994.03 N Cases = 400, N Observations = 5393

19. Time DIF

21. Gender DIF

22. Ethnicity DIF

23. Primary Drug DIF

24. Group Differences on DIF Factors Factor No DIF ModelDIF Model ZSig.Z Time -0.271.001-0.281.001 Gender 0.217.0410.206ns Ethnicity -0.733ns-0.845ns Primary Drug 0.333.0150.257ns

25. Treatment Effects Treatment Variables No DIFDIF ZSig.Z Any outpatient tx. -0.606.001-0.605.001 Times in outpatient tx. 0.003ns0.003ns Any intensive outpatient -0.325ns-0.324ns Days, intensive outpatient tx. -0.007ns-0.007ns Any residential treatment 0.435.0010.434.001 Nights in residential tx. -0.306.001-0.306.001 Any methadone tx. 0.636.0010.635.001 Days taking methadone -0.023ns-0.023ns Recovery Monitor. Checkups -0.248.018-0.249.018

26. Conclusions The MIMIC model is a promising tool for assessing the presence and impact of DIF on at the scale level (DTF). Controlling for DIF reduced differences in SPS measures as a function of gender and primary drug. Treatment effects as measured by the SPS were not affected by gender, ethnicity, primary drug or time DIF.

27. MIMIC: Strengths Assess DIF and DTF on multiple factors DIF factors can be discrete or continuous variables Distinguish between treatment and DIF effects Can be used in conjunction with longitudinal analysis methods (e.g., multilevel modeling).

28. MIMIC: Limitations/Caveats In order to specify the model, at least one item must be free of DIF (or have minimal DIF). Can not detect non-uniform DIF—DIF in the discrimination parameter Obtaining group specific item parameters is not straightforward Assumes consistent factor structure across groups

29. Useful References Muthén, B. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557-585. Fleishman, L.A., & Lawrence, W.F. (2003). Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care, 41(7 Suppl.) III75-III86. MacIntosh, R. & Hashim, S. (2003). Converting MIMIC model parameters to IRT parameters in DIF analysis. Applied Psychological Measurement, 27, 372-379. Finch, H. (2005). The MIMIC Model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4):278-295.

30. Contact Information A copy of this presentation will be at: www.chestnut.org/li/posters For information on this method and a paper on it, please contact Barth Riley at bbriley@uic.edu.