2. Michelle White, Ph.D. Chestnut Health Systems, Bloomington, IL Presentation for the Joint Meeting on Adolescent Treatment Effectiveness, Baltimore, MD, March 28, 2006, Maryland E Room. Preparation of this manuscript was supported by funding from the Center for Substance Abuse Treatment (CSAT Contract no. 270-2003- 00006). The content of this poster are the opinions of the author and do not reflect the views or policies of the government. Available on line at www.chestnut.org/LI/Posters or by contacting the author at (309) 827- 6026 or by e-Mail: mwhite@Chestnut.Org The Effects of a Quality Assurance (QA) and Certification Program on the Quality and Validity of Adolescent Substance Abuse Treatment Data

3. Summary Important judgments about how we treat adolescents for substance use disorders are typically based on data collected by clinicians and research staff. There has been much debate about the quality of self-report data, but little has been said about the impact of a quality assurance and certification protocol for interviewers collecting self- report data on the quality of that data. We now have data that shows that investing in an appropriate training and quality assurance technology can improve the quality of data collected from adolescents receiving substance abuse treatment. The Global Appraisal of Individual Needs (GAIN) Coordinating Center provides a clear and comprehensive training and quality assurance/certification program for the use of its bio-psychosocial assessments. This program has evolved over time and has been collecting data from many adolescent substance abuse treatment providers (81 providers currently submit data) for 9 years. This study examines the impact of our current quality assurance and certification program on the validity of the data being collected. We will compare interviews by levels of certifications achieved (site interviewer, administrator, or local trainer) and with interviews conducted by non- certified staff (interviews by people who eventually became certified, interviews by staff who were supposed to become certified but did not achieve it, and interviews by staff who were trained before the current certification program existed). We will also control for other potentially important factors like prior GAIN interview experience, job role (clinical versus research), and ability to meet certification deadlines. The dependent variables will be assessment duration, data quality (amount of missing/bad data, inconsistencies, randomness of response patterns, and atypical response patterns), and internal consistency (alpha). Implications for workforce development and issues for practical application of assessment certification programs will be discussed.

4. Treatment Planning Decisions are largely based on self-report information Newer studies show self-report data to be, overall, reliable and valid Context may affect reliability (e.g. reports of criminal acts by an inmate to a judge less reliable than a treatment study with certificate of confidentiality) Staff training and supervision are key to ensuring the most reliable and valid self-report data regardless of setting

5. Goals of this presentation 1. Describe GAIN training and quality assurance and certification program 2. Evaluate the extent to which this process improves the efficiency, reliability and validity of GAIN data in practice

6. The GAIN has a comprehensive training and certification program Train the trainer model to have “onsite expert” Administration certification Local trainer certification Local training / Site interviewer certification

7. GAIN Training National trainings held in Bloomington IL about 4 times/year and is 4 days in length and include breakfasts, lunches, materials, certification to local trainer level, licensing, software purchase (for single agency), and support for 1 year (up to 4 hours). On-Site trainings held round the year as contracted and are 3 and a half days in length. Covers administration, quality assurance, clinical use, software use, training issues, and other issues

8. Who attends training? Day-to-day person(s) who will be training others to administer the GAIN Person who will training/supervising others to use the GAIN Clinical supervisor/trainer Trainees range in education from GED/high school to PhD/MD Trainees range in technical experience from administrative to clinical to research

9. Administration Certification Process requires: Train the trainer coursework certification Submission of 2 or more taped interviews and participation in a written and oral review process Quality assurance audiotape reviews to demonstrate mastery of materials (typically 2-4) First tape must be submitted within 2 weeks and process must be completed within 3 months of coursework

10. Local Trainer Certification Local trainers can directly train and certify site interviewers on the GAIN within their agency Must be a certified administrator and deliver local training and conduct QA reviews on staff Must submit taped assessment of someone they trained AND completed feedback 2 stages of review to pass (1 with trainee not ready, 1 ready for certification) Process must be completed within 6 months of train-the-trainer coursework certification

11. Local Trainings/Site Interviewer Local trainings and QA are more flexible/do not have to follow rigid process but still have accountability for same standards for completing certification like the administration level A site interviewer is trained by a local trainer and does not have coursework certification A site interviewer CANNOT go on to local trainer status until completing coursework certification as part of a national training

12. Administration certification shown to reduce GAIN Administration time 0 20 40 60 80 100 120 140 160 Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Jun-04 Jul-04 Aug-04 Full (n=415) (127 to 116 min) Core (n=428) (118 to 96 min) Core + (n=140) (143 to 119 min) Total (n=983) (132 to 108 min) Practice, QA and Certification Reduced Duration by 31%

13. What Does a QA Reviewer do? Listens to the GAIN interview audiotape and paper copy to determine the quality of the administration. Reports issues using specific behavioral criteria found within four main sections of the feedback: Documentation, Instructions, Items, and Engagement. Reviewer rates each of 4 sections as: Excellent, Sufficient, Minor Problems, Problems and rates overall administration as pending (not certified) or certified

14. QA Reviewer Ratings 100% Agreement in certification status rating in blind reviews (CHS reviewers) Increase in inter-rater reliability from first attempt to last attempt at local trainer certification by trainees with CHS reviewer (Kappas improve): Documentation (.28 to.66) Instructions (.25 to.58) Items (.17 to.65) Engagement (.14 to.45)

15. Data Source Used data collected from 7,269 adolescents interviewed with GAIN version 5 as part of 89 CSAT adolescent treatment grants since 2002 Data are from geographically disperse programs CSAT adolescent treatment clients are similar to those in the public treatment system as represented by the TEDS public 2003 admissions

16. Geographic Location of Sites ART EAT SCY TCE YORP AK AL AR AZ CA CO DC DE FL GA HI IA ID IN KS LA MD ME MI MN MO MS MT NC ND NE NH NJ NM NV NY OH OK OR PA RI SC SD TN TX UT VA WA WV WY PR VT WI IL KY MA CT DC Program

17. Demographics 30% 19% 58% 16% 6% 17% 83% 18% 42% 17% 23% 20% 73% 29% 0%10%20%30%40%50%60%70%80%90%100% Female African American Caucasian Hispanic Mixed/Other 12 to 14 years old 15 to 17 years old TEDS (n=153,251) CSAT (n=7,226)

18. Clinical Severity 82% 33% 50% 48% 53% 37% 61% 53% 68% 74% 0%10%20%30%40%50%60%70%80%90%100% First used under age 15 Prior Treatment Weekly use at intake Past Year Dependence Criminal Justice System TEDS (n=153,251) CSAT (n=7,226)

19. Primary, Secondary or Tertiary SUD Problems 57% 82% 8% 4% 7% 6% 60% 5% 3% 7% 2% 25% 0%10%20%30%40%50%60%70%80%90%100% Alcohol Marijuana/Hash Cocaine/Crack Heroin/Opiates Meth/amphetamines Any Other TEDS (n=153,251) CSAT (n=7,226)

20. Other ASAM Issues (not in TEDS) 0.31 0.08 0.84 0.27 0.79 0.59 0.70 0.76 0.70 0.81 0.82 0.45 0% 10%20%30%40%50%60%70%80%90%100% Any withdrawal symptoms past week Severe withdrawal (11+ symptoms) Sexually active in past 90 days Major health problems Any co-occurring psychiatric Ever physical, sexual or emotional victimization Doesn't acknowledge AOD problem Doesn't acknowledges need for treatment Regular alcohol use in recovery environment Regular drug use in recovery environment Any violence or illegal activity Any past year violent crime

21. Outcome Measures Number of Inconsistencies –Count of 65 paired items consistently answered by over 90% of the clients, that are inconsistent Duration – time doing the interview (not including breaks) in Minutes Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2-misunderstanding, 3-denial, 4-misrepresentation Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting) Proportion of Missing Data on 99 Items used in the GAIN’s core 10 Change measures. Atypicalness a measure of endorsing high severity items without first endorsing the typical prior items (e.g.., suicide without depression) on the 123 Items of the GAIN’s 4 main psychopathology and psychopathy scales; This measures is based on the RASCH outfit statistic and reported in Logits Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales; This measures is based on the RASCH infit statistic and reported in Logits

22. Pre vs. Post Certification -0.39 -0.25 -0.24 -0.10 -0.04 -0.03 -0.6-0.4-0.20 Proportion of Inconsistencies (100%)* Duration (in Minutes)* Denial/Misrepresentation (Staff Rating)* Context Effect (Staff Report) Proportion of Missing Data (100%) Atypicalness (Outfit in Logits) Randomness (Infit in Logits) Cohen's D \a \a Cohen's d (Post Certification - Pre Certification)/Pooled STD * p<.05

23. Level of Certification -0.40-0.200.000.200.40 Cohen's f \a Pre-Certification Administrator Cert. Site Certified Local Trainer Cert. \a Cohen's f (Group Mean – Pooled Mean/Pooled STD * p<.05 Proportion of Inconsistencies (100%) * Duration (in Minutes) * Denial/Misrepresentation (Staff Rating) * Context Effect (Staff Report) * Proportion of Missing Data (100%) * Atypicalness (Outfit in Logits) Randomness (Infit in Logits) Certified Interviewers generally do better This one in the wrong direction

24. Staff Experience 0 50 100 150 200 250 300 350 400 450 500 0 10 203040506070 80 90 100 Number of Prior Interviews by Staff GAIN’s complete

25. Impact of Staff Experience Major improvement over the first 15 interviews Most improvements have occurred by 60 interviews

26. Discussion Completing certification improves interview: lower administration time, fewer inconsistencies No significant difference on Atypicalness and randomness, which depend more on individual severity factors of clients Contrary to common expectations, “trained and certified” clinicians did better than research assistants in terms of our efficiency, reliability and validity measures. Staff Experience is also clearly a factor in achieving better quality data

27. Qualitative information verifies value of program… Post-certification quotes from individuals initially resistant to going through process state that clinicians and researchers feel they do a better job interviewing after having gone through the training, QA, and certification program. For example…

28. Quotes… [My QA reviewer’s] “comments were clear and directive. She was supportive, tactful, and empowering. I am very impressed and feel quite positive about my experience…” [My QA reviewer] “was fantastic! I really wish I would have called him sooner in my administration certification process. He helped me feel more relaxed about the process by easing my fears and explaining things in a way that can only be described as supportive and understanding.”

29. Quotes… “As you know, mid way through my certification [my reviewer] was assigned another project, while that might have been an awkward time, it was not. In no way did I feel hampered by the change and I found [new reviewer] to be as wonderful to work with…the staff have been professional, knowledgeable, and very approachable. They have never seemed unwilling to help, and not only that, they are so positive…They could teach a great many people [in the World] what customer service is. I will look back on this experience with pleasure and good memories – not something I can say about a lot of procedures.”

30. Acknowledgement The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using data provided by CSAT adolescent treatment grantees under the Adolescent Residential Treatment (ART), Effective Adolescent Treatment (EAT), Strengthening Communities for Youth (SCY), Targeted Capacity Expansion (TCE), and Young Offender Re-entry Program (YORP) grants (TI013313, TI013309, TI013344, TI013354, TI013356, TI013305, TI013340, TI130022, TI03345, TI012208, TI013323, TI14376, TI14261, TI14189,TI14252, TI14315, TI14283, TI14267, TI14188, TI14103, TI14272, TI14090, TI14271, TI14355, TI14196, TI14214, TI14254, TI14311, TI15678, TI15670, TI15486, TI15511, TI15433, TI15479, TI15682, TI15483, TI15674, TI15467, TI15686, TI15481, TI15461, TI15475, TI15413, TI15562, TI15514, TI15672, TI15478, TI15447, TI15545, TI15671)). The author would like to thank Lexy Adkins, Rod Funk, Melissa Ives, Melissa Jerse, Sarah Knecht, & Mike Dennis for their help preparing the presentation. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org.