1. Adult Consumer Assessments of Care in New York Chip Felton, Senior Deputy Commissioner Jeff Kirk Doug Dornan New York State Office of Mental Health Center for Information Technology and Evaluation Research

2. Overview Development of the NYSOMH Mental Health Services Survey (MHSS) and Brief Quality of Life Assessment (BQLA) Development of the NYSOMH Mental Health Services Survey (MHSS) and Brief Quality of Life Assessment (BQLA) Annual Adult Consumer Survey Project Annual Adult Consumer Survey Project Recovery Oriented Systems Indicators (ROSI) Project Recovery Oriented Systems Indicators (ROSI) Project

3. Development of the NYSOMH Mental Health Services Survey (MHSS) Version 1 (1996-98): Version 1 (1996-98): MHSIP-informed items, response choices and domains (access, approp/support for recovery, outcomes/perceived efficacy) + BASIS-32 + Rosenberg SE + SF12 + self-help involvement = big mail survey (150 items!) MHSIP-informed items, response choices and domains (access, approp/support for recovery, outcomes/perceived efficacy) + BASIS-32 + Rosenberg SE + SF12 + self-help involvement = big mail survey (150 items!) Administered 3x (6 month intervals) to over 6,000 adults receiving state-operated outpatient services; response rate 47%. Administered 3x (6 month intervals) to over 6,000 adults receiving state-operated outpatient services; response rate 47%. Developed web-based report card for management and print report card for recipients Developed web-based report card for management and print report card for recipients Version 2 (1999-present) Version 2 (1999-present) Shortened instrument by limiting content to assessment of care domains (access, appropriateness, perceived efficacy) Shortened instrument by limiting content to assessment of care domains (access, appropriateness, perceived efficacy) Adopted neutral wording for items and 4-point poor - excellent scale in effort to eliminate or reduce “satisfaction effect” (ceiling effect) Adopted neutral wording for items and 4-point poor - excellent scale in effort to eliminate or reduce “satisfaction effect” (ceiling effect) Developed in anticipation of need to monitor Medicaid managed care plans (but plans ultimately never implemented) Developed in anticipation of need to monitor Medicaid managed care plans (but plans ultimately never implemented) Several administrations: Several administrations: 1) 2001-2002, NYC area state-operated outpatient services (approx 3,000) (methods: online, peer specialist guided) 1) 2001-2002, NYC area state-operated outpatient services (approx 3,000) (methods: online, peer specialist guided) 2) 2003-2004, statewide, 5-county sample, group sessions (approx 300/year) 2) 2003-2004, statewide, 5-county sample, group sessions (approx 300/year) Succeeded in getting lower ratings ! :-) Succeeded in getting lower ratings ! :-)

4. Item Construction: Access Examples MHSIP (and MHSS version 1) MHSIP (and MHSS version 1) “Staff were willing to see me as often as I felt it was necessary” “Staff were willing to see me as often as I felt it was necessary” “I do better in school and or work” “I do better in school and or work” MHSS – “How would you rate…” MHSS – “How would you rate…” “The ease of getting services when you needed them” “The ease of getting services when you needed them” “Helpfulness of services on your involvement in work or school” “Helpfulness of services on your involvement in work or school”

5. Development of the NYSOMH Brief Quality of Life Assessment (BQLA) Rationale for development of a brief QOL instrument: Rationale for development of a brief QOL instrument: QOL an important outcome and social indicator domain QOL an important outcome and social indicator domain Existing measures not very parsimonious, hence not suitable for routine large scale surveillance, yet much content never used (e.g. Lehman’s QOL interview: 1 hour, face-face, “objective” QOL and “subjective” satisfaction-like QOL in multiple life domains; published studies rarely reported data other than “subjective” item means; “subjective” items highly inter- correlated) Existing measures not very parsimonious, hence not suitable for routine large scale surveillance, yet much content never used (e.g. Lehman’s QOL interview: 1 hour, face-face, “objective” QOL and “subjective” satisfaction-like QOL in multiple life domains; published studies rarely reported data other than “subjective” item means; “subjective” items highly inter- correlated) Combination of experience using and analyzing results from longer QOL assessments, analyses of item correlations indicating much redundancy, desire for very brief measure yielded global indicator approach to QOL assessment. Combination of experience using and analyzing results from longer QOL assessments, analyses of item correlations indicating much redundancy, desire for very brief measure yielded global indicator approach to QOL assessment. BQLA developed in late 1990’s; 1 global rating item per life domain; 4-point poor-excellent scale; 10-15 items BQLA developed in late 1990’s; 1 global rating item per life domain; 4-point poor-excellent scale; 10-15 items BQLA now administered along with the MHSS BQLA now administered along with the MHSS

6. Example Items Brief Quality of Life Assessment “Overall, how would you rate…” “Overall, how would you rate…” “Your involvement in work, employment” “Your involvement in work, employment” “How you feel about yourself, your self- esteem” “How you feel about yourself, your self- esteem”

7. Annual Adult Consumer Survey Project Need to conduct an annual consumer assessment of care survey Need to conduct an annual consumer assessment of care survey Current methodology to survey consumers across New York State developed in 2003 Current methodology to survey consumers across New York State developed in 2003

8. Survey Methodology Stratified random sample of counties (2 counties in each of 4 regions) Stratified random sample of counties (2 counties in each of 4 regions) Partnered with local peer groups to conduct surveys Partnered with local peer groups to conduct surveys Trained peer advocates in survey administration Trained peer advocates in survey administration Local peer organizations assisted in recruiting participants and setting up evaluation meetings Local peer organizations assisted in recruiting participants and setting up evaluation meetings

9. Survey Instruments (2004 Survey) Mental Health Services Survey (MHSS) Mental Health Services Survey (MHSS) 32 fixed items & 3 open-ended response item 32 fixed items & 3 open-ended response item MHSIP MHSIP 28 fixed items & 1 open-ended response item 28 fixed items & 1 open-ended response item Quality of Life Assessment (QOLA) Quality of Life Assessment (QOLA) 15 fixed items & 1 open-ended response item 15 fixed items & 1 open-ended response item Recovery Oriented Systems Indicators (ROSI) Recovery Oriented Systems Indicators (ROSI) 42 fixed items & 1 open-ended item 42 fixed items & 1 open-ended item

10. Comparison Between MHSS and MHSIP Comparison of Content Comparison of Content Mean and Standard Deviation of Subscales Consumer Responses Mean and Standard Deviation of Subscales Consumer Responses Percent Positive Responses Percent Positive Responses Cross tabs Cross tabs Cronbach’s Alpha on Subscales Cronbach’s Alpha on Subscales Factor Analysis Factor Analysis Correlation with QOLA and ROSI Correlation with QOLA and ROSI

11. Comparison of Content: MHSS and MHSIP MHSS MHSIP (v1.1) 28- items Domains Access (10 items) Access (10 items) Appropriateness (12) Appropriateness (12) Outcomes (8) Outcomes (8) Overall Assessment (2) Overall Assessment (2) Access (6 items) Access (6 items) Appropriateness (11) Appropriateness (11) Outcomes (8) Outcomes (8) General Satisfaction (3) General Satisfaction (3) Response Scale 1 = Poor 2 = Fair 3 = Good and 4 = Excellent 9 = Does not apply 1 = Strongly Agree 2 = Agree 3 = Neutral 4 = Disagree 5 = Strongly Disagree 9 = not applicable

12. MHSSMHSIP MeanSTDMeanSTD Access3.60.8623.71.847 Appropriateness3.72.8793.81.758 Outcomes3.59.8903.77.816 Overall Assessment/ Satisfaction 3.851.023.89.963 Means and Standard Deviations for Subscales: MHSS and MHSIP (Adjusted Scores for 1-5 scale, N = 651)

13. Percent Positive Responses: MHSS and MHSIP (Adjusted Scores for 1-5 scale, N=651) MHSSMHSIP Access60%68% Appropriateness64%72% Outcomes55%70% Overall Assessment/ Satisfaction76%74%

14. Cross tabs: MHSS and MHSIP Adjusted scores for 1-5 scale, ≥3.5=(+); <3.5=(-) Comparison of Responses Sub-Scale +/+-/- % Agree + MHSS / - MHSIP - MHSS / + MHSIP Access 327 (51%)144 (22%) 73% 63 (10%)113 (17%) Appropriate 369 (57%)131(20%) 77% 45 (7%)100 (16%) Outcomes 310 (48%)143 (22%)70%48 (7%)144 (23%) Overall Assessment 420 (66%)98 (15%)81%68 (11%)53 (8%)

15. MHSSMHSIP Access.898.852 Appropriateness.926.908 Outcomes.888.907 Overall Assessment/ Satisfaction.795.883 * One item was added to the 2004 MHSS survey to assess consumer inclusion in treatment planning Cronbach’s alpha: MHSS and MHSIP Subscales (2003-2004 data, N = 651)

16. MHSS Factor Analysis (1 factor)

17. MHSIP Factor Analysis (1 Factor)

18. Pearson Correlations of Scales MHSSMHSIPQOLA MHSS--- MHSIP.75**(n=650)--- QOLA.53**(n=387).50**(n=386)--- ROSI.72**(n=388).77**(n=387).48**(n=387) **p<.01

19. Conclusions & Next Steps MHSS performs equally or better than the MHSIP (v.1.1) MHSS performs equally or better than the MHSIP (v.1.1) Scales have comparable variability but MHSS has greater variability Scales have comparable variability but MHSS has greater variability The scales have similar consumer ratings, but MHSS may be less likely to illicit positive responses The scales have similar consumer ratings, but MHSS may be less likely to illicit positive responses Both have good internal consistency Both have good internal consistency Both scales measure a single construct, but outcomes may be considered a distinct factor Both scales measure a single construct, but outcomes may be considered a distinct factor Both are correlated with the QOLA and the ROSI Both are correlated with the QOLA and the ROSI Re-evaluate MHSS against MHSIP 2.0 Re-evaluate MHSS against MHSIP 2.0 Implement web-based version of MHSS in state- operated facilities (pilot scheduled for May 2005) Implement web-based version of MHSS in state- operated facilities (pilot scheduled for May 2005)

20. Recovery Oriented Systems Indicators (ROSI) Project

21. Mental Health Recovery: What Helps & What Hinders A means to assess the recovery orientation of state and local mental health systems A means to assess the recovery orientation of state and local mental health systems 10 focus groups in nine states 10 focus groups in nine states 1000 pages of transcripts qualitatively analyzed, coded, and reduced to a 42 item self-report survey and a 30-item administrative profile, which together combine to make the ROSI 1000 pages of transcripts qualitatively analyzed, coded, and reduced to a 42 item self-report survey and a 30-item administrative profile, which together combine to make the ROSI

22. Mental Health Recovery: What Helps & What Hinders Survey and administrative profile are consumer-driven and recovery focused Survey and administrative profile are consumer-driven and recovery focused ROSI is compatible with either MHSS or MHSIP (v1.1) ROSI is compatible with either MHSS or MHSIP (v1.1) Correlated with MHSS and MHSIP Correlated with MHSS and MHSIP Not designed to replace, but to augment MHSS or MHSIP Not designed to replace, but to augment MHSS or MHSIP

23. Mental Health Recovery: What Helps & What Hinders Factor analyses of the prototype test led to 8 dimensions in assessing recovery orientation (N = 219 in 7 states) Factor analyses of the prototype test led to 8 dimensions in assessing recovery orientation (N = 219 in 7 states) Person-centered decision making & choice Person-centered decision making & choice Self-care & wellness Self-care & wellness Basic life resources Basic life resources Invalidated personhood Invalidated personhood Access Access Meaningful activities & roles Meaningful activities & roles Staff treatment knowledge Staff treatment knowledge Peer advocacy Peer advocacy

24. Mental Health Recovery: What Helps & What Hinders Selected results form preliminary analysis of the ROSI NYS Self-Report Pilot: Selected results form preliminary analysis of the ROSI NYS Self-Report Pilot: “Staff use pressure, threats, or force in my treatment”; 76% responded ‘often, almost always, or always’ “Staff use pressure, threats, or force in my treatment”; 76% responded ‘often, almost always, or always’ “Mental health staff interfere with my personal relationship”; 73% responded ‘often, almost always, or always’ “Mental health staff interfere with my personal relationship”; 73% responded ‘often, almost always, or always’

25. Mental Health Recovery: What Helps & What Hinders Selected results form preliminary analysis of the ROSI NYS Self-Report Pilot (cont.): Selected results form preliminary analysis of the ROSI NYS Self-Report Pilot (cont.): “Staff see more as an equal partner in my treatment program”; 56% agreed ‘often, almost always, or always’ “Staff see more as an equal partner in my treatment program”; 56% agreed ‘often, almost always, or always’ “Mental health staff help me build on my strengths”; 59% agreed ‘often, almost always, or always’ “Mental health staff help me build on my strengths”; 59% agreed ‘often, almost always, or always’ “Staff see me as a whole person”; 80% ‘agreed or strongly agreed’ “Staff see me as a whole person”; 80% ‘agreed or strongly agreed’

26. Mental Health Recovery: What Helps & What Hinders Next Steps Next Steps National dissemination and pilot of the ROSI National dissemination and pilot of the ROSI