1. Presented by Gregory B. Teague, Ph.D. Matthew Johnsen, Ph.D. Joseph Rogers Bonnie Schell, M.A. (See additional credits at end) Research on Consumer-Operated Service Programs: Effectiveness Findings and Policy Implications of a Large Multi-Site Study Findings from the SAMHSA/CMHS Consumer-Operated Service Program Multisite Research Initiative

2. 2 - 1 - Consumer-Operated Services Multisite Research Initiative: Overview

3. 3 Consumer-Operated Services: An Invocation "Who then can so softly bind up the wound of another as he who has felt the same wound himself?” – Thomas Jefferson

4. 4 Consumer-Operated Services: Context Peer-delivered services for persons with serious mental illness have grown in number and diversity in recent years There is growing recognition of their value (cf. IOM Quality Chasm and New Freedom Commission reports) There is preliminary but limited evidence of their effectiveness in improving symptoms, promoting larger social networks, and enhancing quality of life

5. 5 Context (cont’d.) However, evidence has been limited to –uncontrolled studies, –demonstrations of feasibility, –& preliminary findings. Weak evidence-base has hindered –peer-run program credibility, –resource stability & sustainability, –& opportunities for integration into the continuum of community care.

6. 6 COSP Multisite Research Initiative: Design Overview Research question: –To what extent are consumer-operated programs effective as an adjunct to traditional mental health services in improving the outcomes of adults with serious mental illness? Experimental design: –Random assignment to one of two conditions –Traditional MH Services (TMHS) or TMHS+COSP Parallel cost study Consumer involvement at all levels

7. 7 COS Criteria for COSP-MRI “Administratively controlled and operated by consumers…planned, delivered, and evaluated by consumers themselves” In full operation for at least 2 years Does not include all types of peer-run and self-help services

8. 8 Design Overview (cont’d) Participants –Persons 18+ with diagnosable mental / behavioral / emotional disorder and functional impairment –N = 1827 enrolled in study Eight program sites –CA, CT, FL, IL, ME, MO, PA, TN Three general program models –Drop-In (4 sites) –Peer Support (2 sites) –Education/Advocacy (2 sites) Began in 1998

9. 9 Design Overview (cont’d) One-year longitudinal follow-up –4 measurement points: 0, 4, 8, 12 months Common interview protocol Outcome domains –Employment, Empowerment, Housing, Service Satisfaction, Social Inclusion, Symptoms, Well- being Conventional RCT approach –Intent-to-treat analysis –Optimized, common a priori hypothesis

10. 10 Primary Hypothesis Informed by presumed underlying consumer experience: “I am not alone” Theoretically justified by literature on well- being and research on peer support programs and consumer/survivor recovery The well-being construct was developed from the validated scales in the protocol which measured existential dimensions of participant experience: Recovery, empowerment, quality of life, social inclusion & acceptance, meaning of life, hope

11. 11 Primary Hypothesis “Participants offered both traditional and consumer-operated services would show greater improvement in well-being over time than participants offered only traditional mental health services.”

12. 12 - 2 - Program/Fidelity Measure: The “Consumer Operated Services Program – Fidelity Assessment Common Ingredients Tool” (COSP-FACIT) http://www.umassmed.edu/entities/cmhsr/uploads/FACIT.pdf

13. 13 Initial COSP Measurement Context Common measurement at participant level Potentially important aspects of intervention not measured in common protocol COS not operationalized; no existing measures Diversity among programs Contrast between experimental and control conditions not specified

14. 14 Development of the FACIT Involvement of COS directors and staff as well as researchers at all stages Identification and definition of common ingredients of consumer-operated services –Involvement of the Consumer Advisory Panel Identification/selection of feasible indicators Specification of performance anchors (typically 4-5) for each indicator

15. 15 Hypothesized Common Ingredients of Consumer-Operated Services Structure –Consumer operated –Participant responsive –Links to other supports Environment –Accessibility –Safety –Informal setting –Reasonable accommodation Belief Systems –Peer principle –Helpers principle –Empowerment –Choice –Recovery –Acceptance and Respect for diversity –Spiritual growth

16. 16 Hypothesized Common Ingredients of Consumer-Operated Services Peer Support –Peer support –Telling our stories –Consciousness-raising –Crisis prevention –Peer mentoring and teaching Education –Self-management / problem-solving –Education Advocacy –Self-advocacy –Peer advocacy –Systems advocacy

17. 17 Sample Element: 1.1 Structure - Consumer Operated 1.1.1 Board Participation - Consumers constitute the majority on the board or the group which decides all policies and procedures. 1.1.2 Consumer Staff - With limited exception, staff consists of consumers who are hired by and operate the program. 1.1.3 Hiring Decisions - Consumers have control over hiring decisions. 1.1.4 Budget Control - Consumers have control of the operating budget. 1.1.5 Volunteer Opportunities - Role opportunities for participants may include board and leadership positions, volunteer jobs, and paid staff positions.

18. 18 Sample Anchors: 1.1.1 Board Participation 1No member of the board is self-identified as a consumer 2 1-50% of the board are self-identified as consumers 3 51% or more of the board are self-identified as consumers but less than 51% of the officers are self-identified as consumers 4 51% or more of the board are self-identified as consumers and more than 51% of the officers are self identified as consumers 5 90-100% of the board are self-identified as consumers and all of the officers are self-identified as consumers

19. 19 Application of the FACIT Data collection –Two rounds of site visits – years 2 and 4 –Interviews with program directors, staff, & recipients in both COS and TMHS –Independent ratings by site visitors Conciliation –Raters established agreement following any initial disagreement Pilot testing (Round 1) –Evaluated feasibility and inter-rater reliability, modified for round 2

20. 20 FACIT: Psychometric Analysis Factor and internal consistency analyses within major domains Conservative elimination of weak or contrary variables Maximal retention of original variables to optimize content validity Delineation of provisional scales for use in fidelity-outcome analyses –Partial overlap of hypothesized and empirically- defined constructs

21. 21 FACIT Research Questions Is the FACIT reliable and valid as a measurement tool? Are the Common Ingredients in fact common? Does the FACIT distinguish between consumer-operated and traditional services? Does the FACIT distinguish among models of consumer-operated services?

22. 22 FACIT: Inter-Rater Reliability Inter-rater reliability, assessed in Round 1 –Average across all items.70 –Average, all items, COSP.72 –Average, all items, TMHS.67 –Overall score.97 Inter-rater reliability, refined measure –Average of all items retained.78 Acceptable inter-rater reliability at item level Excellent inter-rater reliability for total scale

23. 23 FACIT Subscales: Internal Consistency (1) (N = 16) Scale/SubscaleCr. Alpha# items STRUCTURE.939 7 –Consumer Ownership.9815 –Responsiveness.8112 ENVIRONMENT.738 7 –Inclusion.7515 –Accessibility.8992 BELIEF SYSTEMS.738 8 –Peer Ideology.8024 –Choice & Respect.6862 –Spirituality & Accountability.6832

24. 24 FACIT Subscales: Internal Consistency (2) (N = 16) Scale/SubscaleCr. Alpha# items PEER SUPPORT.885 6 –Encouragement.9293 –Self-Expression.8153 –Self-Expression2.7602 EDUCATION.888 5 ADVOCACY.893 3 TOTAL.963 36

25. 25 Overall FACIT Scores: Consumer- Operated and Traditional Services

26. 26 Generally high performance on most dimensions –COS organizations received 75% of all possible points –COS organizations received 82% of possible points on four consumer-defined organizational process scales (Factor 1 of 2-factor solution: 52% out of total 80% explained) Some variability across subscales and sites Room to increase – no problem with ceiling effects Presence of Common Ingredients in COSP

27. 27 All overall FACIT scores for COSP were higher than the score for any TMHS (p =.004) Mean overall scores –COSP76%(68% - 83%) –TMHS42%(25% - 54%) Mean COSP scores were higher than TMHS on all main subscales (t-test p-values:.000 -.030) –Greatest difference on Structure –Least differences on Belief Systems, Peer Support, and Education COS programs scored higher than their respective TMHS on most subscales Consumer-Operated vs. Traditional Services

28. 28 Variability in overall scores across sites (68% - 83%) Some variation in overall scores by model type –Consumer Run Drop In Centers74% –Peer Support Programs75% –Education & Advocacy Programs82% Greater variation in subscales across model types and sites Differences and Similarities Among COS Models

29. 29 Variations Among Service Types Within Domains

30. 30 All models show high ratings on Belief Systems Education and Advocacy programs appear higher on Education and Advocacy domains Programs that more explicitly include peer support components appear higher on Peer Support More formally structured programs appear higher on Structure and Environment Differences and Similarities Among COS Models

31. 31 FACIT: Conclusions The FACIT measures salient features of a wide range of consumer-operated service models The models included in the COSP study were consistent with general specifications for the common ingredients and were significantly more so than control programs Generalizability is not yet known, but there is positive evidence for acceptability, feasibility, reliability, validity, sensitivity of the FACIT Are there indications that these common ingredients also active ingredients?

32. 32 - 3 - COSP-MRI Outcome Measurement & Findings

33. 33 Creation of Well-being Measure Rationale –to develop a measure that was supported by theory –measure hypothesized to be most sensitive to primary program effect Started with 14 potential scale components Factor analysis: –1 factor much more important than others –led to selection of 8 scales, chosen by theory and with loading of 0.5 or greater

34. 34 Final Well-Being Components Total Herth Hope Index (Herth, 1991) Quality of Life Scale (QOL Interview excerpts, Lehman, 1983) Meaning of Life Framework Subscale (Life Regard Index, Battista and Almond, 1973) Subjective Social Inclusion Scale (QOL Interview excerpts, Lehman, 1983) Empowerment / Making Decisions Scale (Rogers et al., 1997) Personal Empowerment Scale (Segal et al., 1995) Recovery Assessment Scale (Corrigan et al., 1999) Social Acceptance Scale (Well-Being Project, Campbell and Schraiber, 1989)

35. 35 Computation of Well-Being Measure Standardize each scale on mean / SD computed over all time points Reliability: Cronbach’s alpha of 0.88 Validity: correlations of included scales with –Symptoms (-0.4 to -0.5) –Excluded scales (-0.5 to 0.4) Relationship with prior use of COS –Significantly related to any previous use, but not to use in the 4 months prior to baseline

36. 36 Intent-to-treat (ITT) Analysis: Methodology Multi-level modeling, using SAS (PROC MIXED) –treating participant as fixed effect –incorporating auto-regressive covariance structure –time treated as linear (values of 0, 4, 8, and 12) Number of participants included in analysis = 1622 Pooled test of primary hypothesis –as per randomized clinical trial model –followed by examination of patterns within sites and clusters of sites

37. 37 ITT Analysis: Primary Hypothesis Model estimated (Cluster as 2 nd level): Well-being = Time + Group + Cluster + Participant + Time*Group + Cluster*Time + Cluster*Group + Cluster*Time*Group Time effect highly significant (p <.0001): overall increase in Well-Being for study participants Time*Group interaction (tests primary hypothesis) not significant (p = 0.23), but Cluster*Time*Group interaction significant (p = 0.0066)

38. 38 ITT Analysis: Well-being Trend Over Time

39. 39 ITT Analysis: Findings by Cluster Primary hypothesis of study-wide effect on well- being not so far supported, but Time*Group interaction not interpretable Examined results by cluster –Drop-in Cluster: Time*Group interaction significant (p=.0017), and COSP group improved more –Education / Advocacy Cluster: Time*Group interaction significant (p=.0188), but COSP group improved somewhat less –Peer Support Cluster: Time*Group interaction not significant

40. 40 ITT Analysis: Well-being Trend Within Clusters

41. 41 ITT Analysis: Site-level Findings Site*Time*Group interaction not significant at p=.05 in any cluster model, but some site- level results differed Site-level findings: –Drop-In: significant Time*Group interaction, associated with significantly greater COSP improvement in 2 sites –Education/ Advocacy: marginally significant Time*Group interaction, associated with slightly less COSP improvement in one site –Peer Support: no significant Time*Group interactions

42. 42 Effect Sizes (Intent-To-Treat) Time effect is significant overall and at majority of sites Positive effects are small to moderate Crossover (including low engagement) suggests that some effects may be understated Cautions about possible sources of site variability ClusterESSiteES Drop-In.391.59 2-.06 3.52 4.43 Peer Supp.055.00 7.06 Edu/Adv-.126-.57 8-.12

43. 43 As-Treated Analyses: Approach As-treated (AT) analyses must address potential for selection effects Significant crossover confirms likelihood of selection effects in this study Use of propensity scores to improve equivalence of comparison groups Engagement / utilization examined as presence-absence and by amount Power reduced at site level, but findings examined at site level for verification of pattern found at overall level

44. 44 As-Treated Analyses: Findings Analyses using middle “propensity” group –Participants who were not likely predisposed either for or against use of COS Examined overall effect of minimal engagement (any use vs. no use) Significant Time*Group interaction, p =.042 Greater increase in Well-Being among those who used COS, effect size =.302 Pattern generally observed across sites, with varying strength

45. 45 Well-Being Over Time by Use of COS (Any Use vs. No Use)

46. 46 Mean Change in Well-Being by COS Use (Any Use vs. No Use, Site Level)

47. 47 As-Treated Analyses: Findings (cont’d.) Examined overall effect of level of use (High use vs. Low use vs. No use) Significant group effect, p =.017 Greatest increase in Well-Being among those who used COS more Pattern generally observed across sites, with varying strength

48. 48 Well-Being over Time by COS Use (High vs. Low vs. No Use)

49. 49 Net Change in Well-Being by COS Use (High vs. Low vs. No Use)

50. 50 Further Investigations Pending analyses –Other outcomes –Participant characteristics Current additional analyses using fidelity data –Impact of program elements on intervention effect (experimental framework) –Relationship of program elements to program effects (observational framework) –Sensitivity testing for spurious revelation or obscuration

51. * * * FIDELITY AND SITE EFFECT (ITT) Correlations: FACIT Scales with Difference in Change in Well-being (N = 8; Ranges = sensitivity tests) * p <.05

52. 52 Environment Subscale: “Inclusion” Variables Cost – services free of charge Program rules – ensure physical safety, developed by consumers Social environment – no hierarchy; sense of freedom and self-expression; warmth among participants and staff Sense of community – fellowship, mutual caring, and belonging Lack of coerciveness – choice, no threats or unwanted treatment; tolerance of harmless behavior

53. 53 Peer Support Subscale: “Self-Expression” Variables Artistic Expression – opportunities for telling one’s story in visual arts, music, poetry * Formal peer support – structured groups for listening, empathy, compassion based on common experience Telling our stories – opportunities for sharing life experiences * * Included in Self-Expression 2

54. * * * * * * FIDELITY AND SITE EFFECT (ITT) Correlations: Difference in FACIT Scales with Difference in Well-being Slope (N = 7: One site omitted – outlier on FACIT difference pattern) * p <.05

55. FIDELITY AND PROGRAM EFFECT (ITT) Partial Correlations: Adjusted FACIT Scales with Change in Well-being, controlling for Condition (N = 16; Ranges = sensitivity tests) * p <.05 * *

56. * * * * * * * * * * FIDELITY AND PROGRAM EFFECT (ITT) Partial Correlations: Adjusted FACIT Scales with Change in Well-being, controlling for Condition (N = 14: Two programs [1 COSP, 1 TMHS] with negative WB change omitted) * p <.05

57. 57 Effectiveness Findings: Summary Crossover: low engagement in COS programs Significant time effect: overall increase in well- being among study participants Significant small effect of COS intervention in a subset of sites (ITT) More general positive effect when actual participation is taken into account (AT); greater use associated with greater increase in well- being Strong relationship between increase in well- being and recovery-oriented program features Findings not limited to one program model

58. 58 - 4 - Summary & Conclusions

59. 59 Summary & Conclusions: Study Context Consumer-driven health care – underlying rationale for the COSP study –History of advocacy, theory, and critique; cf. IOM Quality Chasm and New Freedom Commission reports –Needed: expansion of the evidence base Challenging design –Variable, loosely specified interventions –RCT vs. volunteer culture: experimental intervention as adjunct; “cold calls” in many sites –Substantial crossover within modestly powered study Innovative structure and process –Integration of multiple perspectives –Impact of COSP-MRI on individual COS sites

60. 60 Impact on Service Sites Infusion of resources Some change in service population Data collection –Services –Costs Attention to fidelity Identity as programs –Internal –External

61. 61 Summary & Conclusions: Study Limitations Sample of sites –Generalizability not established –Small N, especially within cluster Novel measures –Primary outcome variable (synthetic Well-being) –Intervention/fidelity measure (FACIT) Theoretical context –Findings are embedded in recovery theory –Relationship to other outcome domains to be determined Analysis still under way…

62. 62 Summary & Conclusions: Methodological Contributions Fidelity & model measurement –New instrument for consumer-operated services and critical service features –Importance of measuring interventions for understanding outcome results Multisite studies –Natural variation can reveal phenomena that might be less visible within more uniform site samples

63. 63 Summary & Conclusions: Substantive Findings Evidence base for COS as discrete programs –Adding COS to traditional services adds incrementally to well-being Evidence base for recovery theory –Program features specified for and found in COS are related to increases in well-being independent of setting COSP as a business –Variability in unit cost: data & study impact –Efficiency and management practices matter

64. 64 Summary & Conclusions: Policy Implications Stronger basis for commitment to recovery– oriented services –Particular consumer-supported program features contribute to recovery and are effective within various settings Further support for bringing COS into the mental health service fold –COS should be recognized and included as such within the service continuum –COS should have appropriate fiscal and organizational support

65. 65 Contributors to Presentation Steven Banks Crystal R. Blyler Jean Campbell Sally Clay Carolyn Lichtenstein Betsy McDonel Herr Mark Salzer Vijaya Sampath Joseph Sonnefeld Kristin Zempolich

66. 66 COSP-MRI Investigators & Scientific Contributors* Steven Banks Crystal R. Blyler Jean Campbell Patrick W. Corrigan Dianne C. Côté Nancy Erwin Susan Essock Matthew Johnsen Betsy McDonel Herr * Partial list Carolyn Lichtenstein Jeffrey G. Noel Ruth O. Ralph E. Sally Rogers Joseph Rogers Mark S. Salzer L. Joseph Sonnefeld Tom Summerfelt Gregory B. Teague Brian Yates

67. 67 Consumer Advisory Panel Jean Campbell, Sally Clay, Dianne Cote, Zahira Duvall, Janine Elkanich, Nancy Erwin, Louetta Hix, Lorraine Keck, Kathryn Kidder, Barbara Lee, Terrance Means, Helen Minth, Carol Mussey, Joanne O'Connor, Jean Risman, Joseph Rogers, Yvette Sangster, Bonnie Schell, James Scott, Florence Schroeter, Paula Stockdale, Greg Warren, Michael E. Weiss, Jeanie Whitecraft

68. 68 Common Ingredients Subcommittee Crystal Blyler, Jean Campbell, Sally Clay, Patrick Corrigan, Dianne Cote, Sita Diehl, Zahira Duvall, Janine Elkanich, Susan Essock, Steve Fischer, Betsy McDonel Herr, Louetta Hix, Karen Hirsch, Matthew Johnsen, Anika Keens-Douglas, David Lambert, Sara Lickey, Brian McCorkle, Carol Mussey, Joanne O’Connor, Dena Plemmons, Ruth Ralph, Magda Randolph, Jean Risman, Yvette Sangster, Carole Silverman, Joe Sonnefeld, Tom Summerfelt, Sharon Togut, Greg Teague, Eileen Zeller