1. Mixed Methods Evaluation of the Massachusetts Peer Specialist Training
Linda Cabral
University of Massachusetts Medical School
Center for Health Policy and Research
American Evaluation Conference, November 2010
San Antonio, TX

2. Center for Health Policy and Research/Commonwealth Medicine, UMMS
Evaluation Team
Center for Health Policy and Research/Commonwealth Medicine, UMMS
Heather Strother
Linda Cabral
Kathy Muhr
Laura Sefton
Acknowledgements
Peer Specialist Training Director, The Transformation Center, Boston, MA
Lyn Legere

3. Presentation Overview
Background on program evaluated
Evaluation aim and questions
Evaluation design and methods
Selected evaluation results
Discussion
Questions

4. Peer Specialist Training Background
Peer specialists are current or former recipients of mental health services who work with their peers to help them achieve their recovery goals.
New professional role in Massachusetts’ mental health system.
Recently awarded contracts by state’s mental health authority require mental health agencies to employ peer specialists.
A peer-operated community organization, the Transformation Center, runs a statewide training to prepare peer specialists for this role. 4

5. Peer Specialist Training Background
9-week training comprised of one full day of orientation, a weekend retreat with day classes and night activities, and six days of courses scheduled on a weekly basis.
34 training modules focus on building the competencies and skills of participants to work in peer specialist roles.
Four trainings conducted annually since August 2006 with approximately 25 participants per training.

6. Peer Specialist Training Background
Training objectives include an understanding of the:
peer specialist job and the skills to do that job;
recovery process and how to use one’s own recovery story to help others;
ability to establish healing relationships; and
importance of self-care
Training participants take a certification exam after completing curriculum to become certified peer specialists

7. Evaluation Aims The aims of the evaluation were to:
Assess what is working well and whether there are opportunities to enhance the training program for both the training participants and the trainers.
Determine the degree to which the training and certification program is establishing a competent workforce of certified peer specialists throughout the mental health system in Massachusetts.

8. Evaluation Questions
What is working well and what improvements can be made with regard to the peer specialist training and certification program?
What personal changes occur for training participants as a result of their participation in the training and the certification exam?
What employment changes occur for training participants (both certified and non-certified) as a result of their participation in the program?
What are the experiences of peer specialists currently working in the state’s mental health system?

9. Evaluation Design
Explanatory sequential mixed methods design used for evaluation
Quantitative
Data and Results
Qualitative
Data and Results
Interpretation
Followed by
Led to

10. Rationale for Mixed Methods Design
To better understand how the training works and how participants experience the training (qualitative)
To measure participant’s changes in personal recovery (quantitative)
To understand participant’s changes in recovery (qualitative)
To measure employment changes (quantitative)
To understand why participants do or do not become employed in peer specialist roles (qualitative)

11. Evaluation Participants
143 participants from six training classes were included in the evaluation (March 2008-May 2009)
2/3 were female, 1/3 male
90% were Caucasian, 6% African American, and 4% Latino
At training entry, 42% of participants were already working in peer specialist positions, 8% were working in non-peer position in mental health field, and 12% were in a non mental health related job
38% of participants were not working at training entry

12. Quantitative Data Collection
A training participant survey was administered at three points in time
First day of training (Time 1)
Last day of training (Time 2)
Six months after completing the training (Time 3)
Survey included closed response questions about participants employment and volunteering status, and their mental health recovery using the Mental Health Recovery Measure (MHRM)

13. Qualitative Data Collection
41 in-person interviews were conducted at Time 3 with a purposive sample of training participants
Participants were sampled by training class, gender, race, work status and job type at training entry, and certification exam results
In addition, results on the MHRM were used to sample participants that showed significant positive and negative changes in their feelings of personal recovery from Time 1 to Time 2.

14. Results: Training Strengths
Enhances participant understanding of the recovery process and how to use their recovery story
I’ve shared my story before this training, but I learned what to focus on and
how to make it relevant for the person/persons I am telling it to. I learned
what to share, to share what’s relevant, not absolutely everything.
Helped participants form strong bonds with each other
Trainers were praised for their hard work and commitment, supportive and nurturing approach, skillful use of their recovery stories to teach and inspire, and high energy and engaging teaching styles.

15. Results: Opportunities for Improvement
Increase the diversity of training participants
Increase emphasis on the reality and challenges of working as a peer specialist, including topics such as:
Working in negative program environments
Navigating boundary issues related to role
Taking care of oneself in role
Working with people in early stages of recovery
Fairly low passing rate for certification exam (50%)

16. Results: Personal Changes (1)
MHRM Group N
T1 mean
T2 mean T3
mean
All MHRM scores 85
90.3
94.0
93.7
MHRM score average or below average (<80) 22
70.6
81.1
82.1
MHRM score above average (>80) 63
97.2
98.4
97.7

17. Results: Personal Changes (2)
Overall personal growth
Validation of one’s own recovery journey
It made me feel, (and I don’t like to say this because my life experience has sucked, it sucks to go through what I’ve been through) that there is some worth out of what I have been through, that I can do something with this lived experience.
Connecting with a community of peers
Improved interpersonal skills
Personal empowerment in relationships and treatment settings

18. Results: Employment Changes (1)
Of those with completed survey employment data (n=89), more than half reported a positive change in their employment related to peer specialist work.
Improved skills for peer specialist work included
Active listening
Sharing a recovery story strategically
Facilitating various types of peer group,
Working with people at different stages of recovery
Helping peers develop their own goals for recovery

19. Results: Employment Changes (2)
Participants shifted their approach from solving their peers’ problems to supporting peers to solve their own problems and make their own decisions
Going to the training, I had a lot of questions about my role, and it
(the training) changed my perspective, answering questions I had
about the role. My job is not to fix people, I am not a clinician but in a
role to help people, provide feedback and encourage.

20. Conclusions
Mixed method evaluation design provided opportunity to quantify the personal and vocational impact of training on participants and to also explore the meaning of these impacts on participants
Quantitative results provide a quick snapshot of the training’s impact on participants
Qualitative results have been quite useful to the Peer Specialist training team, as they have informed them on where/how to make improvements to the training
State agency support for a Phase II

21. Contact Information
Linda M. Cabral, M.M.
Senior Project Director, Research & Evaluation Unit
Center for Health Policy and Research
University of Massachusetts Medical School
333 South Street
Shrewsbury, MA 01545