Presentation is loading. Please wait.

Presentation is loading. Please wait.

Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment Kim T. Mueser Dartmouth Psychiatric Research Center.

Similar presentations

Presentation on theme: "Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment Kim T. Mueser Dartmouth Psychiatric Research Center."— Presentation transcript:

1 Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment Kim T. Mueser Dartmouth Psychiatric Research Center

2 Gary Bond: Man Without a Planet or Brother from Another Planet? Innovative thinker Math major Beloved by students & colleagues alike Unflappable Force to be reckoned with in psychiatric rehabilitation Honest, direct, modest, sensitive, humorous Clearly not of this world, but is he without a planet or…



5 Fidelity: Adherence & Competence Adherence The extent to which the defining characteristics of a standardized intervention are implemented in practice Competence The skillfulness & effectiveness with which a standardized intervention is implemented

6 Potential Data Elements of Fidelity Assessments Record reviews Clinician/administrator interviews Consumer interviews Knowledge tests Role play tests Direct observation Audio/videotape observation

7 Methods for Assessing Fidelity Depend on: Definition of the “treatment provider”: Individual Team Agency Defining elements of practice: Organization & staffing (e.g., ACT team, voc unit) Structure, aim, & scope of services (e.g., rapid job search, services provided in community) Clinician:consumer interactions (e.g., structure, content, & skill of clinician working with consumer)

8 Practices That Have Emphasized Organization & Structure Assertive Community Treatment: 1:10 staff:client ratio Shared caseloads Daily team meetings Services provided in community Direct, not brokered, service provision Supported Employment: Zero exclusion Focus on competitive jobs Rapid job search Follow-along supports Consumer preferences

9 Practices That Have Emphasized Clinician:Client Interactions Integrated Dual Disorder Treatment, Illness Management & Recovery: Psychoeducational skills Motivational interviewing Cognitive-behavioral teaching skills

10 Relevance of Clinical Skills to All Evidence-Based Practices Importance of the relationship between clinician & consumer Utility of specific skills in changing behavior or achieving specific objectives Differences between clinicians in their effectiveness despite adherence to organizational & structural aspects of practice Potential to improve quality of services through targeted training of specific skills

11 Assessment of Clinical Skill at Implementing a Practice Limited utility of interviews with clinicians or consumers as skills must be inferred Knowledge tests: informative about clinician’s understanding of practice, but not ability to do it Role play tests: informative about clinician’s ability to use specific clinical skills in simulated situations Direct observation: informative about clinician’s use of clinical skills during actual interactions, but may be reactive to presence of observer Audio/videotaped observation: informative about clinician’s use of clinical skills during actual interactions, but requires dealing with concerns about recording interactions

12 Advantages of Formal Assessment of Clinical Skill at Practice Identifies areas in need of subsequent training Facilitates understanding of differences in effectiveness of clinicians implementing same practice Useful for research focused on understanding differences between consumers in benefit from a practice May lead to certification of clinicians in practices for which fidelity is defined mainly in terms of the interaction between the clinician & client: Illness Management & Recovery Family Psychoeducation (e.g., multi-family groups, behavioral family therapy) CBT (e.g., for psychosis, PTSD)

13 Two Examples of Clinical Skill-Based Fidelity Scales IMR Clinical Competency Scale CBT for PTSD Fidelity Scale

14 IMR Clinical Competency Scale Developed by Gingerich, Mueser, & Meyer Initial step of brainstorming core skills necessary to implement IMR, based on principles & specific strategies outlined in manual Initial draft of competency items, behavioral anchors, & multiple choice questions Feedback on initial draft from 25 experienced IMR clinicians & trainers Revised draft & additional feedback obtained Scale piloted with experienced IMR clinicians & trainers Feedback obtained, revisions made

15 Overview of Scale 19 items, 5-point ratings, behaviorally anchored Administered to individual providers with experience treating at least 3 individual clients with IMR or leading/co-leading 1 group Ratings based on: Observation of 2 IMR sessions Interview with clinician, including role plays Review of written materials

16 Overview of Scale (cont.) Administered by experienced IMR clinician, trained in implementing scale Requires about 6 hours to complete Organization of scale: Process skills Motivational enhancement strategies Educational strategies Cognitive-behavioral strategies IMR group leader skills IMR curriculum

17 Materials in the Evaluators’ Manual 1. Steps for Evaluators (in checklist form) 2. Competency Items in the MN IMR Clinical Competency Scale Item description Skill requirements Primary measurement Missed Opportunity/Comment Boxes

18 Materials in Manual, cont’d 3. Interview Guide Questions for the interview section Examples of role plays for evaluation 4. Scoring Manual 5-point behaviorally anchored scale Descriptors for 3 anchor points 5.Workshop materials

19 Steps for Evaluators Completing the Scale 1.Clinician meeting-preparation for evaluation 2.Observe or listen to audiotapes of 2 sessions (group or individual or both) 3.Review written materials 4.Conduct clinician interview-evaluate skills not demonstrated in observation and follow-up any questions from observation 5.Feedback session-offer praise for strengths and suggestions for areas of improvement

20 11 Items to be rated using behavioral observation 1. Therapeutic relationship 2. Structure of sessions 3. Efficient use of time 4. Motivational enhancement strategies 6. Goal follow-up 7. Educational strategies 8. Use of reinforcement 9. Home assignments 11. Relaxation Training (if one of sessions includes this topic) 15. Group skill: Involving all members of group 16. Group skill: Enlisting support between group members 17. Group skill: Tailoring IMR materials to experiences and goals of each group member

21 Items to be rated using Written Materials 5. Goal Setting 6. Goal Follow-up 10. Relapse Prevention Training 18. Comprehensive Use of IMR Curriculum Questions about written materials should be followed up during the clinician interview

22 How to evaluate the goal tracking sheets? What to look for? Recovery goal Short-term goals-how many and are they related to the long- term goal Steps-how many and are they small enough Any modifications? Remaining questions for Clinician Interview See Interview Guide Additional questions for clarification: Review how the client decided on the goal Why did the client choose the short-term goals listed?


24 Items to be rated using the Interviewer Guide (* Role Plays) 11. Relaxation Training* 12. Behavioral Tailoring and other medication adherence strategies 13. Social Skills Training* 14. Coping Skills Training 19. Knowledge of IMR modules Followed up from written materials review: 5. Goal Setting 6. Goal Follow-up 10. Relapse Prevention Training 18. Comprehensive Use of IMR Curriculum

25 Feedback Meeting With Practitioner & Supervisor Conducted as soon as possible after completion of the interview Explain purpose of meeting is to review results of evaluation Give each person a copy of the Scoring Summary Sheet Start by noting practitioner’s strengths, followed by areas that might be improved Briefly review ratings and the basis for them Provide suggestions for addressing problems areas

26 CBT for PTSD Program 12-16 individual sessions, for consumers with SMI & PTSD Components of intervention: Orientation, crisis plan, breathing retraining (1 session) Psychoeducation about PTSD (2 sessions) Cognitive restructuring, taught as a skill to manage distressing feelings & address trauma-related thoughts & beliefs (9-13 sessions) Supported by one RCT in SMI; multiple other studies underway in other special populations (e.g., addiction, disaster survivors, adolescents)

27 CBT for PTSD Fidelity Scale Ratings based on audio recordings 17 items rated on 5-point Likert scales Narrative feedback on specific ratings Used to facilitate training & certification of clinicians in treatment model Used to verify fidelity to model over treatment study

28 Items on CBT for PTSD Fidelity Scale 1. Agenda setting 2. Homework review 3. Overview of program 4. Crisis plan 5. Breathing retraining 6. Use of Educational materials 7. Psychoeducation 8. Cognitive restructuring 9. Problem solving 10. Trauma focus 11. Assign homework 12. Manual adherence 13. Teaching effectiveness 14. Interpersonal effectiveness 15. Pacing & efficient use of time 16. Reduction of client distress 17. Overall session quality

29 NJ-Dartmouth PTSD Study Kim Mueser, Dartmouth, PI Stan Rosenberg, Dartmouth, Co-PI Jen Gottlieb, Dartmouth, Project Manager Steve Silverstein, UMDNJ, Site PI Weili Lu, UMDNJ, Site Project Manager Phil Yanos, UMDNJ, Clinical Supervisor Stephanie Marcello, UMDNJ, Clinical Supervisor Stephanie Dove-Williams, UMDNJ, Interviewer Danielle Hawthorne, UMDNJ, Interviewer Danielle Paterno, UMDNJ, Interviewer Lindy Fox, Dartmouth, Assessment Supervisor Rosemarie Wolfe, Dartmouth, Data Manager Eric Slade, University of Maryland, Health Economist

30 Training Model Annual Training Weekly On-site Group Supervision Practice Cases Assigned & Sessions Recorded Fidelity Monitoring Via Tapes If fidelity for sessions 4- 16<3.5

31 Training Model: Monitoring treatment fidelity Frontline clinicians were provided with fidelity ratings for all sessions for their first practice cases If the overall fidelity ratings for sessions 4-16 was above 3.5, clinician was certified to take on protocol case If overall fidelity was below 3.5, then the clinician took a second practice case 17/20 clinicians were certified after 1 case, 3 after a second practice case

32 Participants Completers (n=26) Participants (n=34) Male27%29% Age (M+/- SD)46.8+/-10.645.6 +/-10.9 Race AA 27%35% EA58%50% Hisp. & Other15%

33 CBT Session Progress (N=26)

34 Conclusions Clinical skills are useful to rate for both training purposes & to verify fidelity Audio-based fidelity assessments are feasible to conduct with frontline clinicians working with regular clients Clinical skills assessments have potential to improve other practices (e.g., ACT, supported employment)

Download ppt "Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment Kim T. Mueser Dartmouth Psychiatric Research Center."

Similar presentations

Ads by Google