1. The Use of Standardized Patients To Assess Behavioral Health Consultant Core Competencies Natalie Levkovich, CEO, Suzanne Daub, LCSW, Neftali Serrano, PsyD Health Federation of Philadelphia Collaborative Family Healthcare Association 16 th Annual Conference October 16-18, 2014 Washington, DC U.S.A. Session # B2c October 17, 2014

2. Faculty Disclosure Please include ONE of the following statements: I/We currently have or have had the following relevant financial relationships (in any amount) during the past 12 months: –Neftali Serrano, Lead consultant, primarycareshrink.com primarycareshrink.com

3. Learning Objectives At the conclusion of this session, the participant will be able to: Describe the novel methodology for assessing BHC core competencies Identify the BHC competencies that were assessed List the application of findings for quality improvement Define the benefits of an ongoing practice-based approach to training and assessment for incumbent BHCs

4. Bibliography / Reference Creating a Simulated Mental Health Ward: Lessons Learned. Rossetti J, Musker K, Smyth S, Byrne E, Maney C, Selig K, Jones-Bendel T. J Psychosoc Nurs Ment Health Serv. 2014 Sep 12:1-7. doi: 10.3928/02793695-20140903-02. Consumer and Relationship Factors Associated With Shared Decision Making in Mental Health ConsultationsMatthias MS, Fukui S, Kukla M, Eliacin J, Bonfils KA, Firmin RL, Oles SK, Adams EL, Collins LA, Salyers MP. Psychiatr Serv. 2014 Sep 15. doi: 10.1176/appi.ps.201300563.Matthias MS, Fukui S, Kukla M, Eliacin J, Bonfils KA, Firmin RL, Oles SK, Adams EL, Collins LA, Salyers MP. Psychiatr Serv. 2014 Sep 15. doi: 10.1176/appi.ps.201300563. Twelve tips for asking and responding to difficult questions during a challenging clinical encounter., Soklaridis S, Hunter JJ, Ravitz P., Med Teach. 2014 Sep;36(9):769-74. doi: 10.3109/0142159X.2014.916782. Epub 2014 Jul 14.. 'Thinking on my feet': an improvisation course to enhance students' confidence and responsiveness in the medical interview. Shochet R, King J, Levine R, Clever S, Wright S. Educ Prim Care. 2013 Feb;24(2):119-24.20. Improving physician-patient communication through coaching of simulated encounters. Ravitz P, Lancee WJ, Lawson A, Maunder R, Hunter JJ, Leszcz M, McNaughton N, Pain C. Acad Psychiatry. 2013 Mar 1;37(2):87-93. doi: 10.1176/appi.ap.11070138.

5. Learning Assessment A learning assessment is required for CE credit. A question and answer period will be conducted at the end of this presentation.

6. Rationale Description of network: The Health Federation of Philadelphia is a regional network of community health centers Over 40 Behavioral Health Consultants across 15 organizations participate in a community of practice Need for meaningful standardization of competencies

7. History Partnerships developed between HFP, Philadelphia College of Osteopathic Medicine and the Thomas Scattergood Foundation to enable the project

8. Simulation Protocol Develop rating tools and standardized patient cases PCOM trained standardized patients Simulation occurred at PCOM lab over two days 21 BHCs, 3 Actors (1 male), 2 Raters Simulations were observed and recorded

9. List of Tools Used/ Created Case Mock Medical Record BHC Rating Scale Working Alliance Inventory (WAI) SP and Self Rating Tool Documentation Template Post-Simulation BHC Feedback Survey

10. Simulation Tools

11. Results of the Simulation ●Poor interrater reliability on expert scale noted (mean average difference in pass rate per question, 25%) ●BHC disagreement with expert rater was normally distributed with a mean disagreement percentage of 38% +/-15

12. Distribution of Disagreement Percentage Between Expert Rater & BHC

13. Results of the Simulation ●Items that BHCs rated themselves worst in (≈ 61+% pass rate) included: o aspects related to communication with PCP o functional understanding of patient concern o integration of medication and medical comorbidities ●Items that expert raters marked as poorest (≈27+% pass rate) included: o SOAP note skills, communication with PCP, o motivational interviewing skills and o collaborative planning with patient

14. Results of the Simulation ●Congruence of BHC and patient actors’ perceptions of the consult was high and generally positive ●BHCs rated themselves slightly lower in general compared to patient ratings

15. Results of the Simulation A: The patient and I agreed on the steps to be taken to improve his / her situation. B: The patient and I agreed on the steps to be taken to improve his / her situation. D: I have doubts about what we were trying to accomplish in the session. I: I have doubts about what we were trying to accomplish in the session.

16. BHC Feedback Strong agreement that: Experience was well organized Experience was useful for competency development Patient feedback will help shape future practice Rater feedback targeted clinical skills for improvement

17. Feedback For Standardized Patient Exercise 1.Improve interrater reliability through training of raters 2.Use more dispassionate raters when possible 3.Develop rating tool rubric 4.Simulation may be best for non-novice BHCs

18. Feedback For Network Training 1.Focus training on a.motivational interviewing b.integration of health behavior change/ medical comorbidities c.formalize SOAP note training 2.Improve integration and use of standardized assessment tools