1. Evaluating Residents and Faculty in the Family Practice Center Using a 360 Degree Evalution Tool Bruce Vanderhoff, MD, and Pat Martin, MA, LPCC – Grant Family Medicine Residency

2. Why a 360 Degree Evaluation?  ACGME recommended tool for assessing the general competencies  Challenge of evaluating resident performance in the FPC –Nonpeer input –Reinforcing the role of nursing/office staff as teachers  Favorable organizational experience  History of success in business

3. Why the FPC?  Needed meaningful input from multiple sources –Previous evaluations clearly incomplete –Failure to identify problems interfering with effective office practice –Credibility of feedback related to interpersonal skills –Defense mechanisms

4. Why Incorporate the Competencies  Recognition as valid, universal expectations in family medicine –ACGME requirement –Expectations related to competencies defined in orientation –Convenient framework for assessing the breadth of skills required in the FPC

5. Development  Select, interested group –Subsequent feedback and revision by faculty and resident leaders  Challenges –Identifying observable, measurable behaviors reflecting each competency –Long enough to provide useful information, brief enough to be completed

6. Resident Evaluators  Peers excluded (multiple opportunities within our program)  Chosen: –Self –FPC Medical Directors –Self-selected faculty and community preceptors –FPC Nurse Manager –Resident’s nurse –Front office staff –Patients who routinely see resident

7. Faculty Evaluators  Self  Program Director  2 Peers chosen by Behaviorist  FPC Nurse Manager  Faculty Person’s Nurse  Staff persons from FMC and Med Ed  Chief Residents plus one R1, R2 and R3

8. Targeted Instruments  Instruments targeted to the distinct groups of evaluators –Medical Personnel –Office Staff –Patients  Instruments appropriate to the Evaluee –Residents –Faculty

9.  PATIENT CARE  Demonstrates effective interviewing skills  Documents patient management plan  All appropriate flow sheets complete  Provides and documents preventive health services and health education  Provides definite follow up instructions  Functions as a team player and works collaboratively with health care team Competency-based Questions

10. Questions  PROFESSIONALISM  Dress and mannerisms are appropriate  Is compassionate and respectful in interactions with patients and staff  Shows sensitivity to cultural, age and gender differences  Is punctual and reliable in handling details of patient care  Responds to pages in a timely way  Applies ethical principles to practice and in dealing with others

11. Questions  SYSTEM BASED PRACTICE  Orders cost effective medications, tests and consults  Completes the chart documentation appropriately to maximize billing and coding  MEDICAL KNOWLEDGE  Applies current literature to patient care  Demonstrates procedural competence  Develops appropriate differential diagnosis, treatment plans, and follow up care  Demonstrates appropriate medication usage and is aware of side effects and drug interactions  Obtains adequate history

12. Questions  PRACTICE BASED LEARNING  Applies medical principles appropriately to specific patient needs  Manages time well and sees an appropriate volume of patients in a timely way  INTERPERSONAL SKILLS (COMMUNICATION)  Communicates effectively with patients and staff  Deals with conflict in an appropriate and mature manner  Shows active listening skills with patients, peers and staff  Avoids use of “medicalese” with patients

13. Development and Implementation  Final review and comment by full faculty  Inservices for residents, faculty, nursing and office staffs  Introductory letters to evaluators, including community preceptors

14. TO: RESIDENTS FROM: PAT MARTIN RE: 360 DEGREE EVALUATION In order to meet the ACGME requirements for evaluation in the specific competencies we have adopted the use of the 360 degree evaluation as one tool. This is a self-evaluation combined with an evaluation by the center medical directors, faculty, community preceptors, nursing, front office staff, and patients. From the attached list please select one faculty member and 2 outside preceptors to complete your evaluation. If you are at the East Main center do not select Dr. Fitch; if at Grove City, do not select Dr. McCloy. As center directors they will be evaluating every resident at their center. When you turn in your self-evaluation, I will contact the faculty and preceptors to give them the evaluation forms to complete. If you have any questions, please contact me on beeper 4545 or at 566-0903. Thanks.

15. TO: MARIA TRAVIS, RN AMY JACKSON, RN AMY JACKSON, RN FROM: PAT MARTIN RE: RESIDENT 360 DEGREE FEEDBACK On Wednesday, October 2 the staff at the family practice centers will be asked to participate in the resident 360 degree evaluation. Each nurse will evaluate her residents. The front office staff will also evaluate each resident. Rather than have the front office staff evaluate each resident individually they may divide the forms or do their evaluations of each resident as a group. As nurse managers you will evaluate each resident at your center. Please return the forms to me in person or via in-house mail to the eastside family practice center. The nurses will also have to get the patient responses. They should try to get 3 patients who are regular patients of the resident. I know that will not be easy for the interns, but I know they will do their best. The deadline for return of all forms is Tuesday, October 15. Thank you for your cooperation in this resident evaluation. If you have any questions, please feel free to call, page or email. Thanks again.

16. September 23, 2002 Dear ____________________________, In order to be in compliance with the ACGME guidelines we are required to evaluate the residents in identified areas of competency. As one part of this evaluation we have adopted the 360 degree format in which each resident will be evaluated by him/herself, peers, faculty, community preceptors, and patients. You have been selected to evaluate a resident. Please complete the attached form and return it to me. You may send it in-house mail to Pat Martin, Medical Education, Grant or by US mail from your office. I would like to have all forms returned by Tuesday, October 15. Thank you for participating in this resident evaluation. Results will be presented as a single score for each item and will not be identified by respondent. If you have any questions, please feel free to contact me at 566-0903 or through the Grant operator (be sure to ask for Pat Martin in Family Practice). Thanks again for your assistance.

17. Initial Responses  Resistance –Fear (residents and staff) of reprisals – “time to get even” –Dismissal – “goofy project of the month” –Apathy –Concern that time and effort would go unrewarded  Most – subtle interest

18. Sharing Evaluations  Emphasized limited confidentiality from the start  One-to-one interview with the Director of Behavioral Science –Confidential evaluation summary prior to the meeting  Avg score each competency from each resp grp  Avg score each question from each resp grp  All written comments without change

19. Reports  Copy of 360 degree evaluation: –Permanent record –Behavioral Science Director –Advisor –Resident (Actually received the Behavioral Science Director’s copy after their meeting)

20. Behaviorist Meeting  Resident asked their impressions first –Anything surprise you? –Why do you think someone might have said that? –What one thing would you like to focus on for the next review?

21. Communicating Evaluations  Meet with advisor –Discuss evaluation –Identify specific goals which had been set as a result of the 360 degree evaluation –Discuss plans for any needed behavioral or practice changes  Advisor provides faculty with a summary at the annual resident review

22. Faculty Evaluations  Reviewed by PD and Behaviorist –Summary report developed  Summary report reviewed by PD with the faculty individually –Discuss evaluation –Identify specific goals which had been set as a result of the 360 degree evaluation –Discuss plans for any needed behavioral or practice changes  Serves as annual evaluation

23. Insights  Very thoughtful responses  Physicians downgrade their medical knowledge compared with physician evaluators (Faculty downgrade their teaching skills)  Nursing and office staff evaluations corresponded with faculty impressions and added powerful insights –Able to present concerns in patient context that impressed residents (delayed paperwork = delay in patient getting sick pay)

24. Faculty Insights  Professional Balance –Limit setting (learning to say no) –Reasonable self-expectations  Dress and appearance  Penmanship and Documentation  Teaching issues –Dogmatic –Listen more –Psycosocial –Punctuality

25. Results  Most consistent feedback from patients was that physicians need to be more aware of cost of healthcare  Community preceptors often provided feedback similar to that a junior partner might receive

26. Resident Feedback  Overwhelmingly positive  Unique opportunity to see themselves through others’ eyes  More likely to learn of perceived strengths than weaknesses  Particularly valuable for the consistent performers who otherwise often lack this kind of detailed feedback (attention otherwise goes to the “problem” resident)  Appreciate the insights into practice management they had previously lacked