1. AECOM COGME Seminar Implementing the Competencies 360 Degree Evaluations Catherine C. Skae, MD Director, Pediatric Residency Program Children’s Hospital at Montefiore

2. Outline  Definition of 360 degree evaluation  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

3. Outline  Definition of 360 degree evaluation  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

4. Definition  Consist of measurement tools completed by multiple people in a person’s sphere of influence  Evaluators are superiors, peers, subordinates, patients, and families  Two practical challenges: constructing the surveys and orchestrating data collection

5. Outline  Definition of 360 degree evaluations  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

6. ACGME Toolbox Suggested Best Methods of Evaluation 1= Most Desirable: Patient Care- Work within a team Practice-Based Learning & Improvement- Use of information technology Facilitate learning of others

7. ACGME Toolbox Suggested Best Methods of Evaluation 1= Most Desirable: Professionalism- Ethically sound practice Sensitive to culture, age, gender, disability issues Systems-based Practice- Understand interaction of their practices w/ the larger system Advocate for patients within the healthcare system

8. ACGME Toolbox Suggested Best Methods of Evaluation 2= Next Best Method: Patient Care- Caring & respectful behaviors Counsel & educate patients and families Performance of medical procedures Interpersonal & Communication Skills- Creation of therapeutic relationships Listening skills Professionalism- Respectful, altruistic Syst-Based Practice- Practice cost-effective care

9. ACGME Toolbox Suggested Best Methods of Evaluation 3= A Potentially Applicable Method: Patient Care- Interviewing; Developing and Carrying out Management Plans Medical Knowledge- Investigatory & Analytic thinking Knowledge and Application of Basic Sciences Practice-Based Learning & Improvement- Analyzing own practice for needed improvements

10. Outline  Definition of 360 degree evaluations  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

11. Faculty- Continuity Clinic Intern/Resident Faculty- Inpatient/ Elective

12. Faculty- Continuity Clinic Junior Residents Intern/Resident Senior Residents Faculty- Inpatient/ Elective

13. Intern 360 Degree Evaluation CHILDREN’S HOSPITAL AT MONTEFIORE EDIATRIC ROTATION EVALUATION JUNIOR/SENIOR RESIDENT: _________________________ INTERN: ________________________________(1, 2= below expectations; 3 = meeting expectations; 4, 5 = exceeding expectations) ALL 1,2, AND 3 RATINGS REQUIRE COMMENTS AT END OF FORM I. PATIENT CARE Fails to gather essential and accurate information via history and P.E.1 2 3 4 5 N/A Gathers essential and accurate information via history /P.E. Experiences difficulty in developing and carrying out plans 1 2 3 4 5 N/A Develops and carries out patient management plans appropriately Has difficulty taking responsibility and lacks accountability for decisions 1 2 3 4 5 N/A Demonstrates responsibility and accountability for decisions Has difficulty performing technical procedures 1 2 3 4 5 N/A Performs technical procedures skillfully II. MEDICAL KNOWLEDGE Does not demonstrate independent investigatory and analytic thinking 1 2 3 4 5 N/A Demonstrates independent investigator and analytic thinking Does not have adequate knowledge base of basic and clinical sciences 1 2 3 4 5 N/A Knows and applies basic and clinical sciences III. INTERPERSONAL SKILLS AND COMMUNICATION Does not use effective listening skills to elicit information 1 2 3 4 5 N/A Uses effective listening skills to elicit information Does not work effectively as member of health care team 1 2 3 4 5 N/A Works effectively as a member of a health care team Has difficulty in creating & sustaining therapeutic relationships 1 2 3 4 5 N/A Creates and sustains therapeutic relationships

14. Faculty- Continuity Clinic Junior Residents Intern/Resident Senior Residents Physician Assistants/ Nurses Faculty- Inpatient/ Elective

15. Faculty- Continuity Clinic Junior Residents Social Workers Intern/Resident Senior Residents Physician Assistants/ Nurses Faculty- Inpatient/ Elective

16. Social Work 360 Degree Evaluation ALBERT EINSTEIN COLLEGE OF MEDICINE- CHILDREN’S HOSPITAL AT MONTEFIORE PEDIATRIC ROTATION EVALUATION SOCIAL WORKER: _________________________ RESIDENT: _______________________(1,2 = below expectations; 3 = meeting expectations; 4,5 = exceeding expectations) DATE OF EVALUATION: ___________________________ I. MEDICAL KNOWLEDGE Knows how to make appropriate Social Services referrals 123 45N/A II. INTERPERSONAL SKILLS AND COMMUNICATION Communicates well with Social Workers 123 45N/A III. PROFESSIONALISM Exhibits a positive attitude toward work123 45N/A Attends and contributes to Interdisciplinary Team Rounds12345N/A PLEASE ADD COMMENTS:

17. Faculty- Continuity Clinic Interns/ Junior Residents Social Workers Patients/ Families Intern/Resident Senior Residents Physician Assistants/ Nurses Faculty- Inpatient/ Elective

18. Faculty- Continuity Clinic Interns/ Junior Residents Social Workers Patients/ Families Intern/Resident Standardized Patients Senior Residents Physician Assistants/ Nurses Faculty- Inpatient/ Elective

19. Outline  Definition of 360 degree evaluations  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

20. Results/ Scholarship Assessing Competencies of Pediatric Residents in an Outpatient Setting. Sandra Braganza, Iman Sharif, Philip Ozuah. Objectives: To determine whether the assessment of pediatric residents’ ACGME competencies by peers, support staff, and parents differs from the assessment by preceptors alone. To identify particular ACGME competencies for which differences between preceptor and peer/staff/parent assessments are most pronounced.

21. Results/ Scholarship Methods: 14 statements assessing: Interpersonal & Communication Skills- Communication and Empathy Professionalism- Respect & Integrity

22. Results/ Scholarship Results: 271 surveys completed Compared to preceptors, resident peers reported more problems with Communication and Empathy. Preceptors noted more problems with Communication than did parents. Parents were more likely to report problems with Empathy and Respect than were preceptors. Conclusion: 360 degree assessment was useful in eliciting different evaluations of competencies by different observers that interact with pediatric residents.

23. Results/ Scholarship Parental Assessment of Quality of Ambulatory Care: How Do Attendings and Residents Compare? Andrew D. Racine, Sheila Liewehr, Laura Weingart, Scott D. Krugman. Eastern Society for Pediatric Research. March 2005. Objective: To compare the quality of pediatric ambulatory care reported by parents of children with resident versus attending primary care providers.

24. Results/ Scholarship Results: 473 surveys were returned No differences were found between residents and attendings in aggregate scores for any of the 5 quality domains But on 2 individual items: does the doctor explain things to your satisfaction, and can the doctor arrange for referrals- attending scores were higher. Conclusions: Parental assessment of the overall quality of care delivered by pediatric residents compares favorably to that delivered in similar settings by attending pediatricians.

25. Results/ Scholarship Determining the Essential Components of Professionalism and Interpersonal/ Communication Skills. Laura Dattner, Philip Ozuah. Pediatric Academic Societies- May 2005. Objective: To determine what pediatric patients and their parents consider to be essential components of professionalism and interpersonal/ communication skills.

26. Results/ Scholarship Methods: Anonymous web-based survey for hospitalized patients and their parents. Respondents were asked eight questions about the residents who cared for them.

27. Results/ Scholarship Results: 253 resident evaluation surveys were completed. Overall, residents received very positive evaluations. Questions which predicted a global score of 5 were: “Does Dr. X. speak to you in ways that are clear and easy to understand?” “Does Dr. X. include you in decisions and planning of care?” “Does Dr. X. listen to your concerns?”

28. Results/ Scholarship Conclusions: Essential components of professionalism and interpersonal/ communication skills which independently predicted an excellent global score were: listening to concerns including families in decisions and planning of care clear communication

29. Results/ Scholarship  Can We Rely on Adolescent Standardized Patients to Assess Communication Skills?  Reliability of Adolescent Standardized Patients in Assessing Professionalism.  The Utility of Unannounced Standardized Patients in Medical Education. Philip Ozuah, Marina Reznik, and Sandra Braganza. Pediatric Academic Societies- May 2005.

30. Results/ Scholarship  Adolescent standardized patients (SP) are a reliable method for use in assessing residents’ competency in communication skills.  There was a strong correlation between faculty and adolescent SP’s ratings of professionalism.  Residents displayed lower levels of professionalism in unannounced standardized patient encounters.

31. Outline  Definition of 360 degree evaluations  ACGME Suggested Best Methods  Organization/ Timing  Results/ Scholarship  Future

32. The Future Peers Fellows Inpatient Nurses Unit Clerks myevaluations.com

33. Thank You!