1. Neurosurgery Resident performance Assessment and modification Program
Edward Benzel, M.D., Program Director Connie Murphy, Program Coordinator Richard Schlenk, M.D., Asst. Program Director
Neurosurgery Resident performance Assessment and modification Program

2. The Next Revolution in Medical Education
“Most clinicians are amateur educators who teach intuitively”
“the fundamental basis of residency training has changed very little in 100 years”
Donlin Long, M.D.

3. Generation 1

4. Why Evaluate? Surgeons evaluate constantly
In the OR
At the bedside
During conferences
“We know a ‘good’ and ‘bad’ resident” ?????????????????

5. Competence and Performance
COMPETENCE - “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served.”
Epstein RM, Hundert EM. JAMA 2002;287:226-35

6. Competence and Performance
“Competence is not an achievement, but rather a habit of lifelong learning.”
Leach DC . JAMA 2002; 287:243-4

7. Competence and Performance
“Competence is contextual, reflecting the relationship between a person’s abilities and the tasks he or she is required to perform in a particular situation in the real world.”
Klass D. Am J Phys Med Rehabil 2000; 79:481-6

8. A Tale

9. Don’t forget to check that all hidden slides are appropriate.
GENERAL COMPETENCIES The residency program must require its residents to develop the competencies in the 6 areas below to the level expected of a new practitioner. Toward this end, programs must define the specific knowledge, skills, and attitudes required and provide educational experiences as needed in order for their residents to demonstrate the competencies.

10. Don’t forget to check that all hidden slides are appropriate.
RESTRUCTURE Thought Process Educational Format Commitment to Education “out of the box”

11. Don’t forget to check that all hidden slides are appropriate.
LINK Goals and Objectives to Competency Assessment via Core Competencies

12. Generation 2

13. Don’t forget to check that all hidden slides are appropriate.

14. Assessment Strategies Subjective Accuracy? Validity? Utility?
Don’t forget to check that all hidden slides are appropriate.
Assessment Strategies Subjective Accuracy? Validity? Utility?

15. There indeed exists a scientific foundation….
Don’t forget to check that all hidden slides are appropriate.

16. Generation 3

19. KEEP IT SIMPLE

20. ACGME Toolbox

21. Key
A 360 evaluation
B Clinical Evaluation
C Consult Evaluation (CSR)
D Written Exams (In-Service)
E Portfolio
F Research Evaluation
G Case logs (Part of Portfolio)
H Surgical Performance Evaluation

22. The 6 Core competencies Patient Care Medical Knowledge
Interpersonal and Communication Skills
Professionalism
Systems-Based Practice
Practice-Based Learning and Improvement

23. I. Patient Care COMPETENCY TOOL Caring and Respect A, B, C
Interviewing
B, C
Decision Making
Management Plans
Educating patients and families A
Procedures
Exam B
Bedside procedures/Surgical Procedures H
Preventative health
A, B
Teamwork

24. VI. Systems-Based Practice
Understands interaction practice and system A
Knowledge of practice and delivery systems
A, C
Practice cost-effectiveness
A, B, C
Advocate for patients within system

25. Portfolio

26. “AND” (Academic Niche Development)
An Important Transparent Part of the Outcomes of the Resident’s Assessment Process

27. Mission Statement
CCF graduating residents should be prepared to occupy a unique niche within neurosurgery. They should be expected to be an “expert” in a specific area or endeavor. To this end, they should have produced independent work or attained degrees that support this label. It is this unique skill set that will make graduating residents recruitable to academic institutions or allow them to make a significant contribution to the field of neurosurgery.

28. Examples of the Niches Political Niche: Business Niche:
Public Policy Degree
White House Fellowship
Political Campaign Work
PAC activity
National Neurosurgical Political Activity
Business Niche:
Business School Degree
Patent Development
Spin-off Company formation
International Niche:
Third World relief effort
International Neurosurgical Education
Basic Science Niche:
Develop a set of Questions
Create an independent and funded research agenda
Clinical Science Niche:
Biostatistics or Public Health Degree
Focused Clinical Fellowship paired with a coherent clinical research agenda
Law Niche:
Law Degree
Externship in Law Firm
Publish in the field of law as it pertains to neurosurgery
Literature Niche:
Publish Fiction or Non-Fiction Books and Pieces

29. Portfolio

30. PROGRAM GOALS AND OBJECTIVES FOR EACH GRADUATE YEAR Each resident reads and must understand the expectations for their upcoming academic year Signature and Date required It is placed in the resident’s portfolio

31. OPERATIVE EXPERIENCE CASE-LOG Each resident is responsible for entering his/her cases (operative reports) into the ABNS database A report is generated and placed in the resident’s portfolio

32. ETHICS-BASED SUMMARY One per semester The resident writes an ethical case report, following guidelines. It is placed in the resident’s portfolio Discussion at his/her review

35. EVIDENCE-BASED CASE REPORT One per semester The resident writes a patient case summary. It is placed in the resident’s portfolio Discussion at his/her review

36. CURRICULUM VITAE (CV)
The Resident is responsible for updating his/her CV as needed An updated copy is kept in the resident’s portfolio Discussed at his/her review

37. Faculty and Ancillary Staff Evaluation Tools

38. Neurosurgery Residency Program 360o Resident Evaluation
Date:     
If you never had contact with this Resident, please place an X in the box and return NO CONTACT
Rate the resident on the performance statements according to the following scale: Please list NA next to question, if it does not apply to this resident
5=EXCELLENT 4=GOOD 3=AVERAGE 2=POOR 1=UNACCEPTABLE 0=FAILURE TO COMPLY 1 2 3 4 5
Communicates effectively with patients and families
Communicates effectively with other health care professionals
Selects appropriate treatments
Maintains quality medical records
Handles transfer of care appropriately
Maintains confidentiality of patients and their families
Responds to pages efficiently
Coordinates patient care with other health care professionals
Respects the rights of patients
Manages health care resources efficiently
Manages stress effectively
Handles consultations in a timely manner
Educates medical colleagues and co-workers
Is an effective member of the health care team
Treats all members of the health care team with respect
Exhibits professional and ethical behavior
Comment:
Evaluator:(Please print)____________________________
Evaluators Signature:______________________________date:________________
All evaluators must place name on evaluation / The evaluations are kept confidential, the residents will not have access to the actual evaluations, only results are given.

39. BIANNUAL PERFORMANCE SCORE: 4.22
Situation Specific Performance Commentary: Academically, as polished as any graduate (ever) from this program. Extraordinarily solid clinician and surgeon. Truly, all things to all people. At the very top of the National Neurosurgery Graduate pool. Will be a "star".
CLINICAL BIOPSY COMMENTS:
* Dr Doe consistently demonstrates excellent competency providing outstanding care to all my patients. I have found him to be extremely valuable as another opinion in this most difficult cases. I often think of MQ as a colleague that a student given his knowledge base and excellent clinical acumen.
* Very good. Great handle on patients.
*In complete control of the service.
SURGICAL BIOPSY COMMENTS:     
* Directed complicated procedure very well. works well with fellows.
*This was a complex case with a great deal of prop thought into formulating a plan which Dr. Sample placed an integral role. His surgical skills are quite on target. I could not have performed the operation any better than Dr. Doe. I have no criticisms whatsoever.
*Thoughtful pre-op planning. He does not need me to show him how to do this operation. (cervical laminectomy and instrumented fusion).
*High level of involvement in the surgical decision-making process. Excellent technical skills. Tentative but careful approach with an unfamiliarly operative case.
*Dr Doe is an outstanding spine surgeon. He clearly is heads and shoulders above any resident I have worked with.
360 degree COMMENTS:
*Dr. Doe is a great person on the team. He is courteous to all the nursing staff and very helpful when needed. It is a pleasure to know him.
*Dr. Doe is very competent in the OR and pleasant. Good to work with.
BIANNUAL INTERVIEW COMMENTS:
*Dr. Doe’s overall performance was superb. He could place more emphasis on career planning (AND) and demonstrate a greater passion for Neurosurgery. Did not demonstrate a clear career plan. Will meet with Dr X to further establish and refine such. Board score was suboptimal. Will take more seriously next year. Will refine ethics and evidence based medicine reports.
BIANNUAL PERFORMANCE SCORE:
4.22
_____________________________________ _______________________________________________
Resident Signature Date Program Director Signature Date

40. The 6 Core competencies Patient Care Medical Knowledge
Interpersonal and Communication Skills
Professionalism
Systems-Based Practice
Practice-Based Learning and Improvement

41. Neurosurgery Residency Program Clinical Evaluation Biopsy Tool*
Rotating Resident:     
Date:     
Patient Problem/Dx:     
New Patient?
Follow Up Visit?
Setting
Amb
In-pt ER
Other
Complexity
High
Moderate
Low
Rate the resident on a scale from 1-5:
Please list NA next to question if it is not applicable to this resident.
Incomplete
Unacceptable
Poor
Average
Good
Superior 1 2 3 4 5
Medical History/ Interviewing Skills
Physical Examination Skills
Humanistic Qualities/ Professionalism
Sensitivity to cultural, ethnic, race issues
Clinical Judgment
Counseling Skills
Organization/ Efficiency
Evidence-based practice
Cost-effectiveness
Incorporation of practice/system issues
Overall Clinical Competence
(must comment if rate 2 or below) Comments:      
Evaluator Name:     
Evaluator Signature:     

42. Superior irrespective of training level
Neurosurgery Residency Program Surgical Performance Biopsy Instrument*
Rotating Resident:     
Date:
Surgical Procedure:     
Rate the resident on a scale from 1-5:
Incomplete
Unsatisfactory
Poor
Average
Good
Superior irrespective of training level 1 2 3 4 5
Surgical Planning
Patient Positioning
Respect for Tissue
Speed and Efficiency
Use of Instruments
Knowledge of Instruments
Flow of Operation
Use of Assistants
Knowledge of Specific Procedure
Overall Rating
must comment if rate 2 or below. Comments:     
Evaluator Name:     
Evaluator Signature:     

43. Situation Specific Performance Commentary
Department of Neurological Surgery
RESIDENT NAME:      
DATE:      
Please list any significant comments (positive and/or negative) regarding the clinical performance of this resident. Please provide specific detailed information:      
Evaluator Name:      

45. SYSTEMS-BASED PRACTICE
PROFESSIONALISM
Is respectful, compassionate and
shows integrity 4
Commitment to ethical principles 5
Administrative obligations
Sensitivity and responsiveness to patients
.375
Committed to self-improvement (compassion)
     
SYSTEMS-BASED PRACTICE
Advocates for quality care
Practices cost-effective medicine
CORE COMPETENCY SCORE
4.33
Ethics Case Report:     
Evidence Methodology Report     
Accomplishments from previous 6 months     
3.5
Goals for the next 6 months     
Chart Maintenance - Comment:      3
Passion for Neurosurgery (commitment for self-improvement) regarding knowledge base and
clinical care – Comment:     
Commitment to academic process – Comment:     
4.5
360 degree evaluations
4.9
PORTFOLIO SCORE
4.11

46. BIANNUAL PERFORMANCE SCORE: 4.22
Situation Specific Performance Commentary: Academically, as polished as any graduate (ever) from this program. Extraordinarily solid clinician and surgeon. Truly, all things to all people. At the very top of the National Neurosurgery Graduate pool. Will be a "star".
CLINICAL BIOPSY COMMENTS:
* Dr Doe consistently demonstrates excellent competency providing outstanding care to all my patients. I have found him to be extremely valuable as another opinion in this most difficult cases. I often think of MQ as a colleague that a student given his knowledge base and excellent clinical acumen.
* Very good. Great handle on patients.
*In complete control of the service.
SURGICAL BIOPSY COMMENTS:     
* Directed complicated procedure very well. works well with fellows.
*This was a complex case with a great deal of prop thought into formulating a plan which Dr. Sample placed an integral role. His surgical skills are quite on target. I could not have performed the operation any better than Dr. Doe. I have no criticisms whatsoever.
*Thoughtful pre-op planning. He does not need me to show him how to do this operation. (cervical laminectomy and instrumented fusion).
*High level of involvement in the surgical decision-making process. Excellent technical skills. Tentative but careful approach with an unfamiliarly operative case.
*Dr Doe is an outstanding spine surgeon. He clearly is heads and shoulders above any resident I have worked with.
360 degree COMMENTS:
*Dr. Doe is a great person on the team. He is courteous to all the nursing staff and very helpful when needed. It is a pleasure to know him.
*Dr. Doe is very competent in the OR and pleasant. Good to work with.
BIANNUAL INTERVIEW COMMENTS:
*Dr. Doe’s overall performance was superb. He could place more emphasis on career planning (AND) and demonstrate a greater passion for Neurosurgery. Did not demonstrate a clear career plan. Will meet with Dr X to further establish and refine such. Board score was suboptimal. Will take more seriously next year. Will refine ethics and evidence based medicine reports.
BIANNUAL PERFORMANCE SCORE:
4.22
_____________________________________ _______________________________________________
Resident Signature Date Program Director Signature Date

47. First Outcomes Results

48. Third Outcomes Results

49. Outcomes Graph Results 5 biannual reviews

50. Practice Hygiene Do All This in a Collegial and Professional Manner
Do What’s Right
(Patient Care, Medical Knowledge, Practice Based Learning and Improvement)
Document What You Did
(System Based Practice
Bill for What You Documented
(System Based Practice)
Make Sure You’re Fairly Reimbursed for What You Billed
Do All This in a Collegial and Professional Manner
(Professionalism and Interpersonal and Communication Skills)

51. Assessment Strategies Objective? Accuracy? Validity? Utility?

52. Numeric/Global scoring and rating
Practical advantages / generation of “data”
“Comfort” with numeric scoring
Reducing competency to a single score ?
Summative score derived from subscores
Measurements of qualities across multiple competencies
Overall score may be satisfactory even if one competency is deficient

53. Essential Questions We Asked Ourselves re Generation 3
Are the faculty and residents objectively assessing resident strengths and weaknesses?
Are the current tools optimjally stimulating the resdients to self reflect and improve?
Are the 6 ACGME competencies being fully incorporated into the process?
Is critical resident assessment information being assessed in a meaningful and accessible manner?
Are meaningful goals and objectives linked to the 6 core competencies?
Are programmatic educational outcomes optimally assessed?

54. ACGME Toolbox of Assessment Methods …Global Ratings
“A number of problems with global ratings have been documented:
… scores can be highly subjective when raters are not well trained…
… sometimes all competencies are rated the same regardless of performance…
… scores may be biased when raters inappropriately make severe or lenient judgments or avoid using the extreme ends of a rating scale…
… research reports are mixed about reproducibility (reliability) of ratings…
… reproducibility appears easier to achieve for ratings of knowledge and more difficult to achieve for patient care and interpersonal and communication skills…”
http ://

55. ?

56. Indices provide little or no constructive feedback

57. What does a score mean to a resident?

58. Not Truly Linking Goals and Objectives to Competency Assessment via Core Competencies
Don’t forget to check that all hidden slides are appropriate.

59. Neurosurgery Residency Program Clinical Evaluation Biopsy Tool*
Rotating Resident:     
Date:     
Patient Problem/Dx:     
New Patient?
Follow Up Visit?
Setting
Amb
In-pt ER
Other
Complexity
High
Moderate
Low
Rate the resident on a scale from 1-5:
Please list NA next to question if it is not applicable to this resident.
Incomplete
Unacceptable
Poor
Average
Good
Superior 1 2 3 4 5
Medical History/ Interviewing Skills
Physical Examination Skills
Humanistic Qualities/ Professionalism
Sensitivity to cultural, ethnic, race issues
Clinical Judgment
Counseling Skills
Organization/ Efficiency
Evidence-based practice
Cost-effectiveness
Incorporation of practice/system issues
Overall Clinical Competence
(must comment if rate 2 or below) Comments:      
Evaluator Name:     
Evaluator Signature:     

60. Residency Assessment …the current issues
Standards for the domains of competence (in 6 ACGME categories) are not defined – ie, goals and objectives not linked to competencies
Competency standards per year not distinct – ie, yearly goals and objectives not clearly delineated with current process
Limited self-assessment of clinical, surgical, and research experience – ie NO self assessment

61. Residency Assessment …the current issues
Portfolio based system remains largely paper based
No immediate access to portfolio by evaluators, mentees, or program director
Lack of compliance !!!!!

62. The Cycle of Performance Improvement
Populate the Biannual Assessment Report
(report card)
Outcomes

63. Pilot Program Third Generation
“times change, knowledge increases, and our training programs should evolve with both”
D Long

64. Generation 4

65. IN PROGRESS…
…Electronic Portfolio Based Competency Assessement and Improvement program
Secure Academic Website
ePortfolio
Resident Assessments
Rotation Schedules
Conference Calendars
Chat Rooms for Residents and Faculty
Links to …………
etc………………….

67. a scientific foundation….
There indeed exists
a scientific foundation….

68. Review and Standards Committee
Lily Angelov
Edward Benzel
Ajit Krishnaney
Andre Machado
Connie Murphy
Shaye Moskowitz
Richard Schlenk
Robert Weil
Michael Speck

69. Questions to ask…
Are the faculty and residents “getting to know” their strengths and weaknesses with our current tools?
Are we optimally stimulating the residents to reflect and improve?
Are we recording the information in a meaningful and accessible manner?
Are the 6 ACGME competencies being fully incorporated into the process?
Are we linking Goals and Objectives to the Competency Assessment Process?

70. PORTFOLIO Reflective Learning Feedback Responsibility ePortfolio

71. What is a Portfolio?? (Reckase 1995)
“A purposeful collection of student work that exhibits to the student (and/or to others) the student’s efforts, progress, or achievement in (a) given area(s). This collection must include student participation in selection of portfolio content; the criteria for selection; the criteria for judging merit; and evidence of student reflection”.

72. Advantages of Portfolio Assessment
Lay the groundwork for reflective learning
Consistent with principle that assessment should promote learning – ie, what we study, we improve
Assessments should be formative so that they provide useful feedback for improving performance and documenting progress
Informs rather than just measures
Model process for continuous professional development and life long learning
The learner takes responsibility for creating his/her own portfolio

73. Requirements of a Portfolio Assessment
Develop an array of competence-specific tools as well as the criteria for evaluation – done
Apply the assessment criteria that provides formative as well as summative feedback – yet to be done
Invest in mentor / faculty development - ongoing

74. Goals of revamping the Neurosurgery resident assessment process
Enhance significance of assessments for the residents – ie, increase constructive feedback
Engage residents in the process – stimulates self reflection
ACGME Core Competency assessment compliance – this is a good thing
Creation of an e-Portfolio system/database – can be standardized and templated
Launch a cutting edge educational pilot program – with charge from Neurosurgery RRC

75. Setting ACGME Competency Standards
Must “set the bar” in terms of competency expectations
Assessment standards must accurately reflect year by year expectations
Document resident achievement of clearly defined standards
Divergence away from resident to resident comparison

76. Committee Guiding Principles
Enhance resident learning
Stimulate self-reflection and self-assessment, resulting in improved performance
Use explicit criterion based standards whose foundation is the ACGME 6 core competencies – ie, link goals and objectives to core competencies
Narrative competency assessment rather than scoring tools used to assess resident progress – the most difficult aspect to accept c

81. Advantages of an e-portfolio system
Instant access to competency profiles.
Individualize for needs of each resident
Enhance efficiency of Program Director and Program Coordinator
Meets the needs of ACGME and RRC
Makes Bi-annual Review meaningful – Club/Clout for Director
Potential for widespread adoption at the national level
Facilitation of RRC Review Process – Fewer onsite reviews required

82. GPS Triangulation Peer assessment Faculty assessment Self assessment
Competence

83. Mentor Summative Evaluations Self Assessment Faculty Evaluation
Formative
Evaluations
Peer Evaluation
360 °
Summative
Evaluations

84. Summative Evaluations
(per competency)
Goals and objectives
Biannual Review
Sign off
Documented changes
Remedial action

85. BIANNUAL PERFORMANCE SCORE: 4.22
Situation Specific Performance Commentary: Academically, as polished as any graduate (ever) from this program. Extraordinarily solid clinician and surgeon. Truly, all things to all people. At the very top of the National Neurosurgery Graduate pool. Will be a "star".
CLINICAL BIOPSY COMMENTS:
* Dr Doe consistently demonstrates excellent competency providing outstanding care to all my patients. I have found him to be extremely valuable as another opinion in this most difficult cases. I often think of MQ as a colleague that a student given his knowledge base and excellent clinical acumen.
* Very good. Great handle on patients.
*In complete control of the service.
SURGICAL BIOPSY COMMENTS:     
* Directed complicated procedure very well. works well with fellows.
*This was a complex case with a great deal of prop thought into formulating a plan which Dr. Sample placed an integral role. His surgical skills are quite on target. I could not have performed the operation any better than Dr. Doe. I have no criticisms whatsoever.
*Thoughtful pre-op planning. He does not need me to show him how to do this operation. (cervical laminectomy and instrumented fusion).
*High level of involvement in the surgical decision-making process. Excellent technical skills. Tentative but careful approach with an unfamiliarly operative case.
*Dr Doe is an outstanding spine surgeon. He clearly is heads and shoulders above any resident I have worked with.
360 degree COMMENTS:
*Dr. Doe is a great person on the team. He is courteous to all the nursing staff and very helpful when needed. It is a pleasure to know him.
*Dr. Doe is very competent in the OR and pleasant. Good to work with.
BIANNUAL INTERVIEW COMMENTS:
*Dr. Doe’s overall performance was superb. He could place more emphasis on career planning (AND) and demonstrate a greater passion for Neurosurgery. Did not demonstrate a clear career plan. Will meet with Dr X to further establish and refine such. Board score was suboptimal. Will take more seriously next year. Will refine ethics and evidence based medicine reports.
BIANNUAL PERFORMANCE SCORE:
4.22
_____________________________________ _______________________________________________
Resident Signature Date Program Director Signature Date

86. Summary Narrative based formative assessment
Year by year ACGME competency standards increase emphasis on self assessment, peer assessment, and self reflection
Increased and objective involvement of mentors
Well-defined means by which to engage more faculty in the process
Creation of innovative web based assessment tool and portfolio
Time and effort investment !!!

87. Practice Hygiene
Do What’s Right
(Patient Care, Medical Knowledge, Practice Based Learning and Improvement)
Document What You Did
(System Based Practice
Bill for What You Documented
(System Based Practice)
Make Sure You’re Fairly Reimbursed for What You Billed
Do All This in a Collegial, Professional, and Responsible Manner using Informed Self Reflection and Feedback to Optimize the Process
(Professionalism and Interpersonal and Communication Skills)

88. Our Options 1. Stay the Course
2. Use Existing Outcome Assessment Scheme, then change
3. Collectively Develop and Implement New Scheme

89. With Energy With Ideas With Resources
Looking for Partners
With Energy
With Ideas
With Resources

90. THANK YOU!!!