1. 1 Creative Commons License Attribution-NonCommercial-ShareAlike 2.0 You are free: to copy, distribute, and display this presentation, and/or to make derivative works Under the following conditions: Attribution. You must give the original authors credit. Noncommercial. You may not use this work for commercial purposes. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. See http://creativecommons.org/licenses/by-nc-sa/2.0/ for full license.http://creativecommons.org/licenses/by-nc-sa/2.0/

2. Portfolios in Medical Education: Bridging the Gap from Data to Discovery CREOG 2007 Education Retreat Rio Mar, Puerto Rico Lee A. Learman, MD, PhD Director of Curricular Affairs UCSF Office of Graduate Medical Education

3. Learning Objectives Participants will emerge able to: define and list essential components of a learning portfolio define and list essential components of a learning portfolio distinguish an evidence database, formative portfolio and summative portfolio distinguish an evidence database, formative portfolio and summative portfolio describe potential uses in GME, including the proposed ACGME learning portfolio describe potential uses in GME, including the proposed ACGME learning portfolio

4. Portfolios for Assessment Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. Potential uses: Potential uses: Evidence database: by program vs. learner Evidence database: by program vs. learner Formative: coaching, feedback Formative: coaching, feedback Summative: decisions regarding progress Summative: decisions regarding progress Dannefer EF, Henson LC. Acad Med 2007;82:403.

5. Why Portfolios? Exciting and innovative tool for resident learning and development Exciting and innovative tool for resident learning and development Already used in K-12, higher education, and multiple professions Already used in K-12, higher education, and multiple professions Relieving burden while increasing accuracy for program directors Relieving burden while increasing accuracy for program directors added or lifted - eye of the beholder added or lifted - eye of the beholder Building a community of practice for GME Building a community of practice for GME Within and across specialties to raise the bar Within and across specialties to raise the bar http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

6. Why Portfolios? Needs identified in the literature: Needs identified in the literature: Focus on complex tasks, integrated competencies Focus on complex tasks, integrated competencies More feedback needed as learning objectives become defined and measured more accurately More feedback needed as learning objectives become defined and measured more accurately Context-dependence: sample multiple sources Context-dependence: sample multiple sources New ways to analyze, summarize all the data New ways to analyze, summarize all the data Portfolios can meet these needs Portfolios can meet these needs Dannefer EF, Henson LC. Acad Med 2007;82:403.

7. Self-Reflection & Portfolios Evidence Warehouse Learners Mentors A method for assessing reflection Instructions on reflection

8. Bridge from Data to Discovery Learner receives, instructions and selects experience Evidence Warehouse Reflects and summarizes Discusses reflection with mentor (formative) Reflection is scored as evidence of a specific competency or PBLI (summative)

9. Reflection at UCSF: Instructions Describe the situation that taught you the most about [specific competency] Describe the situation that taught you the most about [specific competency] Describe challenge(s) faced, strategies used Describe challenge(s) faced, strategies used Describe sources of feedback (people, data) Describe sources of feedback (people, data) Relate the situation to previous similar ones Relate the situation to previous similar ones Include details to illustrate challenges you faced and lessons you learned Include details to illustrate challenges you faced and lessons you learned List conclusions re: strengths, opportunities for improvement, and use examples to justify conclusions List conclusions re: strengths, opportunities for improvement, and use examples to justify conclusions Learman LA, Autry AM, Pliska L, OSullivan PS. WGEA Meeting 2006.

10. Sent: Wed 1/10/2007 10:04 PM Subject: Proposal for HROB at SFGH Hello all! As many of you know, Thursday mornings at SFGH pose a challenge to even the most efficient of residents. Juan and I were discussing the following minor changes to resident roles and Thursday mornings HROB clinic that could potentially make a major difference… Please look over these proposed changes. I would love your feedback – most of this was born from the question of how to make M&M a more consistently educational experience for all (i.e., how can the OB chief and R2 get to OB M&M on time?) - 1) All residents (including OB team) arrive at 5M on Thursdays at 8:00 am to start prepping charts; attending and fellow arrive at 8:30 to hear presentations of prepped charts. Conference should be finished by 9 am. If chart prep not complete and patients present at 9 am, at least 2 residents should leave conference to start seeing patients. 2) Night float chief helps to prep some charts on Wed night for Thursday morning so that conference can be finished by 9 am 3) OB Chief must leave clinic at noon to start M&M - supported by clinic attending 4) OB R2 next to leave, ideally at noon as well 5) Gyn R2 and Clinic Chief last to leave clinic, to cover those 12:30 must-see patients (Gyn R1 covering pager and consults if Gyn R2 needed after 12) 6) If >8 patients remain to be seen at 12:00, then OB R2 stays with the rest of the residents while OB Chief leads M&M alone. If needed, other attendings (mobilized by Dr.V) help to see patients Without compromising patient care for education, I'm hoping this may help both clinic flow and M&M utility. Thanks for your time! The Evidence

11. Sent: Thu 1/11/2007 9:38 PM To: OBGYN Resident Class of 2007 Subject: RE: Proposal for HROB at SFGH If all the rest of the chiefs agree with this plan, I'll forward a summary to the rest of the residents and we'll make a go of it next week. As for the main concern of folks arriving at an earlier time, 1) Gyn team is almost always finished rounding by 7:30 because the OR chief must be at OB board rounds. In the remaining half hour, juniors can complete a bit of work. 2) OB Chief will make sure that am rounds finish by 8 am (just as we make our best effort to do on Wed am for conference) and bring charts down from L&D, where at least some have been prepped by the NF Chief. 3) GYN OR Chief (covering for OB Chief) will remind juniors on GYN team to be at conference by 8am and help to get the OB team out of rounds by 8 am 4) GYN Clinic Chief will arrive at 5M at 8:00 instead of 8:30 5) Jeopardy and other clinic residents (OB intern) will arrive at 5M at 8:00 instead of 8:30. Dr. L's suggestion of ensuring that our ancillary services are aware that patients must be seen/in line to see an MD by 11:30-11:45 should also be addressed with M and G (Would you mind doing this, T?). G is aware and enthusiastic about the proposed plan - she has her chart preparations completed by Wednesday evenings, so it shouldn't be a problem to get the charts to the NF chief. Thanks to everyone for being so adaptable! The Evidence

12. Reflective Self-Assessment of Systems-based Practice Instructions: Select a clinical situation you remember during the past year that taught you the most about either practicing cost- effective health care that does not compromise quality of care OR assisting patients in dealing with healthcare system complexities (e.g., surmounting logistical barriers to optimal care – appointments, diagnostic tests). (1) Describe the setting and context including who was present. Setting: San Francisco General Hospital, Thursday morning moderate risk obstetrics clinic. Those involved in the system change are residents, faculty staffing the clinic, clinic staff and clinic flow nurse (2) What challenges did you face in practicing cost-effective healthcare or surmounting systemic barriers to optimal care? The challenge I hoped to address was optimizing the care of patients with complex obstetric problems in the outpatient setting. The major obstacles have been in existence for years: many patients requiring both clinical attention and ancillary services, within a limited time. One additional challenge is finishing the clinic at a time that would allow some or all of the residents to attend M&M conference, a major learning opportunity. (3) Describe what efforts you made to surmount the challenges. What past experiences did you bring to this situation? In the past, there has been a common feeling that the clinic is just that way – its hopeless to try and change it. I tried to separate myself from that because I wanted people to want to change! In conjunction with one of our faculty members, I came up with ideas on how residents could contribute time that did not risk violating duty hour regulations in order to improve clinic flow. For example, the night float team helping to prepare charts, residents arriving a half hour earlier to finish preparing charts, and then presenting to attendings two at once so that we could start seeing patients as soon as they were placed in rooms. I trusted that that every resident in our program would be interested in bettering the system for all. The proposal involved and that a small sacrifice on one rotation would mean educational benefit for them in another rotation (i.e., what goes around, comes around). I used this reasoning when I sent out the email proposal to my fellow chief residents and the program/clinic directors. The Reflection

13. (4) How did you obtain feedback and from whom? Include details providing evidence that you made cost-effective decisions without compromising the quality of care or that you succeeded in assisting the patient in dealing with system complexities? For the last two weeks, our goal of getting at least two residents to the M&M conference has been successful, but the rest of the residents are not having the opportunity to benefit from this educational conference as I had hoped. I have been asking the faculty who staff the clinic, residents, and the clinic flow nurse how the system has been working. The reviews have been mixed, and since it has been only been implemented for a short time, its difficult to determine whether or not this system change is actually beneficial. (5) List what conclusions you drew from situation regarding your strengths and opportunities for improvement, and use examples from the situation to justify your conclusions. I found that motivating people towards change was simple and that those involved were willing to attempt to motivate change themselves, once the activation energy was there. I felt that communicating the importance of this change was one of my strengths. However, putting emotional investment in it means that I have been disappointed that in the first two weeks it hasnt seemed to work. (6) What changes, if any, do you plan to make if you face a similar situation in the future? I plan to encourage future residents to continue with this system and possibly find even more adjustments to improve its efficiency The Reflection

14. Reflection Scenarios: ICS & P Addressing challenging patients Angry or frustrated Worried, scared or guarded (including DV) Difficult or controlling (care-seeking or refusing) Language barrier Transgender Confronting own limitations Stereotypes (PSA, IVDU, dwarfism) Disclosing difficult diagnosis (HIV, cancer) Learman LA, Autry AM, Pliska L, OSullivan PS. WGEA Meeting 2006.

15. Reflection Scenarios: Surgical Surgical Skills 9Routine Cesarean with complication 3Emergency Cesarean delivery 6Surgical decision-making (including conflict with attending) 6Gyn surgery – complications 4Gyn surgery – technical challenges 2Other Learman LA, Autry AM, Pliska L, OSullivan PS. WGEA Meeting 2006.

16. ACGME Portfolio Management Tool An interactive web-based development tool that residents can use throughout their residencies to record and organize their learning and to reflect and receive feedback on their skills as physicians, building evidence that allows them to chart their own progress over time. http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

17. ACGME Portfolio Management Tool First and foremost...a learning tool for residents that enables them to: First and foremost...a learning tool for residents that enables them to: track their experiences track their experiences self-reflect on those experiences share their insights with mentors, share their insights with mentors, receive real-time formal feedback receive real-time formal feedback A repository for resident work products and professional documents meeting the needs of many groups including licensing and certification boards A repository for resident work products and professional documents meeting the needs of many groups including licensing and certification boards http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

18. Portfolio Functions - ACGME Growth Model (formative) – tracks learner development over time Growth Model (formative) – tracks learner development over time Showcase Model (summative) – snapshot demonstrating achievement of identified outcomes as for a grade, promotion, or graduation Showcase Model (summative) – snapshot demonstrating achievement of identified outcomes as for a grade, promotion, or graduation Hybrid – supports both purposes Hybrid – supports both purposes http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

19. Portfolio Approach at CCLCM Dannefer EF, Henson LC. Acad Med 2007;82:403.

20. Evidence-based Essentials Time for reflection and mentorship Time for reflection and mentorship Separation of formative and summative Separation of formative and summative Students select evidence of learning Students select evidence of learning Essays required to aid reflection on integration of competencies Essays required to aid reflection on integration of competencies Rigorous measurement standards for summative assessments (fair, valid, reliable) Rigorous measurement standards for summative assessments (fair, valid, reliable) Dannefer EF, Henson LC. Acad Med 2007;82:403.

21. UCSF Scoring Rubric 1. Describes encounter only. 2. Unsupported opinions about lessons learned. 3. Superficial justification of lessons learned. 4. Discussion well-supported with examples of challenges, techniques and lessons learned. challenges, techniques and lessons learned. 5. Analyzes factors from experience that contribute to progress. to progress. 6. Justifies strategies used and evidence for effectiveness. effectiveness. moresuperficial deeper Learman LA, OSullivan PS. AAMC-RIME 2006.

22. Reflection Growth by PGY Surgical skill^3.30 (0.73)3.13 (1.07)-0.19 Professionalism2.70 (1.18)2.68 (1.66)-0.01 Communication2.63 (1.19)2.63 (1.20)0.00 Evidence-based medicine2.35 (1.07)3.07 (1.08)0.65 Systems-based practice1.97 (1.31)2.37 (1.60)0.28 Practice improvement1.53 (1.20)1.78 (1.26)0.20 OVERALL # 2.28 (0.47)2.64 (0.62)0.36 Skill for ReflectionYears 1-2 Mean (SD) Years 3-4 Mean (SD) Effect Size* * Effect size = (difference in means)/pooled standard deviation ^Skill scores range from1-6 and include all exercises completed by 32 residents. #Overall scores include only the 25 residents who completed all six reflections. Learman LA, OSullivan PA. AERA Annual Meeting 2007.

23. Resident Feedback Focus groups after 1 st year of reflection 05/06: Valued reflection, but had incomplete and variable understanding Valued reflection, but had incomplete and variable understanding Preferred reflection-in-action (in real time) over reflection-on-action (delayed) Preferred reflection-in-action (in real time) over reflection-on-action (delayed) Discounted value of written (versus oral) reflections Discounted value of written (versus oral) reflections Felt that specific assignments were constraining and artificial Felt that specific assignments were constraining and artificial Wanted feedback on and discussion of reflections Wanted feedback on and discussion of reflections Foster-Barber, Chittenden, Learman, OSullivan. UCSF Education Day 2007

24. Improvements in 06/07 Better explanation of role of reflection in medical education Better explanation of role of reflection in medical education Choice among 6 options, 3 for each semi-annual meeting Choice among 6 options, 3 for each semi-annual meeting Clearer instructions Clearer instructions Exercises divided into discrete tasks Exercises divided into discrete tasks More lead time for sharing with peers More lead time for sharing with peers More lead time for review by program directors prior to feedback session More lead time for review by program directors prior to feedback session Foster-Barber, Chittenden, Learman, OSullivan. UCSF Education Day 2007

25. Lessons Learned on Reflection Optimal design of exercises unclear Optimal design of exercises unclear Need to: Need to: Hone residents understanding of reflection Hone residents understanding of reflection Give more freedom in content/timing of reflections Give more freedom in content/timing of reflections Provide mechanisms to ensure timely feedback Provide mechanisms to ensure timely feedback Introduce reflection exercises earlier in medical education to improve their acceptability to residents Introduce reflection exercises earlier in medical education to improve their acceptability to residents Foster-Barber, Chittenden, Learman, OSullivan. UCSF Education Day 2007

26. Reflecting About Portfolios

27. What They Are Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. A learning tool enabling residents to: A learning tool enabling residents to: track their experiences track their experiences self-reflect on those experiences self-reflect on those experiences share their insights with mentors, share their insights with mentors, receive real-time formal feedback receive real-time formal feedback

28. Consider What do you think? What do you think? Fad or Formidable Innovation Fad or Formidable Innovation Which aspects seem more/less valuable? Which aspects seem more/less valuable? Tracking progress Tracking progress Refection and self-assessment Refection and self-assessment Feedback and mentorship Feedback and mentorship

29. Their Purposes or Functions To store evidence of learning To store evidence of learning a true portfolio a true portfolio To promote feedback, reflection, growth To promote feedback, reflection, growth = formative To determine advancement, graduation To determine advancement, graduation = summative

30. Consider Which functions will be easiest to implement in your residency program? Which functions will be easiest to implement in your residency program? Data warehouse Data warehouse Formative review of growth, development Formative review of growth, development Summative decisions Summative decisions Why? Why?

31. The Essentials for Success Time for reflection and mentorship Time for reflection and mentorship Separation of formative and summative Separation of formative and summative Learners own access, grant permission Learners own access, grant permission Essays assigned to aid reflection Essays assigned to aid reflection Summative decisions are based on fair, valid, and reliable assessments Summative decisions are based on fair, valid, and reliable assessments » Not even the best technology can make-up for an absence of any of these essentials!

32. Consider One Mentorship and Feedback Mentorship and Feedback How is time currently set aside for mentorship? Are mentors different than faculty responsible for advancement and graduation? Do they review evidence of learning and provide formative feedback? How might this be improved? How is time currently set aside for mentorship? Are mentors different than faculty responsible for advancement and graduation? Do they review evidence of learning and provide formative feedback? How might this be improved? Owning the Evidence Owning the Evidence Currently, are any assessments or evidence of learning owned by your residents and shared with you at their discretion? What kinds of evidence could be? Currently, are any assessments or evidence of learning owned by your residents and shared with you at their discretion? What kinds of evidence could be? Reflecting on Learning Reflecting on Learning What opportunities (or time) do your residents have to do structured reflections about their progress? How could this be created or expanded? What opportunities (or time) do your residents have to do structured reflections about their progress? How could this be created or expanded?

33. ACGME Learning Portfolio Timeline Now: Alpha test, cultivate early adopters, specialty user-groups Now: Alpha test, cultivate early adopters, specialty user-groups Early 2008: Create beta phase prototype Early 2008: Create beta phase prototype Mid 2008 – Late 2009: Beta testing phase Mid 2008 – Late 2009: Beta testing phase Early 2010: Finalize initial roll-out prototype Early 2010: Finalize initial roll-out prototype 2010: Initial roll-out available, voluntary 2010: Initial roll-out available, voluntary 2016: Full implementation, available for all 2016: Full implementation, available for all Ongoing: Consider linkages to UME and MOC Ongoing: Consider linkages to UME and MOC http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

34. Coming Sooner Than 2016 PIF Transition Document PIF Transition Document PBLI: Describe one learning activity in which residents engage to identify strengths, deficiencies, and limits in their knowledge and expertise (self-reflection and self-assessment); set learning and improvement goals; identify and perform appropriate learning activities to achieve self-identified goals (life-long learning) PBLI: Describe one learning activity in which residents engage to identify strengths, deficiencies, and limits in their knowledge and expertise (self-reflection and self-assessment); set learning and improvement goals; identify and perform appropriate learning activities to achieve self-identified goals (life-long learning) New PIF likely to elaborate... New PIF likely to elaborate...

35. Opportunities Studies across GME programs to establish fairness, validity and reliability of measures Faculty development for giving high- quality, behaviorally-anchored feedback Faculty development for advisors and summative assessors Optimize how self-reflection happens Participate in beta-testing of ALP

36. To Learn More... Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine [of Case Western Reserve University]. Academic Medicine 2007;82:493-502. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine [of Case Western Reserve University]. Academic Medicine 2007;82:493-502. Challis M. AMEE Medical Education Guide No. 11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher 1999;21(4):370-86. Challis M. AMEE Medical Education Guide No. 11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher 1999;21(4):370-86. OSullivan PS, Cogbill KK, McLain T, Reckase MD, Clardy JA. Portfolios as a novel approach for residency evaluation. Academic Psychiatry 2002;26(3):173-79. OSullivan PS, Cogbill KK, McLain T, Reckase MD, Clardy JA. Portfolios as a novel approach for residency evaluation. Academic Psychiatry 2002;26(3):173-79.

37. Acknowledgments Meg Autry, Laura Pliska, and Patricia OSullivan for development and implementation of the reflection exercises Patricia OSullivan for feedback regarding earlier versions of this presentation