1. http://www.eportfolios.ac.uk ePortfolios Using ePortfolios to support annual appraisal in undergraduate medicine Simon Cotterill, Philip Bradley, Rosie Stacy School of Medical Education Development

2. http://www.eportfolios.ac.uk ePortfolios Reflective approach to evidencing achievements: Clinical logbooks and personal portfolios, which allow students to identify strengths and weaknesses and to focus their learning appropriately, can provide such information. Using these will emphasise the importance of maintaining a portfolio of evidence of achievement, which will be necessary once they have become doctors and their licence to practise is regularly revalidated." "Clinical logbooks and personal portfolios, which allow students to identify strengths and weaknesses and to focus their learning appropriately, can provide such information. Using these will emphasise the importance of maintaining a portfolio of evidence of achievement, which will be necessary once they have become doctors and their licence to practise is regularly revalidated." Tomorrow's Doctors, GMC, 2002 Appraisal Students must receive regular, structured and constructive appraisal from their teachers during the mainly clinical years of the curriculum. This allows the medical school to judge their clinical knowledge and competence against the principles set out in Good Medical Practice. "Students must receive regular, structured and constructive appraisal from their teachers during the mainly clinical years of the curriculum. This allows the medical school to judge their clinical knowledge and competence against the principles set out in Good Medical Practice." Tomorrow's Doctors, GMC, 2002 Background #1

3. http://www.eportfolios.ac.uk ePortfolios Built on 2 years experience of using ePortfolios in MBBS from Sept 2003 Background #2 FDTL-4 Collaborative project: http://www.eportfolios.ac.uk/FDTL4 Cotterill SJ., McDonald AM., Drummond P., Hammond GR. Design, implementation and evaluation of a 'generic' ePortfolio: the Newcastle experience Proc. ePortfolio 2004 (ISBN 2-9524576-0-3)

4. http://www.eportfolios.ac.uk ePortfolios Aims and Objectives of the Appraisal Project (2005/6) Support and further develop the use of portfolios and related assessment processes in the MBBS curriculum. Provide students with the facility to evidence their achievement of specific programme outcomes in their ePortfolios These will be seen by and discussed in an annual appraisal. Support the one-to-one discussion/coaching and the evaluation and refinement of the process with the intention of eventually rolling it out across the full 5 years of the programme. Support the evaluation of this

5. http://www.eportfolios.ac.uk ePortfolios

6. http://www.eportfolios.ac.uk ePortfolios

7. http://www.eportfolios.ac.uk ePortfolios

8. http://www.eportfolios.ac.uk ePortfolios Student Perceptions of the ePortfolio and Appraisal Process Online Questionnaire (LSE) 14 fixed-response questions (6 point Likert scale) 9 open-ended questions 213 Respondents (90% response rate) Evaluation 2005/6

9. http://www.eportfolios.ac.uk ePortfolios Portfolio The ‘My Evidence’ section of the portfolio within LSE was used to support the appraisal process. question % agree (scores 4-6) Overall the ePortfolio was easy to use74% The ‘My Evidence’ section of the portfolio was easy to navigate81% Attaching evidence (files, links etc). to the portfolio was straight forward 91% It was easy to print off the ‘My Evidence’ section87% Having a word limit on the ‘reflective accounts’ helped me improve my summative skills 64% I felt that there was sufficient guidance and instructions about using the portfolio 48% Overall I found the process of evidencing my development useful47%

10. http://www.eportfolios.ac.uk ePortfolios The Appraisal Session This relates to your face-to-face appraisal session question % agree (scores 4-6) I had a clear understanding of what I needed to do for the appraisal 60% I felt apprehensive about having an appraisal63% My appraiser made me feel at ease91% The session was well facilitated86% The appraisal session referred to my evidence from the portfolio90% Overall the appraisal session was a positive experience for me70% Having had the appraisal will influence the way I approach my learning next year 58%

11. http://www.eportfolios.ac.uk ePortfolios Analysis of the qualitative data What did you most helpful about the appraisal session ? 102 students responded Reflection43 -Holistic / Overview12 Personal / Interaction35 Reassurance 8 Process (portfolio / appraisal) 17 Learning Evidencing Skills17 Planning / goal setting14 Specific Advice 8 Categorisation of anonymised data by SC

12. http://www.eportfolios.ac.uk ePortfolios “The fact that you actually spoke about your progress made you feel that it wasn't just about exams” Personal / Interaction (n=35) “It was really good to talk about the course and how I was doing and what I could do better. I really found it beneficial to have someone to bounce things off.” “It was nice to feel the university is kind of 'monitoring' us. That we aren’t just candidate numbers.” What did you most helpful about the appraisal session ? Reflection (n=43) “It made me think about what I have achieved this year.” “Allowed me to see my own achievements.” “I was able to clarify to myself via the facilitator both the things that have gone well regarding my learning this year, and the things that I needed to improve.”

13. http://www.eportfolios.ac.uk ePortfolios Holistic / Overview (n=12) Included recognition of ‘life-wide learning’: “The appraiser helped me realise that there were other activities that I did which would also constitute evidence, something that I hadn't known previously…” “Brought things together. Made me sum up my overall progress of the year, which was a really nice thing to do. Positive reflection.” “Help to give an overall view of this academic year.” “It brought together different aspects of my learning to date.” “It linked in the extra curricular activities to the medical course.” What did you most helpful about the appraisal session ?

14. http://www.eportfolios.ac.uk ePortfolios Process (portfolio / appraisal) n=17 What did you most helpful about the appraisal session ? “Having a timetabled session forced me to compile my portfolio better as it gave me a deadline to work towards.” “It allowed me to find out if I was completing the portfolio and using it in the correct way. It gave me some feedback on my general progress this year.” Learning Evidencing Skillsn=17 “before the appraisal session I didn’t know what sort of evidence could be included in some of the sections, for example, 'managing my own learning' seemed vague to me and I didn’t know what to include as evidence…” “helpful to be able to reflect upon my development especially learning what counted as evidence.”

15. http://www.eportfolios.ac.uk ePortfolios Reassurance n=8 “Chance to talk through how I felt the year went and what I was worried about.” “It reassured me that what I had been doing was right.” What did you most helpful about the appraisal session ? Planning / goal setting n=14 “…thinking about how I will continue next year was useful as it is good to make plans now rather than just let yourself slip back into bad habits.” “It highlights what I need to improve on and how it could be done”

16. http://www.eportfolios.ac.uk ePortfolios What did you least helpful about the appraisal session ? 75 students responded Personal / Interaction 20 Timing (near exams / lectures) 19 Rational / understanding the process 9 Limited time of session 7 No benefit (reason unspecified) 7

17. http://www.eportfolios.ac.uk ePortfolios Timing (near exams / lectures) n=19 “It came at a time when I wanted to be revising for exams. It would be more useful after the exams.” “It was at a time when we all had better things to be doing, such as revising for exams. It would have been better to have it before Easter.” What did you least helpful about the appraisal session ? “… the session itself seemed to be more for the benefit of the appraiser than for me, as it mainly comprised of me simply reading out sections from my portfolio, which felt a little pointless.” “I suppose there was a lot of ticking boxes and it wasn't very personal.” Personal / Interaction (n=20) n.b. this is a very ‘mixed bag’

18. http://www.eportfolios.ac.uk ePortfolios What did you least helpful about the appraisal session ? Rational / understanding the process (n=9) “I didn't really know before the session how much evidence or how many reflections were required to satisfactory evidence.” “I didn't really know what I had to do. It made preparation difficult.” Limited time of session (n=8) “It felt a little rushed” “The time - too short which meant we could not discuss fully some of the issues I raised.”

19. http://www.eportfolios.ac.uk ePortfolios Following the session what aspect of your development do you feel that you most need to improve on ? (open ended question, answers categorised by SC) “Learning some Geordie to be able to communicate more effectively with patients during hospital visits!” Organisation & Time management 34 Study Skills21 Creativity / Resourcefulness11 Motivation 9 Communication Skills 6 Reflection 6 Extra curricula activities 6 Clinical skills 2 Misc. 9

20. http://www.eportfolios.ac.uk ePortfolios Appraiser Feedback (n=7) “The other thing that would definitely make the appraisal easier would be if the students were asked to let us have their form in advance. Even a couple of days before would be okay by me and it would mean you entered the appraisal with a clearer picture of how each student felt they were doing.” “A common problem was the students filled in the summary of evidence (in a hurry the night before) and had no evidence attached to support it. I therefore spent a fair amount of time explaining how to get and attach evidence and how the should only really refer to this evidence in the summary of evidence box.”

21. http://www.eportfolios.ac.uk ePortfolios Appraiser Feedback (n=7) “Some did not grasp what the appraisals were about and clearly placed a low value on them. Many said they would rather be revising. I think some form of tangible reward should be offered - though not sure what - they seem to value grades, and you don't want this to be an assessment process - bit of a conundrum there.” “Most sessions went over the 15 minutes. I'd say 20 minutes would be adequate.” “I would urge you to consider doing this in second year as well as I believe the appraisal gives the student an opportunity to reflect on the year; something that only a small proportion of the year do as a matter of course!”

22. http://www.eportfolios.ac.uk ePortfolios Summary #1 Overall, 70% of Stage 1 students found it a positive experience Students found the process useful for reflection, interaction/discussion & planning. Some students found this process instructive on how to evidence their development for future appraisals/assessment 58% felt it would influence their learning in Stage 2 Many students were not clear on the purpose/rationale and specific requirements (a common issue in portfolio learning, increased training planned). Timing – near exams – was an issue

23. http://www.eportfolios.ac.uk ePortfolios Summary #2 Overall, appraisal supported by ePortfolio has been successful for MBBS stage 1 Following successful evaluation it is being rolled across the programme Refinements to the process were implemented for 2006/7 4 extra skills to evidence for the later years Timing of appraisals brought forward – away from exams Features of ‘My Evidence’ were incorporated back into the ‘generic’ ePET portfolio Later expanded to link Blogs to structured skill/outcomes (EPICS-2)

24. http://www.eportfolios.ac.uk ePortfolios Further information: S.J.Cotterill@ncl.ac.uk http://www.eportfolios.ac.uk