1. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Implementation and evaluation of an electronic portfolio to support undergraduate medicine Simon Cotterill, Tony McDonald, Geoff Hammond, Philip Bradley School of Medical Education Development University of Newcastle upon Tyne, UK

2. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Regional medical schoolRegional medical school –1,400 students –1,600 contributing staff –Multiple points of entry: Standard 5 year programmeStandard 5 year programme Accelerated (4 year) programmeAccelerated (4 year) programme Joint programme with DurhamJoint programme with Durham Outcomes based CurriculumOutcomes based Curriculum Undergraduate Medicine at Newcastle - The MBBS Programme

3. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Managed Environments for Portfolio-based Reflective Learning. Integrated Support for Evidencing Outcomes. An FDTL-4 project: Newcastle University (lead site) Leeds University Sheffield University Dundee University (for consultancy) Developing Web based portfolios to support reflective approaches for evidencing the attainment of programme outcomes in undergraduate Medicine. Closely integrated with on-line curricula and study guides to become an integral component of managed learning environments for Medicine. Project funded by

4. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 ‘Generic’ ePortfolio framework Generic tools (CV, Meetings recorder etc) + Context-Specific toolsGeneric tools (CV, Meetings recorder etc) + Context-Specific tools ePortfolio is customised by course and year-groupePortfolio is customised by course and year-group Stand-alone embedded within a VLEStand-alone embedded within a VLE MBBS Phase I ePortfolio Learning Diary, Meetings with Personal Tutors, CV, SWOTLearning Diary, Meetings with Personal Tutors, CV, SWOT Learning Outcomes, Log of Family and Patient visitsLearning Outcomes, Log of Family and Patient visits Non-mandatory, choice of paper or online for diary, meetings & logNon-mandatory, choice of paper or online for diary, meetings & log Year 4 Student Selected Components (SSCs) Learning Outcomes and Action PlanLearning Outcomes and Action Plan Mandatory for 1 of the students’ 3 SSCsMandatory for 1 of the students’ 3 SSCs Content not assessed/graded (could inform borderline cases)Content not assessed/graded (could inform borderline cases) Development and Piloting in 2003/4

5. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Learning Diary and Planning Tool

6. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Recording Meetings with Personal Tutors

7. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004CV

8. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Year 4 SSC Portfolio: Action Plan

9. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Year 4 SSC Portfolio: Action Plan

10. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Research and Evaluation Studies *Preliminary Findings*

11. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Evaluation of a Phase I Portfolio/Log Book (paper and electronic versions). (paper and electronic versions). Cotterill SJ, Sarma S, McDonald AM, Bradley P Ethical approval / informed consent Focus groups - facilitated by a 4th year student (Sean Sarma) Questionnaires - based on findings of the focus groups (on-going) Key issues from the focus groups Generally well received but what motivation ? not assessed not viewed by tutors (staff interface completed later) Need better clarity of purpose Wanted a more quirky / fun design ! Learning diary - useful at first, less so over time Good to browse the Learning Outcomes

12. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Ethical approval / informed consent Questionnaires (before and after using the ePortfolio) change in awareness of intended learning outcomes ? factors influencing use of the ePortfolio attitudes and perceptions of using the ePortfolio student perception of impact of using the ePortfolio evaluation of technical features and ‘usability’ 186 students completed the portfolio (~100% compliance) 105 questionnaires completed (56% of cohort) at 30/06/2004 Evaluation of an electronic portfolio to facilitate reflective learning in stage 4 medical students during their student selected components. Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR

13. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Outcomes recorded and time to complete: median number of learning outcomes:5 median number of ePortfolio sessions:6 median time taken to complete ePortfolio: 3 hours (IQR 2-4 hours) Locations of computers used: 77% used computers at the University 59% used computers at home 41% used computers on location at the SSC Usability: 67% found the ePortfolio easy to navigate 80% found the online instructions easy to understand 79% thought that the LO action plan was quite intuitive 74% thought that the submission process was easy to understand

14. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Evaluation - positive “ It encouraged me to really give thought to what I wanted to achieve during the option, which was especially useful as this was my first option. As a result of the portfolio I think I got much more out of the option than I would have otherwise. ” “ It made me concentrate on creating aims at the start of the option and allowed me to plan the option with my supervisor in a defined way. Overall it made my learning for the option more organised and focused. ”

15. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Evaluation - negative “ Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could. ” “ Access to computer as option took place in Barnard Castle, records may have been more detailed if completed straight after meetings etc. ”

16. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Building the ePortfolio was a useful learning experience I feel that I have recorded good evidence that I have achieved specific learning outcomes from this option. 80% thought it was a useful learning experience 83% felt they had recorded good evidence

17. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Having clearly defined intended learning outcomes influenced the way in which I approached the option Building the ePortfolio gave me a sense of achievement 72% felt that the LOs influenced their approach 62% felt a sense of achievement 56% were proud of their portfolio

18. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 At the end of the option I spent time considering what I had learned from this option. The use of the ePortfolio facilitated the interaction between me and my supervisor 90% discussed the plan with their supervisor. However, only 39% felt it facilitated their interaction with the supervisor 93% reflected on their learning after the option

19. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 What did you most like about using the ePortfolio ? facilitated organisation and planning (n=31) usability (n=14) sense of achievement (n=11) encouraged reflection after the SSC (n=6) What did you least like about using the ePortfolio ? time consuming (n=13) technical problems (n=10) outcomes hard to define / measure (n=9) repetitive (n=8) What were the main barriers to using the ePortfolio ? access to computers / Internet (n=20) technical problems (n=11) Key points raised in the open-ended questions

20. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004Conclusions ePortfolios implemented / piloted in 2003/4ePortfolios implemented / piloted in 2003/4 Most students perceive these as beneficialMost students perceive these as beneficial But unless assessed the motivation to complete is reducedBut unless assessed the motivation to complete is reduced The ePortfolio is being piloted in other contextsThe ePortfolio is being piloted in other contexts –Dentistry –Biosciences –Contract Research Staff

21. © 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Further information: Further information: http://www.eportfolios.ac.uk S.J.Cotterill@ncl.ac.uk