1. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Electronic Portfolio Approaches to Support Development of Contract Research Staff Faculty of Medical Sciences University of Newcastle upon Tyne * School of Medical Education Development + Career Development Coordinator, Faculty Research Strategy Group Faculty Research Strategy Group Simon Cotterill*, Lesley Heseltine+, Paul Drummond*, Tony McDonald* Newcastle University

2. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Overview 1.Electronic portfolios 2.Contract Research Staff (CRS) 3.Electronic Portfolio Approaches for CRS 4.Feasibility Study, Evaluation & Future Plans 5.Questions / Discussion !

3. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 What are Portfolios ? UnstructuredFormativeSample Best work Factual / Quantitative Learner Owned StructuredSummative All work Representative Reflective / Interpretive Employer Owned

4. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 What is an Electronic Portfolio ? 1.A computer application which stores the user’s information on a local drive. 2.A static Web page eg. CV on the Internet 3.A dynamic database-driven system that is accessed over a network.

5. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Highly customisableHighly customisable Multi-purpose eg. formative & summative = reduced duplicationMulti-purpose eg. formative & summative = reduced duplication Multiple structures / viewsMultiple structures / views Easier cross-referencingEasier cross-referencing SharableSharable Secure access from a range of locationsSecure access from a range of locations TransportableTransportable SearchableSearchable Reduced adminReduced admin Not left on the bus !Not left on the bus ! 10 ‘value added’ features of an online approach to portfolios

6. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Managed Environments for Portfolio-based Reflective Learning Integrated Support for Evidencing Outcomes. An FDTL-4 project: Newcastle University (lead site) Leeds University Sheffield University Dundee University (for consultancy) Developing Web based portfolios to support reflective approaches for evidencing the attainment of programme outcomes in undergraduate Medicine. Closely integrated with on-line curricula and study guides to become an integral component of managed learning environments for Medicine.

7. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Aims of the FDTL-4 project To develop an on-line portfolio system to support a reflective approach to evidencing the attainment of programme outcomes.To develop an on-line portfolio system to support a reflective approach to evidencing the attainment of programme outcomes. To promote the development of the reflective capabilities of medical students, giving greater responsibility for managing their own learning and preparing for aspects of work-based and lifelong learning.To promote the development of the reflective capabilities of medical students, giving greater responsibility for managing their own learning and preparing for aspects of work-based and lifelong learning. To develop strategies to facilitate assessment of curriculum outcomes that are not amenable to traditional instruments of assessment.To develop strategies to facilitate assessment of curriculum outcomes that are not amenable to traditional instruments of assessment.

8. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Managed Environments for Portfolio-based Reflective Learning - Integrated Support for Evidencing Outcomes. Managed Environments for Portfolio-based Reflective Learning - Integrated Support for Evidencing Outcomes. Generic ePortfolio Stand-alone Portfolio framework Customisation: Component selection Outcomes / skills sets Terminology MBBS ePortfolio Integrated into the VLE Piloting in 2003/4: Years 1-2 diary / log-book Year 3 rotation in Child Health Year 4 placements (SSCs) FDTL-4 Newcastle development work:

9. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Current Research and Evaluation Studies -Questionnaires (before and after using the ePortfolio) change in awareness of intended learning outcomes ?change in awareness of intended learning outcomes ? factors influencing use of the ePortfolio:factors influencing use of the ePortfolio: - Continuing Learning Inventory (Oddi, 1986) - prior reflective practice - attitudes to computers in education - demographic data attitudes and perceptions of using the electronic portfolioattitudes and perceptions of using the electronic portfolio perception of impact of using the electronic portfolioperception of impact of using the electronic portfolio evaluation of technical features and ‘usability’evaluation of technical features and ‘usability’ -Focus groups - Supervisors’ Questionnaire (random sample) 1.Evaluation of an electronic portfolio to facilitate reflective learning in stage 4 medical students during their student selected components. Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR

10. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Current Research and Evaluation Studies (cont.) -Focus groups -Questionnaire 2. Evaluation of a Phase I Portfolio/Log Book (paper and electronic versions). Cotterill SJ, Sarma S, McDonald AM, Bradley P 3. Feasibility Study of an Electronic Portfolio to Support Faculty Contract Research Staff (CPD) Cotterill SJ, Heseltine L, Drummond P, McDonald AM -Piloting -Structured telephone interviews

11. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Using the ePortfolio in Other Contexts Dentistry A national consortium of undergraduate dental schools and postgraduate VDTsA national consortium of undergraduate dental schools and postgraduate VDTs Postgraduate Medicine Building on the MyPIMD projectBuilding on the MyPIMD project Contract Research Staff Feasibility studyFeasibility study Biosciences at Newcastle A ‘mini-project’ funded by LTSN-BSA ‘mini-project’ funded by LTSN-BS

12. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 2. Contract Research Staff (CRS)

13. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Key Problems for CRS Insecurity in employment and unpredictability of future fundingInsecurity in employment and unpredictability of future funding Lack of obvious, visible career pathsLack of obvious, visible career paths Lack of career development opportunitiesLack of career development opportunities Feelings of being a ‘second class citizen’ with little recognition or statusFeelings of being a ‘second class citizen’ with little recognition or status Lack of consistency in terms and conditions of serviceLack of consistency in terms and conditions of service

14. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Supporting CRS within the Faculty of Medical Sciences at Newcastle A scheme was developed in early 2003 designed to enhance career progression and job security of research staff on fixed term contracts. The scheme adheres to the principals of the Concordat and has three strands: –Active Career Management –Continuity of Employment –Knowledge Exchange

15. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Supporting CRS within the Faculty of Medical Sciences at Newcastle #2 First steps: Knowing who our CRS areKnowing who our CRS are Views of our CRSViews of our CRS –Contract Research On-line Survey (CROS) – Contract Research Staff Working Party Effective communication with CRSEffective communication with CRS

16. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Supporting CRS within the Faculty of Medical Sciences at Newcastle #3 Project Launched May 2003 Active Career ManagementActive Career Management –Induction of new CRS –Performance and Development Review –Encouragement of Independence Continuity of EmploymentContinuity of Employment –Bridging Funding –Support for Fellowship Applications Knowledge ExchangeKnowledge Exchange –Programme of Workshops –CRS Web site electronic portfolios ?

17. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 3. Electronic Portfolio Approaches for CRS

18. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Research Career Builder - Skills Matrix http://www.shef.ac.uk/~gmpcrs/rcb.html An FDTL project including input from Nigel Beacham, Crispin Coombs, Abigail Davis, Stephen Hirst, Ursula Iliffe, Karen Kellard, Margaret Parkin, Claire Palmer, Jenny Park, Sheila Price, Joan Stenson, Patrick Webb This is a Skills Matrix with skills in 3 different areas: Core Research Skills (8)Core Research Skills (8) Personal and Professional Skills (8)Personal and Professional Skills (8) Specialist SkillsSpecialist Skills

19. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

20. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

21. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

22. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

23. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

24. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

25. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004

26. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 4. Feasibility Study, Evaluation & Future Plans Small scale piloting with volunteers (n=11)Small scale piloting with volunteers (n=11) Brief instructions on using the portfolioBrief instructions on using the portfolio “…please also try and think about how this might be useful from the viewpoint of new / junior CRS. Please also consider the following issues: were the tools easy to use, was navigation easy, did you experience any particular problems ?”“…please also try and think about how this might be useful from the viewpoint of new / junior CRS. Please also consider the following issues: were the tools easy to use, was navigation easy, did you experience any particular problems ?” Structured telephone interviews (n=7 at time of analysis) + Email feedbackStructured telephone interviews (n=7 at time of analysis) + Email feedback

27. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Usability / Navigability All gave positive feedback on usability / navigability “The beauty of this tool is that, being web-based, it is easily accessible. Once familiar with the sections within the tool I found it easy to use and simple to navigate.” However: “It appears very helpful, but a bit daunting when considering what can be filled in to get an up to date record...”

28. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 “The only other comment I had was that it was sometimes a bit tricky to get back to the main page-and I had to look for the eportfolio link. A link back to that page in the main body of the page may be helpful for IT phobes like myself.” CV “Qualifications: very British, very difficult to fill in for non- native researchers with different educational histories..” “Good but adding qualifications is tedious – need a facility to add multiple qualifications on a single screen”

29. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 “Being a Post-Doc researcher myself this is a good idea as it would enable me to quickly reference courses and information which would otherwise be lounging in a file somewhere. I have no doubt that the eportfolio will be useful when CRS are applying for new positions, especially the CV option.” “When I got to the PDF view of my CV and My Skills and Achievements I discovered loads of grammatical and spelling mistakes. Is there a way to have a spelling grammar check running as you enter the information?”

30. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Cross-referencing records of courses and presentations with the Skills matrix 6/7 found this straight forward. 1 person found this difficult: “I had difficulty understanding what was happening with cross referencing and some instruction in this would be helpful. Perhaps it would be useful to add examples of inputs for each section, or better still, have a model eportfolio which could be looked at.”

31. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 “I particularly liked the way it records conferences and courses-as I can never remember what I have done and when. The way it links to research skills is really good and I can see benefits of it for my own personal and professional development and as a resource-e.g.. if applying for other jobs and need to articulate strengths and weaknesses and provide evidence. The great thing about it is you can just enter a short amount of information at a time and come back to it later.”

32. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 The volunteers were able to use the ePortfolio for CRSThe volunteers were able to use the ePortfolio for CRS The volunteers generally found it straight forward but did provide feedback on how navigability of specific parts of the ePortfolio could be improved.The volunteers generally found it straight forward but did provide feedback on how navigability of specific parts of the ePortfolio could be improved. The FDTL4 ePortfolio framework developed for medicine has proved applicable in another context.The FDTL4 ePortfolio framework developed for medicine has proved applicable in another context. Conclusions

33. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Limitations: Volunteers were self-selected and included more ‘senior’ CRS (mostly from science backgrounds)Volunteers were self-selected and included more ‘senior’ CRS (mostly from science backgrounds) Small numbers: more large scale piloting and systematic evaluation is requiredSmall numbers: more large scale piloting and systematic evaluation is required Electronic portfolio approaches may also be applied to staff in HE for a range of purposes.Electronic portfolio approaches may also be applied to staff in HE for a range of purposes.

34. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Future Plans Automatically include records of course attendance (where booked and confirmed via the CRS Website).Automatically include records of course attendance (where booked and confirmed via the CRS Website). Make the skills records more sophisticated eg. recording levels of competency.Make the skills records more sophisticated eg. recording levels of competency. Link into ‘myProfiles’ data for publications and other information (a small minority of CRS may use the myProfiles system to record publications and other PDR & RAE information).Link into ‘myProfiles’ data for publications and other information (a small minority of CRS may use the myProfiles system to record publications and other PDR & RAE information).

35. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Future Plans #2 Explore additional linkages with MIS information eg. HR data and other course attendance information recorded by Staff Development and other units in the University.Explore additional linkages with MIS information eg. HR data and other course attendance information recorded by Staff Development and other units in the University. Incorporate other eportfolio developments from out FDTL4 ePortfolios project (http://www.eportfolios.ac.uk)Incorporate other eportfolio developments from out FDTL4 ePortfolios project (http://www.eportfolios.ac.uk)http://www.eportfolios.ac.uk Research and evaluation of using ePortfolios for CRSResearch and evaluation of using ePortfolios for CRS

36. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 further information http://www.eportfolios.ac.uk http://medical.faculty.ncl.ac.uk/crs Simon Cotterill: S.J.Cotterill@ncl.ac.uk

37. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 The Personal Profile at Newcastle RAE focus. Records:RAE focus. Records: –Teaching and Learning (UG and PG teaching and teaching development) –Research (Research programmes, publications, awards, staff and students) –Third Strand (spinouts, outreach work and industrial collaborations) –Roles (administrative duties, training and support roles) Integrates centrally held data sets: RIS awards data under "Research Grants held"RIS awards data under "Research Grants held" Graduate School records of PG student supervision under "Research Students".Graduate School records of PG student supervision under "Research Students".

38. © 2004, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios for CRS SEDA Conference 2004 Portfolios for HE Staff Baume D, Yorke M. The reliability of assessment by portfolio on a course to develop and acredit teachers in higher education. Studies in higher education. 2002; 27: 7-25 HE Staff may also benefit from electronic portfolio approaches including personal profiles and research plans which tie in to management information systems and networked learning environments.