1. Educational Solutions for Workforce Development Psychology Using Portfolios to Evaluate Leadership Competence: Can reflective learning be combined with assessment? National Health Systems Education Scotland 2007

2. Educational Solutions for Workforce Development Psychology Reflective Learning.

3. Educational Solutions for Workforce Development Psychology Reflective Learning – Educational Rationale: Concrete Experience Learner Conceptualizing Active experimentation Reflection (Adapted from Kolb, 1984)

4. Educational Solutions for Workforce Development Psychology Why use portfolios to evaluate evidence of Leadership competence? Strengths : authenticity. Portfolio development charts growth over time → more authentic form of assessment which accurately represents learner ability. (e.g. by drawing on more than one piece of evidence) (Chang, 2001) Within leadership education, portfolios often are perceived as evaluation tools that can be used to assess performance in authentic contexts. (Driessen et al 2005) Duque (2003): The strength of portfolios is in the assessment of skills & attitudes, which are difficult to measure with more traditional assessment methods/tools.

5. Educational Solutions for Workforce Development Psychology Miller’s Assessment Triangle

6. Educational Solutions for Workforce Development Psychology Portfolios – Weaknesses: Debate revolves around reliability & validity issues, also what constitutes ‘good’ evidence. Does portfolio assessment simply measure ability to write about professional practice rather than a standard of practice itself ? (e.g. McMullan et al., 2003) Time & effort required for portfolio construction plus the uncertainty about what to include as evidence are germane issues.

7. Educational Solutions for Workforce Development Psychology Portfolio structure As recommended by Abrami and Barrett (2005); Klenowski, Askew and Carnell (2006), The NES leadership Module portfolios comprise: Throughout Module, learners must: Monitor progress against each competence statement listed; Gather evidence to support their judgement in the portfolio; Seek Guidance where appropriate. Experiential evidence of leadership/supervisory competence; Reasons for selecting that evidence; What the portfolio creator learned.

8. Educational Solutions for Workforce Development Psychology Competency framework for portfolio evidence: Example Statement of Supervisory Competence Suggested sources of Evidence Monitor subordinate’s well-being and raise concerns with subordinate as appropriate. Description of specific examples which have occurred, reinforced by extracts from your supervision notes. The important point is to illustrate how you picked up cues which may indicate subordinate anxieties or problems which could influence their performance or learning experience.

9. Educational Solutions for Workforce Development Psychology Portfolio evaluation / Review process. Criteria for review: New process; still evolving. Facilitator will consider – Does evidence offered: Adequately illustrate specific statement of leadership/supervisory competence? Explicitly link leadership/supervisory theory & practice? Adequately illustrate reflective learning (e.g. what happened, how did I deal with it; what might I do differently next time?)

10. Educational Solutions for Workforce Development Psychology Portfolio evaluation / Review process II – Answers to the foregoing questions will be used to classify portfolios as: “ Excellent/highly satisfactory”, “Satisfactory”, & “Need for revision/resubmission”.

11. Educational Solutions for Workforce Development Psychology Can portfolios combine reflective learning & assessment? McMullan et al. (2003) expressed concern over impact of assessment purpose on selection of portfolio evidence. How do we address this? NES offers learner control over portfolio content, by separating: Private reflective learning – online Reflective Journal; “Shared” evidence of progress. (Portfolio) Advantages: a)Reflective Journal remains confidential to author – less inhibited; b)Having to provide a rationale for selection of evidence helps consolidate reflective learning; c)Combining evidence from different sources (e.g. Reflective Journal & Supervision Notes) integrates learning & practice.

12. Educational Solutions for Workforce Development Psychology Feedback from pilot external reviewers: Quotations from personal reflective logs & specific well-grounded examples of good practice worked particularly well. Valuable evidence was often provided on progression & theory- practice links. Participants seem to have benefited from the guidance & examples of good practice offered through Blackboard by the facilitator. In general, manageable & credible system of assessing competence is evolving.

13. Educational Solutions for Workforce Development Psychology Conclusions: Feedback from systematic piloting suggests developing process viable both for learners & facilitators. Ability to “cut & paste” selected material from private online Reflective Journal enables us to combine reflective learning & valid assessment.

14. Educational Solutions for Workforce Development Psychology Issues to consider - Weaknesses in current method: Portfolio review process based on self-report; Portfolios only inform us about…’competencies in an indirect way – there is no observation’ (Delandshere and Arens, 2003) Future: Need triangulation with additional “evidence”:  e.g. observation of leadership/supervision in practice. Feedback from subordinates of the leader/supervisor? (360 assessment) Other?

15. Educational Solutions for Workforce Development Psychology References Abrami, P.C. and Barrett, H. (2005). ‘Directions for research and development on electronic portfolios’. Canadian Journal of Learning and Technology. 31(3). Online version. Chang, C. (2001). Construction and evaluation of a web-based learning portfolio system: An electronic assessment tool. Innovations in Education and Teaching International. 38(2): 144-155. Delandshere, G. and Arens, S.A. (2003). ‘Examining the quality of the evidence in preservice teacher portfolios’. Journal of Teacher Education. 54(1): 57-73. Driessen, E. van der Vleuten, C., Schurwirth, L., van Tartwijk, J. and Vermunt, J. (2005). ‘The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: A case study’. Medical Education. 39(2): 214-220. Duque, G. (2003). ‘Web-based evaluation of medical clerkships: A new approach to immediacy and efficacy of feedback and assessment’. Medical Teacher. 25(5): 510-514. Hall-Marley, S. (2001). ‘Supervisor Feedback Form’. Available online at: www.cfalendar.com [Last accessed April 2006]. www.cfalendar.com

16. Educational Solutions for Workforce Development Psychology References – Cont’d Klenowski, V., Askew, S. and Carnell, E. (2006). ‘Portfolios for learning, assessment and professional development in higher education’. Assessment and Evaluation in Higher Education. 31(3): 267-286. McMullan, M., Endacott, R., Gray, M., Jasper, M., Miller, C., Scholes, J. (2003). ‘Portfolios and assessment of competence: A review of the literature’. Journal of Advanced Nursing. 41(3): 283-294. Miller, G.E. (1990). ‘The assessment of clinical skills/competence/performance’. Academic Medicine (supplement). 65: S63-S7. Rees, C. and Sheard, C. (2004). ‘The reliability of assessment criteria for undergraduate medical students’ communication skills portfolios: the Nottingham experience’. Medical Education. 38(2): 138-144.