1. Educational Solutions for Workforce Development Unit 8 – How to Evaluate Aim To provide an overview of how to effectively evaluate learning programmes Learning outcomes Develop evaluation strategies for your learning programmes Use a range of evaluation tools appropriate to your learning programmes

2. Educational Solutions for Workforce Development Evaluation Training evaluation is a bit like eating five portions of fruit and vegetables a day; everyone knows that they are supposed to do it, everyone says they are planning to do better in the future and few people admit to having got it right. Tamkin P, Yarnall J, Kerrin M (2002)

3. Educational Solutions for Workforce Development What is evaluation ? To ascertain or fix the value or worth of something To assess or appraise To examine a subject or rate it Make a judgement

4. Educational Solutions for Workforce Development Why do we evaluate ? To identify if educational solution meets outcomes To determine the value of educational solution To determine what worked well and what could be improved To demonstrate the impact and value of what you are doing

5. Educational Solutions for Workforce Development Test for the Best www.test4best.scot.nhs.uk)

6. Educational Solutions for Workforce Development Checklist (www.test4best.scot.nhs.uk) Evaluating the impact of a new educational solution can be complex, as a wide range of factors may need to be considered Outcomes and outputs Think about the original analysis of the issue which prompted the learning programme to be developed. Think about the links between –the issue –the outcome i.e. the anticipated change in behaviour, quality, performance and efficiency of a service or individual –the output i.e. the activity developed to achieve the outcome

7. Educational Solutions for Workforce Development Reviewing the links between the issue, the outcome and the outputs Issue There has been an increase in complaints about the way telephone calls to a clinic are handled Outcome Elimination of complaints about call handling Output Telephone skills training for staff who take and make calls How will output address outcome? Clearer frameworks for call handling will help staff deal more effectively with routine calls Improved skills will help staff deal with more complex difficult calls

8. Educational Solutions for Workforce Development Checklist (www.test4best.scot.nhs.uk) Stakeholder consultation What do they want to find out from the evaluation ? This will help establish and agree the aims of any evaluation of impact Information needs Do you have information which will act as a baseline against which you will measure development ? If you don’t have this information, what effect will it have on the aims of your evaluation ?

9. Educational Solutions for Workforce Development Checklist (www.test4best.scot.nhs.uk) Approaches to evaluation What type of information is needed ? Do you need small scale evaluation (self assessment, user feedback) or likely to require external evaluation ( strategic importance) How will the information be gathered ? Frequency and timing of evaluation ( one evaluation or a series of evaluations) Dissemination of findings Who needs to know about the findings of this evaluation ? How will you disseminate the evaluation report ?

10. Educational Solutions for Workforce Development How do we evaluate ? Who is the evaluation for? What do you need to know? Why do you need to know it? Will the evaluation make a difference?

11. Educational Solutions for Workforce Development Evaluation Models

12. Educational Solutions for Workforce Development Kirkpatrick’s Four-level approach (1959) Level 1 Reaction - what they thought and felt about the learning programme Level 3 Application – change in job behaviour and transfer of learning to the workplace Level 2 Learning - the resulting increase in knowledge or capability before and after training Level 4 Impact – the impact of learning on practice in terms of results within the organisation

13. Educational Solutions for Workforce Development Kirkpatrick’s Four level approach examples Level 1 Reaction questionnaires, post its Level 3 Observation, reflective log by participant, CPD evidence Level 2 Pre and post course knowledge questionnaire, Self assessment of knowledge, observation of performance, Level 4 Follow up participants and managers,productivity data look at costs, quality, patient experience

14. Educational Solutions for Workforce Development Learning Programme- Mentor Update Training for Nurses LEVELEVALUATION TOOL AND STRATEGY 4Discuss with participant, students and charge nurse - How learning experience impacted on service and quality of service CPD portfolio 33 month post learning to evaluate theory into practice 2Pre and post course knowledge questionnaire 1Post it feedback 3 structured questions to whole group

15. Educational Solutions for Workforce Development Evaluation models: Hamblin’s five - level approach ( 1974) Level 1 Reactions Level 2 Learning Level 3 Job behaviour Level 4Organisation – the effects on the organisation, from participant’s job to performance changes Level 5Ultimate value – the financial effects, both on the organisation and the economy Hamblin AC (1974) Evaluation and Control of Training, Maidenhead, McGraw Hill.

16. Educational Solutions for Workforce Development Summary Evaluation is an essential component of any new educational development. Evaluation needs to be considered carefully at the start of any learning programme development and well before implementation. There is no right or wrong way to evaluate Remember –Who is the evaluation for ? –What do you need to know ? –Why do you need to know it ? –Will the evaluation make a difference ?

17. Educational Solutions for Workforce Development Group work Think about a learning programme you want/are delivering Worksheet 1 –Stakeholders –Information needs –Evaluation approach –Dissemination Worksheet 2 –Evaluation strategy