Presentation is loading. Please wait.

Presentation is loading. Please wait.

Training Trainers and Educators Unit 8 – How to Evaluate

Similar presentations


Presentation on theme: "Training Trainers and Educators Unit 8 – How to Evaluate"— Presentation transcript:

1 Training Trainers and Educators Unit 8 – How to Evaluate
Aim To provide an overview of how to effectively evaluate teaching and learning Learning outcomes Develop evaluation strategies for their teaching and learning activities Use a range of evaluation tools appropriate to their teaching and learning activities Slide 8.1 You will be familiar with the content of this topic. Outline tutor notes are provided but the expectation is as an experienced facilitator you may wish to add/change to suit your own and audience needs. Within Unit 6 (Developing aims and learning outcomes and planning a learning session), one of the questions to think about in planning and designing teaching and learning was how will you evaluate it ? This unit provides the detail and guidance to support your evaluation. It is important that evaluation is built into programme design at the beginning

2 Evaluation ‘Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education or training, as part of a recognised process, judging its effectiveness, its efficiency and any other outcomes’. Thorpe (1988) Slide 8.2 Discuss how participants currently evaluate their teaching and learning activities

3 Why do we evaluate? To identify if aims and learning outcomes have been achieved To determine the value of the learning session To determine what worked well and what could be improved To demonstrate the impact of the learning in relation to knowledge, performance and service improvement Slide 8.3 Relate back to previous session on constructive alignment, learning aims and outcomes and that evaluation allows us to identify how effective we have been in achieving aims and outcomes. Effective evaluation identifies to what extent the educational solution meets the intended learning outcomes. Without a systematic and robust approach it is impossible to assess whether the educational solution has achieved the learning outcomes set for it Helps determine how much or how little we value something Helps work out what was good, not so good and what could be improved Provides evidence to demonstrate the impact and value of what you are doing

4 Testing for the Best www.test4best.scot.nhs.uk
Slide 8.4 Superb resource for work based education. Encourage participants to visit the website. Testing for the Best in Educational Development is an online resource developed by NES to support the design, development and delivery of educational solutions within the workplace.

5 Checklist (www.test4best.scot.nhs.uk)
Evaluating the impact of a new educational solution can be complex, as a wide range of factors may need to be considered Outcomes and outputs Think about the original analysis of the issue which prompted the learning programme to be developed. Think about the links between the issue the outcome i.e. the anticipated change in behaviour, quality, performance and efficiency of a service or individual the output i.e. the activity developed to achieve the outcome Slide 8.5 Without a systematic and robust approach to evaluating a learning programme , it is impossible to assess whether it is achieving the outcomes set for it. This checklist will help you develop the most appropriate evaluation framework for your learning programme. The checklist has been sourced from Test4 Best - an on line resource developed by NES to support design development and commissioning for NHS Scotland. Outcomes and outputs - The following slide gives an example

6 Reviewing the links between the issue, the outcome and the outputs
There has been an increase in complaints about the way telephone calls to a clinic are handled Outcome Elimination of complaints about call handling Output Telephone skills training for staff who take and make calls How will output address outcome? Clearer frameworks for call handling will help staff deal more effectively with routine calls Improved skills will help staff deal with more complex difficult calls Slide 8.6 Thinking about how output addresses outcome, will inform what it is you need to know from the evaluation. Encourage participants to think about a local scenario.

7 Checklist (www.test4best.scot.nhs.uk )
Stakeholder consultation What do they want to find out from the evaluation ? This will help establish and agree the aims of any evaluation of impact Information needs Do you have information which will act as a baseline against which you will measure development ? If you don’t have this information, what effect will it have on the aims of your evaluation? Slide 8.7 Stakeholder consultation Looking at previous example – Clinic manager would want to know if there is a reduction in complaints Staff would also want to know this information Information needs Baseline would be information you would gather prior to the training e.g. number of telephone complaints, staff confidence in dealing with complex calls

8 Checklist (www.test4best.scot.nhs.uk )
Approaches to evaluation What type of information is needed ? Do you need small scale evaluation (self assessment, user feedback) or likely to require external evaluation ( strategic importance) How will the information be gathered ? Frequency and timing of evaluation ( one evaluation or a series of evaluations) Dissemination of findings Who needs to know about the findings of this evaluation ? How will you disseminate the evaluation report ? Slide 8.8 Approaches to evaluation Type of information Number of complaints Is framework useful for staff Do staff feel more confident dealing with complex difficult calls Do staff have improved skills Type of evaluation would likely to be local within the clinic setting How information gathered audit of complaints observation of staff taking calls Feedback from patients who speak to staff Staff self assessment questionnaire on usefulness of framework, and their confidence levels in dealing with complex calls Frequency and timing Prior to training Post training Maybe at three monthly intervals to ensure sustainability Dissemination of findings - who Staff taking calls Clinic Manager Dissemination of findings - how Written report Staff meeting

9 How do we evaluate? Who is the evaluation for?
What do you need to know? Why do you need to know it? Will the evaluation make a difference? Slide 8.9 There are no right or wrong ways to evaluate. Here are some questions to think about to help inform what your evaluation strategy might be.

10 Evaluation Models Slide 8.10
How do organisations evaluate training and development impact ? Organisations still struggle to evaluate behaviour change and organisational/service impact effectively. There are a range of models that you can use to evaluate your learning programmes. This session will consider two models.

11 Kirkpatrick’s Four-level approach (1959)
Impact – the impact of learning on practice in terms of results within the organisation Level 3 Application – change in job behaviour and transfer of learning to the workplace Level 2 Learning - the resulting increase in knowledge or capability before and after training Slide 8.11 Kirkpatrick is the best known and most widely used framework for classifying evaluation A simple and pragmatic model to help practitioners think about training programmes Level 1 Reaction - what they thought and felt about the learning programme

12 Kirkpatrick’s Four level approach examples
Follow up participants and managers ,productivity data look at costs, quality, patient experience Level 3 Observation, reflective log by participant, CPD evidence Level 2 Pre and post course knowledge questionnaire, Self assessment of knowledge, observation of performance, Slide 8.12 Level 1: Measured at the end of the learning programme, normally by the use of reaction questionnaires. UNIT 8: Handout 1 and 2 Evaluation forms are examples. Using Post its - Could ask participants to write their key learning point and an action they are taking forward back in the workplace. You could ask participants what their light bulb learning moment was Level 2: Measured at the end of the learning programme looking at changes in knowledge, skills or attitude. UNIT 8:Handouts 3 and 4 are examples or observation of performance of skill during the learning programme. Level 3: Measured at defined points after the learning programme dependant on time required to implement changes. This can be variable dependant on both internal and external factors which can support or delay implementation. Learner could complete a reflective documentation or an action plan relating to how they wish to apply their learning into practice or workplace. If action planning is used, it can be followed up through performance development reviews. Level 4: Measured at defined points after the learning programme dependant on time required to implement changes. This can be variable dependant on both internal and external factors which can support or delay implementation. Evaluation looks at the impact of the learning on service needs. It could measure % increase/reduction in relevant area. It looks at hard data e.g. costs, time, and quality soft findings e.g. patient satisfaction, managers questionnaires. Level 1 Reaction questionnaires, post its

13 EVALUATION TOOL AND STRATEGY
Learning Programme- Mentor Update Training for Nurses LEVEL EVALUATION TOOL AND STRATEGY 4 Discuss with participant, students and charge nurse - How learning experience impacted on service and quality of service CPD portfolio 3 3 month post learning to evaluate theory into practice 2 Pre and post course knowledge questionnaire 1 Post it feedback 3 structured questions to whole group Slide 8.13 This is an example of how one facilitator used Kirkpatrick's model to develop an evaluation strategy for their mentor update training for nurses.

14 Level 5 Ultimate value – the financial effects, both
Evaluation models: Hamblin’s five - level approach ( 1974) Level 1 Reactions Level 2 Learning Level 3 Job behaviour Level 4 Organisation – the effects on the organisation, from participant’s job to performance changes Level 5 Ultimate value – the financial effects, both on the organisation and the economy Hamblin AC (1974) Evaluation and Control of Training, Maidenhead, McGraw Hill. Slide 8.14 Also many models used Kirkpatrick's framework as a starting point. The book Kirkpatrick and beyond : A review of models of training evaluation ( on your reference list) discusses 8 such models. This is one. Hamblin was one of the first to modify Kirkpatrick’s model. The first three levels correspond closely , the final level splits into two. There are also alternative models that focus on purpose of evaluation and use alternative measures. These also are discussed in detail within this book.

15 Group work Think about a learning programme you want/are delivering
Worksheet 1 Stakeholders Information needs Evaluation approach Dissemination Worksheet 2 Evaluation strategy Slide 8.15 Distribute Handouts 5 and 6

16 Summary Evaluation is an essential component of any teaching and learning programme Evaluation needs to be considered carefully at the start of any learning programme development and well before implementation. There is no right or wrong way to evaluate Remember Who is the evaluation for? What do you need to know? Why do you need to know it? Will the evaluation make a difference?


Download ppt "Training Trainers and Educators Unit 8 – How to Evaluate"

Similar presentations


Ads by Google