1. Training Trainers and Educators Unit 8 – How to Evaluate
Aim
To provide an overview of how to effectively evaluate teaching and learning
Learning outcomes
Develop evaluation strategies for their teaching and learning activities
Use a range of evaluation tools appropriate to their teaching and learning activities
Slide 8.1
You will be familiar with the content of this topic. Outline tutor notes are provided but the expectation is as an experienced facilitator you may wish to add/change to suit your own and audience needs.
Within Unit 6 (Developing aims and learning outcomes and planning a learning session), one of the questions to think about in planning and designing teaching and learning was how will you evaluate it ?
This unit provides the detail and guidance to support your evaluation.
It is important that evaluation is built into programme design at the beginning

2. Evaluation
‘Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education or training, as part of a recognised process, judging its effectiveness, its efficiency and any other outcomes’.
Thorpe (1988)
Slide 8.2
Discuss how participants currently evaluate their teaching and learning activities

3. Why do we evaluate?
To identify if aims and learning outcomes have been achieved
To determine the value of the learning session
To determine what worked well and what could be improved
To demonstrate the impact of the learning in relation to knowledge, performance and service improvement
Slide 8.3
Relate back to previous session on constructive alignment, learning aims and outcomes and that evaluation allows us to identify how effective we have been in achieving aims and outcomes.
Effective evaluation identifies to what extent the educational solution meets the intended learning outcomes. Without a systematic and robust approach it is impossible to assess whether the educational solution has achieved the learning outcomes set for it
Helps determine how much or how little we value something
Helps work out what was good, not so good and what could be improved
Provides evidence to demonstrate the impact and value of what you are doing

4. Testing for the Best www.test4best.scot.nhs.uk
Slide 8.4
Superb resource for work based education. Encourage participants to visit the website.
Testing for the Best in Educational Development is an online resource developed by NES to support the design, development and delivery of educational solutions within the workplace.

5. Checklist (www.test4best.scot.nhs.uk)
Evaluating the impact of a new educational solution can be complex, as a wide range of factors may need to be considered
Outcomes and outputs
Think about the original analysis of the issue which prompted the learning programme to be developed.
Think about the links between
the issue
the outcome i.e. the anticipated change in behaviour, quality, performance and efficiency of a service or individual
the output i.e. the activity developed to achieve the outcome
Slide 8.5
Without a systematic and robust approach to evaluating a learning programme , it is impossible to assess whether it is achieving the outcomes set for it.
This checklist will help you develop the most appropriate evaluation framework for your learning programme.
The checklist has been sourced from Test4 Best - an on line resource developed by NES to support design development and commissioning for NHS Scotland.
Outcomes and outputs - The following slide gives an example

6. Reviewing the links between the issue, the outcome and the outputs
There has been an increase in complaints about the way telephone calls to a clinic are handled
Outcome
Elimination of complaints about call handling
Output
Telephone skills training for staff who take and make calls
How will output address outcome?
Clearer frameworks for call handling will help staff deal more effectively with routine calls
Improved skills will help staff deal with more complex difficult calls
Slide 8.6
Thinking about how output addresses outcome, will inform what it is you need to know from the evaluation.
Encourage participants to think about a local scenario.

7. Checklist (www.test4best.scot.nhs.uk )
Stakeholder consultation
What do they want to find out from the evaluation ?
This will help establish and agree the aims of any evaluation of impact
Information needs
Do you have information which will act as a baseline against which you will measure development ?
If you don’t have this information, what effect will it have on the aims of your evaluation?
Slide 8.7
Stakeholder consultation
Looking at previous example –
Clinic manager would want to know if there is a reduction in complaints
Staff would also want to know this information
Information needs
Baseline would be information you would gather prior to the training e.g. number of telephone complaints, staff confidence in dealing with complex calls

8. Checklist (www.test4best.scot.nhs.uk )
Approaches to evaluation
What type of information is needed ?
Do you need small scale evaluation (self assessment, user feedback) or likely to require external evaluation ( strategic importance)
How will the information be gathered ?
Frequency and timing of evaluation ( one evaluation or a series of evaluations)
Dissemination of findings
Who needs to know about the findings of this evaluation ?
How will you disseminate the evaluation report ?
Slide 8.8
Approaches to evaluation
Type of information
Number of complaints
Is framework useful for staff
Do staff feel more confident dealing with complex difficult calls
Do staff have improved skills
Type of evaluation
would likely to be local within the clinic setting
How information gathered
audit of complaints
observation of staff taking calls
Feedback from patients who speak to staff
Staff self assessment questionnaire on usefulness of framework, and their confidence levels in dealing with complex calls
Frequency and timing
Prior to training
Post training
Maybe at three monthly intervals to ensure sustainability
Dissemination of findings - who
Staff taking calls
Clinic Manager
Dissemination of findings - how
Written report
Staff meeting

9. How do we evaluate ? Who is the evaluation for?
What do you need to know?
Why do you need to know it?
Will the evaluation make a difference?
Slide 8.9
There are no right or wrong ways to evaluate.
Here are some questions to think about to help inform what your evaluation strategy might be.

10. Evaluation Models Slide 8.10
How do organisations evaluate training and development impact ?
Organisations still struggle to evaluate behaviour change and organisational/service impact effectively.
There are a range of models that you can use to evaluate your learning programmes. This session will consider two models.

11. Kirkpatrick’s Four-level approach (1959)
Impact – the impact of learning on practice in terms of
results within the organisation
Level 3
Application – change in job behaviour and transfer of
learning to the workplace
Level 2
Learning - the resulting increase in knowledge or capability
before and after training
Slide 8.11
Kirkpatrick is the best known and most widely used framework for classifying evaluation
A simple and pragmatic model to help practitioners think about training programmes
Level 1
Reaction - what they thought and felt about the learning
programme

12. Kirkpatrick’s Four level approach examples
Follow up participants and managers ,productivity data
look at costs, quality, patient experience
Level 3
Observation, reflective log by participant,
CPD evidence
Level 2
Pre and post course knowledge questionnaire,
Self assessment of knowledge, observation of performance,
Slide 8.12
Level 1: Measured at the end of the learning programme, normally by the use of reaction questionnaires.
UNIT 8: Handout 1 and 2 Evaluation forms are examples. Using Post its - Could ask participants to write their key learning point and an action they are taking forward back in the workplace. You could ask participants what their light bulb learning moment was
Level 2: Measured at the end of the learning programme looking at changes in knowledge, skills or attitude.
UNIT 8:Handouts 3 and 4 are examples or observation of performance of skill during the learning programme.
Level 3: Measured at defined points after the learning programme dependant on time required to implement changes. This can be variable dependant on both internal and external factors which can support or delay implementation. Learner could complete a reflective documentation or an action plan relating to how they wish to apply their learning into practice or workplace. If action planning is used, it can be followed up through performance development reviews.
Level 4: Measured at defined points after the learning programme dependant on time required to implement changes. This can be variable dependant on both internal and external factors which can support or delay implementation. Evaluation looks at the impact of the learning on service needs. It could measure % increase/reduction in relevant area. It looks at hard data e.g. costs, time, and quality soft findings e.g. patient satisfaction, managers questionnaires.
Level 1
Reaction questionnaires, post its

13. EVALUATION TOOL AND STRATEGY
Learning Programme- Mentor Update Training for Nurses
LEVEL
EVALUATION TOOL AND STRATEGY 4
Discuss with participant, students and charge nurse -
How learning experience impacted on service and quality of service
CPD portfolio 3
3 month post learning to evaluate theory into practice 2
Pre and post course knowledge questionnaire 1
Post it feedback
3 structured questions to whole group
Slide 8.13
This is an example of how one facilitator used Kirkpatrick's model to develop an evaluation strategy for their mentor update training for nurses.

14. Evaluation models: Hamblin’s five - level approach ( 1974)
Level 1 Reactions
Level 2 Learning
Level 3 Job behaviour
Level 4 Organisation – the effects on the organisation, from participant’s job to
performance changes
Level 5 Ultimate value – the financial effects, both
on the organisation and the economy
Hamblin AC (1974) Evaluation and Control of Training, Maidenhead, McGraw Hill.
Slide 8.14
Also many models used Kirkpatrick's framework as a starting point.
The book Kirkpatrick and beyond : A review of models of training evaluation ( on your reference list) discusses 8 such models.
This is one.
Hamblin was one of the first to modify Kirkpatrick’s model. The first three levels correspond closely , the final level splits into two.
There are also alternative models that focus on purpose of evaluation and use alternative measures. These also are discussed in detail within this book.

15. Group work Think about a learning programme you want/are delivering
Worksheet 1
Stakeholders
Information needs
Evaluation approach
Dissemination
Worksheet 2
Evaluation strategy
Slide 8.15
Distribute Handouts 5 and 6

16. Summary
Evaluation is an essential component of any teaching and learning programme
Evaluation needs to be considered carefully at the start of any learning programme development and well before implementation.
There is no right or wrong way to evaluate
Remember
Who is the evaluation for ?
What do you need to know ?
Why do you need to know it ?
Will the evaluation make a difference ?