TYNE AND WEAR FIRE AND RESCUE SERVICE ‘Creating The Safest Community’ Evaluation in the Fire and Rescue Service Vicki Parnaby

National drivers  Performance Management  Integrated Risk Management Planning  Comprehensive Area Assessment  Value for Money  Local Strategic Partnerships

Evaluation in the FRS AC examined national picture –improving, but… –relatively weak processes in place

Evaluation in the FRS Strategic planning and performance management improving Performance monitoring in place in all FRSs 75% FRSs use performance information to drive improvements Evaluation of CS work improving –especially in work with young people Some FRSs introduced evaluation of benefits of joint work However…..

Evaluation in the FRS Not enough FRSs evaluate projects, therefore no confidence of; –targeting the right groups –achieving impact

Evaluation in the FRS “All services can do more to make sure they are targeting the right people and achieving the maximum impact from their investment” “Better strategy and evaluation are required to ensure that fire services are demonstrating value for money” ( source: Audit Commission 09, Analysis of fire and rescue authorities performance )

Our journey…..

Evaluation in TWFRS Background Previous measures of success Anecdotal Evidence Repetition of work Relatively low evaluation rate for community based projects

TWFRS Evaluation - Improvement Project Research Consultation with practitioners – key! Focus Group – Community Safety practitioners and managers Questionnaire to sample of Service Delivery staff Survey of attitudes / opinions towards evaluation Asked for suggestions / recommendations

TWFRS Evaluation - Improvement Project Obstacles to Evaluation: Lack of training Lack of confidence in evaluating Time consuming – rather be out doing it ‘Difficult to evaluate’ some initiatives Lack of understanding why we evaluate – auditors? Process of evaluation unhelpful – 72% focus group said evaluation did not assist their role

TWFRS Evaluation - Improvement Action Standardise evaluation process across TWFRS Produce a simple evaluation framework, including: Step by step guide how to evaluate a project List of evaluation techniques available Easily accessible suite of templates Guidance on working out costs Guidance on data protection Publish evaluation results and make recommendations clear Develop a library to house all evaluations centrally Provide training to all relevant staff – awareness sessions

TWFRS Evaluation Toolkit

Evaluation of Toolkit - Results Is there a structured and easy to follow evaluation process? 0% 100% Are you aware how to evaluate projects properly? 27% 100% Have you received sufficient training to allow you to carry out project evaluation? 20% 100% Old Process New Process

Does the project evaluation Process assist your role? 23%97% Is sufficient guidance and templates available?6%100% Do you understand why TWFRS need to evaluate projects?66%93% Evaluation of Toolkit - Results

Evaluation Library

Evaluation – Impact Outputs: Over 150 evaluations carried out and published using toolkit Outcomes: Evaluations produced which have: - supported external funding application / partnership work - allowed successful projects to be rolled out across service - identified good practice which informs and shapes future of project (via interim evaluation) - demonstrated considerable efficiency savings - led to change in approach / strategy - demonstrated inefficient use of resources and allowed altered level of involvement - produced recommendations used to improve TWFRS operational resilience - assessed an event where outcomes / impact are traditionally difficult to evaluate Impact: More targeted and efficient use of resources, ensuring we continuously improve the service we provide

TWFRS Evaluation Toolkit ‘Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted’ - Albert Einstein

Evaluation in the FRS Way Forward…..

TWFRS Evaluation Showcase Event - May

Evaluation Showcase Event Design and development of a practical toolkit to assist evaluation in the FRS; Training non-evaluators to evaluate; Disseminating good practice; Engaging practitioners Providing VfM assessment as an integral part of evaluation

Evaluation Showcase Event Attended by 28 FRS from England, Scotland, Wales and Northern Ireland 67 delegates Evaluation in FRS Evolution of evaluation Question and answer session Workshops – delivered by practitioners Questions and answer session

Showcase Event – Evaluation Findings Quantitative Meeting objectives – 96% Most useful aspect – 53% (evaluation toolkit) Greater understanding of evaluation – 88% Qualitative Structured debrief

Qualitative Feedback “I believe your toolkit was the first one (I’ve ever seen) developed for a FRS by the FRS personnel, and it gave me a good idea of issues that should be covered in the GMFRS toolkit as well. Our biggest challenge was to develop a toolkit for non-experienced personnel, and it was very encouraging to see that is exactly what you had done.” "As a result of this I amended the project I was working on to include better evaluation” “We are looking at a online toolkit that has been assisted by seeing a working system” “I now use the project start up form for all of our new projects within the education team and we are able to look at the process of evaluation and reporting much more easily as a result of adopting the templates you provided on the day.”

Evaluation in the FRS …….Future?