1. Developing an evaluation model to assess prevention measures (EVAPREM)
Short introduction
Uku Varblane and Tarmo Puolokainen // University of Tartu
Brussels 2018

2. Outline Aim of the model Framework of the model Population survey
Impact measurement
Statistical background analysis
Classification for the prevention activities
Measurements, ideas behind the questionnaire
Conclusion

3. The purpose of the model
Systematically assess the impact and effectiveness of fire prevention activities in participating countries and regions.
GENERAL OVERVIEW- to give an assessment of preventive activities as a whole;
e.g. an “overall index”, which assesses whether the right things („effective“) and in the right amount („efficient“) are done;
SPATIAL - to give an international comparison;
the value of the “index” should be comparable between countries, therefore it should be generalizable across countries, since different countries do different activities for prevention;
TEMPORAL - to give an assessment to the trends;
the value of the index has to be found over the years, so it must be ensured that the input data is (or will be in the future) available annually;
SPECIFIC ACTIVITIES - to give an assessment to specific preventive activities;
the model must be sufficiently detailed to be able to conclude what activities and in what proportion should be changed to ensure the optimum outcome (in other words: what to change to maximize the outcome);
the components of the index should also have a meaning / interpretation on their own.
SPECIFIC TARGET GROUPS - to give an assessment to the activities across target groups (school children, elderly people etc., who would be more vulnerable).

4. The framework of the model
The index consists of three interrelated components:
Impact component, which will characterize the impact of each fire prevention activity
Population survey component, which will characterize the general knowledge and development level of fire prevention in a country in a specific point of time
General statistics component, which will characterize the long-term impact of fire prevention and development of society

5. The framework of the model
General statistics
Population survey
Impact measurement
Change in social development, external factors and practices through change in numbers of emergencies, fire fatalities, etc.
General knowledge of fire safety
Self-reported practices of the population
Each activity has (should have) an impact in the practices of population

6. The framework of the model
Inputs: resources, labour, etc.
Target groups
Population survey: general level of knowledge
Processes: prevention activities
Impact of prevention activities (result of measurement)
Outcomes: (reduced) number of emergencies, fire fatalities, etc.

7. Population survey
Population survey in each country/region would give a baseline information on:
the general knowledge,
attitudes and
self-reported behaviour.
A representative sample of each country/region was asked using the CATI method.
The total sample size from all the five countries is 5669, which is composed of 2015 respondents from Lithuania, 1722 from Estonia, 1104 from Latvia, 428 from Frederiksborg County of Denmark and 400 from Southwest Finland.

8. Statistical background analysis
Statistical background analysis would indicate the long-term impact of prevention activities as well as the general development level of the country/region.
The change in the number of emergencies, fire fatalities, causes of fires (due to carelessness, arsons, ...), etc. can be considered, but the biggest difficulty is that each country defines those measures a little bit differently.
Thus, it can only be used as a background information.

9. Classification of prevention activities
There are multiple prevention activities in every country/region
Different activities have various characteristics, which would take different methods to measure their impact
So, one should classify each activity into a larger group, and this larger group would provide a „standard“ for impact measurement
The impact measurement should be as standardized as possible and as flexible as necessary

10. General (and unknown) population
Personified activities
INDIVIDUAL – AT HOME
Individuals, households, professionals
Face-to-face
Feedback forms, calls
Standardized activities
CLOSED GROUP – WORKPLACE, SCHOOL
School children, work collectives
Face-to-face, exercise
Ex ante and ex post evaluation (tests, randomized groups)
Event-based activities
OPEN GROUP – PUBLIC ROOM
Exhibition/event visitors
Face-to-face, brochures, exhibits-boxes
“exit polls”, feedback forms
Large-scale media campaigns
POPULATION GROUPS  – MASS&SOCIAL MEDIA
Population, targeted group of population
TV- and radio advertisement, social media, newspapers
visibility evaluation, (social marketing campaign evaluation)
General information provision
ENTIRE POPULATION – MASS&SOCIAL MEDIA
TV- and radio appearance, social media, newspaper articles
dependent on each activity
Specific individual
General (and unknown) population

11. What aspects would a typical questionnaire capture?
Knowledge, attitudes and behaviour
Quality of the service (theory behind feedback surveys):
Responsiveness: the degree to which the provider reacts promptly to the customer
Professionalism
Completeness: the degree to which the service is finished
Pleasantness of support: the degree to which the provider uses suitable professional behavior and manners while working with the customer
Overall satisfaction with service
Background information of the responder

12. Interactions between the prevention activities, aggregation issues
Different components are interrelated and linked closely to each other. Thus, it should be taken into account, that these components will effect each other and thus the final results.
However, getting a „pure“ result for each component is impossible, so one has to accept this as a limitation of the model.
Different components might have different priorities, one option to take it into account is to use expert opinions (like Delphi method) to give weights to each component.

13. Steps of the analysis

14. Discussion Any questions? Comments? Ideas? Concerns?

15. Conclusion Index is divided into three components: general statistics, population survey and impact measurement of each prevention activity. Each component is interlinked with each other. The prevention activities are classified into five groups, on which are based the different approaches for measuring the impact.

16. Thank you for your attention!
Developing an evaluation model to assess prevention measures (EVAPREM)
Tarmo Puolokainen
Uku Varblane
Brussels 2018