1 Evaluating Communication Plans Cvetina Yocheva Evaluation Unit DG REGIO 02/12/2009

2 Why evaluation? To improve the policy for the future Critical (reveals what works and what does not) Independent (reliable & objective) Democratic accountability Optimal allocation of resources (taxpayers money) Political pressure from various sides on showing results

3 Structuring the evaluation 4 criteria: relevance, effectiveness efficiency, utility (sustainability, acceptance) Needs Objectives Resources Outputs Results/ Outcomes Impacts Expected/unexpected Short term/long term

4 Evaluation questions Must correspond to real need for information Can concern Relevance: are targets justified in relation to needs? Effectiveness: to what extent targets are achieved? Efficiency: objectives achieved at the lowest cost? Utility: are expected/unexpected effects satisfactory for direct/indirect beneficiaries? Must be answerable given available data The importance vary depending on the type of evaluation (ex ante, formative, ex post)

5. Means: Quantitative Goals: Qualitative Printed material N° of diffused brochures, magazines, etc AwarenessAwareness VisibilityVisibility ImageImage Target audience quotes Cohesion Policy Press relations N° of interviews, TV/radio broadcasts Space in the media N° of interviews, TV/radio broadcasts etc. as a % of the total Attitudes PerceptionPerception PreferencesPreferences Intentions to change behaviourIntentions to change behaviour Target audience Mobilising all contacts Publicity cost Cost of appearance in newspapers, TV/radio, etc. Behaviour Has the target audience changed behaviour as intended? Content analysis Key messages, positive/negative, sources

6 How to measure progress? Goal: raise awareness, transparency, visibility Output indicators N° of people dealt with information (web, info points etc) N° of events/presentations/press releases/interviews Result indicators Local press coverage by OP (targets not set yet) Impact indicators (data via surveys) Level of overall awareness of the role of the OP Level of awareness of individual Funds (ERDF, ESF) Satisfaction with seminars and documentation Source: Communication Plan for BMW OP, Ireland baseline - mid-term target - final target

7 Some examples Q: To what extend objectives have been achieved at an acceptable cost? (efficiency) Indicator: cost of achieving the objective Q: What is the level of awareness among SMEs of the European Small Business Act? (effectiveness) Indicator: level of awareness/understanding Intensity scale: ex. Raise awareness objective Knows the name of the policy/organisation/fund, etc Quotes spontaneously its name Top of mind quote Deeper knowledge (cite specific activities etc.)

8 Methods: qualitative data Contexts Beliefs, perceptions Behaviours Bottom-up understandings Explaining causal patterns Impacts for different groups Innovative categories Interviews Focus groups Expert panels Surveys, questionnaires Participant observation Case studies SWOT analysis Delphi survey Content analysis Desk research

9 If you contract the evaluation to external body Terms of Reference must specify: Scope of the evaluation Users/stakeholders of the study Research questions and available data Indicative budget/time table Required qualifications of the team Structure of the proposal Manage the evaluation process with rigour Communicate the evaluation findings (lessons learnt)

10 Tips Appropriate timing Evaluation should be embedded in a strategy and planned in the programme design Mmaximum contribution of evaluation results Indicative budget: 1%, maximum 5% of the OP value Compare the results with the situation before Make use of the evaluation results: adjust targets or means if necessary

11 Thank you for your attention! More information on Inforegio: /docgener/evaluation/evaluation_en.htm /docgener/evaluation/evaluation_en.htm