prof.dr. Branko Kontić, dr. Davor Kontić CIVITAS Elan Study Tour Ljubljana, 23 October 2013 Role of evaluation in the stage of measures and project planning

Aim of this discussion Key message to the EC – over VANGUARD (why not POINTER?; expecting synthesis report) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting) Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, common measures, transport policy development)

CIVITAS Famework

How does EC know that the project will meet the goals? Impact evaluation Process evaluation Integrated package evaluation City level evaluation Cost-Benefit Analysis

Evaluation Framework (by POINTER, 2009)

Impact Evaluation Framework (by POINTER, 2009)

BAU and CBA Evaluation (by POINTER, 2009)

Process Evaluation Framework (by POINTER, 2009) Measure level: 336 measures (100%) City level: 26 cities ImplementationPreparationOperation Selection POINTER Support Measure Process Evaluation Form Process Topics & Issues Form Focused Measure Process Evaluation Form Selection Measure level: Ca. 120 measures (Ca. 30%) Measure Process Evaluation Focussed Measure Process Evaluation Process Topics & Issues (by POINTER, 2009)

Overview - LCA of measures and evaluation Phases/activities of a measure Evaluation of measure’s and project goals Transfer&upgrade Process evaluation planning implementation operation Impact evaluation CBA

Missing components Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated) Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)

Missing elements in the planning stage 1/2 Role of the Evaluation Plan Purpose (holistic view); it is not only a deliverable Project management context (consultation); better cooperation needed Changes in project implementation (measures’ design, implementation, operation) and links to the evaluation plan Agreements between MLs and Evaluation Team (objectives, data collection, etc.); see next slide Performance assessment (PA) of a measure in the context of evaluation; see next slide; PA follows all (internal and external) barriers and drivers, process evaluation follows only outside barriers and drivers

Missing elements in the planning stage 2/2 Links between evaluation and dissemination: evaluation of dissemination activities, dissemination of evaluation results Links and preparations to strategic and management role of the evaluation, specifically PA and CBA: Is/was money being reasonably spent? Was it worth doing? What could be done better? Which mistakes we should not repeat? Post “lessons learned” approach is less effective!

A sample of an agreement between ML and SEM

A sample of the Performance Assessment table

Possible improvements 1/3 Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly. Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation. The planned evaluation procedure does not always work according to plan, it needs to be flexible. Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis. There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose. Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.

Possible improvements 2/3 Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not. Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour). Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.

Possible improvements 3/3 Role of the evaluation should be somehow integrated in the project approval procedure by the EC! Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.

Improved organization of CIVITAS Elan European Commission Political Steering Group Project Consortium Meeting Partners Site Coordinators EvaluationDissemination Scientific Coordinator Project Coordinator Project Manager Project Management Group Financial & Administrative Reporting European Commission Political Steering Group Project Consortium Meeting Partners Meeting Partners Site Coordinators Site Coordinators EvaluationDisseminationEvaluationDissemination Scientific Coordinator Scientific Coordinator Project Coordinator Project Manager Project Management Group Project Manager Project Management Group Financial & Administrative Reporting ManagerManagerManagerManagerManagerManager Measure Measure Measure Measure Measure Measure Evaluation Manager Evaluation Manager Evaluation Manager Local Dissemination Manager Dissemination Manager Dissemination Manager Local

Improved organization of CIVITAS Elan Pointer Improved Evaluation Project Manager Project Coordinator EC