1. Theory-Based Evaluation:
Untapped potential for EU Regional Policy
Marielle Riché
Evaluation Unit, DG Regional policy, European Commission
Helsinki, 4th October October 2012

2. Content of paper Evaluations in EU regional policy: state of play
TBE are relevant for regional policy: throughout policy cycle and for policy making
TBE still has much untapped potential: too much focus on funds absorption and lack of clear theories of change
: new opening for TBE (stronger focus on results, new regulatory requirements)
Looking forward

3. Evaluations in EU regional policy
Programme and evaluation management: MS or regions (EC: ex post evaluations)
Programmes run for 9 years ( /15)
Cover research, ICT, urban generation, SMEs, environment, social infrastructures etc.
May focus on specific territories, group of people, sectors
May use "integrated approaches", may support "innovation systems" – complex programmes

4. Evaluations in EU regional policy
MS/Regions: some 266 evaluations since 2007
Most on implementation issues and policy relevance, few on outcomes (27%)
Most based on traditional qualitative techniques and monitoring data
EC role: improve evaluation practice
Long-range aim: more rigorous evidence for policy making, building shared knowledge ("Regio collaboration")

5. TBE are relevant for regional policy
Programmes (should) rely on a theory of change
TBE are relevant throughout the programming cycle (ex ante, in itinere, ex post)
Analyse the assumptions underlying the programme
Assess the plausibility of programme achievements as planned
Assess contribution, analyse mechanisms leading to results, context, unintended effects

6. TBE are relevant for regional policy
Fields of intervention where counterfactual impact evaluations not possible
Complicated/complex programmes
Context analysis allows for policy adaptation and replication
Communication on programme outcomes (narratives)
Participative evaluations (partnership principle)

7. An untapped potential
Programmes more focused on funds absorption than on results
Evaluations carried out at the beginning of the period, focused on implementation
No explicit theories in programmes
Over-confidence in indicators to measure actual outcomes
"Myopia" of programmes (focus on first results)
Reluctance to commit to outcomes which are not "under control"

8. 2014-2020: a new opening for more TBE
Political commitment for a "result-based" policy
New logical framework, taking into account context and other factors, where impact is the contribution to observed results
Stronger requirements for ex ante evaluations
Compulsory impact evaluations for each priority axis of programmes (covers different interlinked interventions)
Our preparatory work with Member States: guidance and pilots

9. How to implement these new requirements for 2014-2020?
Our advice to Member States:
Use the evaluation ex ante to strengthen the programme theories of change
Be open to new approaches and allow for discussion on methods in Terms of Reference
Be realistic: take into account budget and time constraints, think of the primary use of evaluation (improve implementation or assess effectiveness? target audience?) .

10. How to implement these new requirements for 2014-2020?
Our advice to evaluators:
Help commissioners to refine evaluation questions
Propose and defend TBE designs mixing methods according to evaluation questions, timing, budget, planned use
Document data collection and analysis processes in evaluation reports

11. Conclusion
We should not miss the opportunity given by ex ante evaluations
Common goals: create a body of evidence on regional development
Need for critical mass of good quality TBE: mix of methods framed by a rigorous design
Need to develop strategies to minimise potential bias in data collection and analysis
Need to develop criteria and standards for theory-based designs .

12. Reference documents
On Inforegio Website:
Guidance documents
Impact evaluation center
Evaluation library (programmes)
EC evaluations
RegionalPolicy