Presentation on theme: "Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership 8 July 2011, Gdansk."— Presentation transcript:
Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership 8 July 2011, Gdansk
Key Fifth Cohesion Report themes (pp256-7) 1. Move on from preoccupation with financial absorption 2. Fewer, simpler policy priorities (concentration) 3. Focus on results and impacts (i.e. performance). 4. More rigorous evaluation (e.g. CIE, CBA for quant; case studies for qual; beneficiary surveys for both quant and qual) 5. Triangulation 6. From productivity to well-being (happiness) and sustainability For the presentation I will focus on 4, 5 and 6
Making the right choice: The example of beneficiary surveys o CED priority (P6 – Targeted Action for Key Deprived Areas) in Yorkshire & Humber Objective 2 programme o ERDF Measure 6.21: support for community based economic projects – 111 (of the 178) CED projects o Main types of projects funded: - Micro-finance and credit - Social enterprises (community businesses) - Labour market access, training and ILM - Education - Targeted environmental improvements - Transport access to jobs - Pure capacity building
What we did Two-stage sampling method: - managers of all the projects contacted for lists of beneficiary enterprises and other organizations - individual enterprises and organizations randomly sampled from the lists (simple random sampling) Useable lists from 33 of the 111 projects, but all the biggest projects chased down Random sampling of 242 organizations and enterprises (110 First Sector enterprises serving mainly non-local markets; 44 First Sector serving mainly local market; 15 community enterprises; 64 mixed social/market organisations; 9 purely social organisations)
Did we do anything right? Direct counterfactual questions
Did we do anything right? Lots of nice new stuff on why it works
What did we do wrong (in Fifth Cohesion Report terms)? Rigour Should have used stratified sampling method Only one bias check (sub-areas) No proper confidence intervals Triangulation Could have used open questions and a third stage of SSIs as a qual-on-quant triangulation Could have analysed secondary data sets for quant-on- quant triangulation Well-being and sustainability Beneficiaries = organisations. Wrong. Should have done residents survey Should have added some sustainability questions
Making the right choice: beneficiary surveys guidance Inherent characteristicComments Best for quant informationCan add open questions and SSIs Big population to sample fromOK, but lets not exaggerate this Good quality sample frameOK, but much easier if beneficiaries +non-beneficiaries surveyed Clear effects and preferably known beforehand what they might be OK, but nowhere near as rigid as many would have us believe Direct counterfactual questionsTriangulation with other quant Can ask why it works questionsBetter if effects clear and understood beforehand RigourMust improve: technical, time and money resources? TriangulationIn-house qual/quant and wider quant/quant Well-being and sustainabilityExcellent tool. Residents surveys? Focus on results/performanceExcellent tool; cost-effective too
Case studies: whats being done well? Example: ERDF Work Package 4: Structural Change and Globalisation Thematic: effect of, and policy response to globalisation Holistic: quantitative and qualitative Quantitative: Three main elements: - By core team statistical analysis - By case study teams - Pilot beneficiary surveys; secondary data analysis Qualitative: - In-depth analysis of documents - Semi-structured interviews with regional experts, policy makers and beneficiaries (176 in total)
Case studies: whats being done well? Multiple full case studies (12) Transparent and logical selection criteria: representative approach rather than extreme/deviant, maximum variation, critical or pardigmatic (Flyvbjerg, 2006) In-depth knowledge of context, including historical context (flexible – back to early 1980s for many) Narrative: telling the story Mini-case studies Common template but with flexibility for regional teams Central core team Generalization: accumulation of evidence; power of the deviant case
Case studies: what next? Rigour Fundamentally a qualitative method, but needs more quant Beneficiary surveys Proper CIE and secondary econometric data analysis? More rigorous qualitative methods Triangulation Already happening, but Regional team SSI/document analysis and core team quant Beneficiary survey as quant-on-qual by regional team Well-being and sustainability Potentially excellent method
Making the right choice: case studies guidance Inherent characteristicComments Best for qual informationSome quant by core and regional teams already; beneficiary survey Particularly useful for thematic and cross-cutting evaluation? OK, but is unusually flexible especially where min-case studies too Best for complex, longer-term and where context important Best where main effects not clear or understood beforehand OK, but beneficiary survey would widen it to clear effects too Poor for measuring counterfactual Good for why it worksOK, but beneficiary survey would help RigourMust improve: technical, time and money resources? TriangulationIn-house qual/quant; more qual/qual possible Well-being and sustainabilityExcellent tool. Not cost-effective as quant Focus on results/performanceExcellent tool; residents survey?
From productivity to well-being: A warning from history: British enterprise zones in the 1980s An urgent need to prioritize amongst the many well-being and sustainability measures. Concentration will narrow the field for each programme, but…….. still and awful lot, especially for well-being A must for all future evaluations: distribution of income and wealth A warning from history: Erickson and Syms, Regional Studies, Salford-Trafford Park EZ, Manchester. Tax reductions (especially property tax – Business Rates Place prosperity versus people prosperity: who wins, who loses. Distribution: combined property tax (rates) plus real rents: 36% of value of the property tax exemption went to industrial tenants and 64% to initial property owners, many ex-regional. We may not like the results we get.
From productivity to well-being and sustainability: Challenges for other methods? Heavily affected? - cost-benefit analysis - SWOT - Evaluability assessment, logic models, concept mapping - Delphi methods, focus groups, experts panels - Observational methods - Stakeholder consultation and participative techniques - Meta-analysis Less heavily affected? - Input-output - Macro models CGE