1. ‘Using evaluation research as a means for policy analysis in a ‘new’ mission-oriented policy context’ E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR A. Gök, UNIMAN / MIoIR 2012 EU–SPRI Conference, Karlsruhe, 12-13 June 2012

2. Outline The ‘new’ policy context Special nature of grand challenges Challenges for evaluation Meeting the evaluation challenges Usefulness of evaluations Impact assessment Behavioural additionality The Structural Funds framework Meeting the evaluation challenges - revisited

3. The ‘new’ policy context Mission – oriented Orientation to dealing with ‘grand challenges’ Not new as policy orientation but some important different features: –From national, military – industry lead projects in the ‘40s & ‘50s to more global, socially-driven endeavours oriented to tackle challenges impossible to solve by single entities or countries or rational planning approaches

4. Special nature of grand challenges Need to break long-standing boundaries; need approaches that are: Multi-disciplinary in science & technology Addressing s&t as well as social innovations Multi-level in governance Cross departmental in policy Multi-actor, multi-agency Cross – sectoral Applying longer-term horizons

5. Grand challengesEvaluation challenges Challenges for evaluation (1/2) Multi-disciplinarity Multi-level gover’nce Policy coordination Multi-actor / agency Long-term approach S&t and social innov. S&t fragmentation Policy silos Broader set of stkh’s Impact identification Multiple impact types Multiple levels of ref.

6. Evaluation challengesChallenges’ groups Challenges for evaluation (2/2) S&t fragmentation Policy silos Broader set of stkh’s Impact identification Multiple impact types Multiple levels of ref. Need for policy learning at operational, policy and especially system level Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change

7. Evaluation challenges Meeting the evaluation challenges Need for policy learning at operational, policy and especially system level Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change Usefulness of evaluations Impact assessment Behavioural additionality

8. Usefulness of evaluations (1/2) INNO Appraisal results showed that: Significant positive correlations with usefulness were identified for: –Use of open tendering process when commissioning an evaluation –Use of external evaluators –Summative over formative evaluations –Evaluations that examined goal attainment and effectiveness and policy/ strategy development –Evaluations that employed: case study analysis; participant surveys; interviews; focus groups/workshops and meetings; peer review –Evaluations that resulted in minor redesign or expansion/prolongation of the measure –Evaluations not conducted primarily for auditors/financial authorities –Evaluations whose reports were published in English Evaluations of measures for science-industry cooperation and creation of start-ups/spin-offs significantly more useful

9. Usefulness of evaluations (2/2)* Broadly, an evaluation may be considered useful if it: delivers the Terms of Reference in a consistent manner provides actionable recommendations delivers value for money delivers some degree of policy learning. BUT evaluations not extensively used to mobilise the community (only 50% targeted beyond policy makers and programme managers) usefulness is highly subjective and context specific * Based on the INNO-APPRAISAL study

10. Impact assessment* Impact assessment limited and simplistic in its approach Assessment of economic impact most dominant Assessment of new impact types rather uncommon Demand for non-economic impacts and spill-over effects What new impacts under the ‘new’ mission orientation? (beyond intended, beyond visible, that can last, that span across and beyond levels of references, that refer to multiple stakeholders, and cross different policy arenas) New sets of criteria and indicators required Known challenges in assessing social impacts (e.g. causal links, evidence, attribution problems, timing) * Based on the INNO-APPRAISAL study

11. Behavioural additionality (1/2)* Emphasis on learning, long-term horizon Gaining importance (addressed in 50% of national evaluations) Three types of use: –behaviour-focused way –Integrated ways –Instrumental way BA evaluations broadly discussed and more often targeted towards the general public and towards users  learning and mobilisation potential A need to demonstrate the conceptual link between the behavioural change and (intended) innovation effect The ‘black box’ remains * Based on the INNO-APPRAISAL study

12. Behavioural additionality (2/2)* If an evaluation is considered as an administrative exercise that was imposed by a supra-national sponsor, the policy learning and behavioural additionality it creates is limited. However, if it becomes part of the policy learning experience its usefulness and behavioural additionality increase.  Context and wider framework within which evaluation is implemented crucial for both usefulness and BA * Based on the INNO-APPRAISAL study

13. The Structural Funds framework If and how the SF framework affects the quality and usefulness of evaluations and thus the potential for BA* SF evaluation requirements in relation to Structures Evaluation design Evaluation execution aim, type, and nature of evaluations Evaluators’ selection Publicity & Dissemination Quality assurance * Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

14. The Structural Funds framework If and how the SF framework affects the quality and usefulness of evaluations and thus the potential for BA* Research hypotheses SF requirements may lead to specific characteristics in delivery & practice of evaluation SF requirements may lead to higher quality evaluations High quality SF evaluations may have greater impact SF regulations demand high standards on structures and processes that inevitable need some institutional learning and structure building * Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

15. SF requirements may lead to specific characteristics in delivery & practice of evaluation  Use of external evaluators  Data Analysis and collection methods: not surprising differences  Evaluation topics: SF requirements do make a difference in guiding the evaluation topics to cover across the different evaluation types (ex-ante, interim, ex-post)  Clearer differences across the evaluation types within each group than across the two groups (SF and non-SF group)  Hypothesis valid but also differences in features may relate more to evaluation type than SF or non-SF framew. * Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

16. SF requirements may lead to higher quality evaluations X Compliance to quality standards is less in SF evaluations Certain standards clearly ‘overlooked’: production of useful recommendations; discussion of results with government and stakeholders  SF requirements do not necessarily lead to high quality appraisals High quality SF evaluations may have greater impact X What clearly improves usefulness of recommendations is the evaluation design and application of quantitative methods Increased discussions with government and wider stakeholders are more caused by high quality non SF eval.  SF requirements for high quality do not necessarily lead to high impact in terms of usefulness and dissemination * Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

17. SF regulations demand high standards on structures and processes that inevitable need some institutional learning and structure building  SF evaluations more as ‘internal’ exercises  limited discussions with government and stakeholders SF regulations too focused on financial aspects and correctness of implementation - lack of qualitative impact assessment or BA  limited quality inputs to new programmes and schemes  limited usefulness of recommendations  high quality SF evaluations ≠ high impacts in terms of usefulness and dissemination to national stakeholders  Typical application of procedures ≠ institutional and policy learning BUT significant capacity building

18. Usefulness: not extensive; relates more to operational issues Impact assessment: Several issues pending; new issues emerging BA: Still a ‘black box’ SF framework: typical application; not opportunity for learning Need for policy learning at operational, policy and especially system level Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change Wider set of impacts Beyond S&T&E Beyond inputs/outputs, Behavioural change Meeting the evaluation challenges revisited

19. Thank you for your attention! E. Amanatidou, UNIMAN / MIoIR, effie.amanatidou@gmail.comeffie.amanatidou@gmail.com I. Garefi, Atlantis Consulting S.A., garefi@atlantisresearch.grgarefi@atlantisresearch.gr P. Cunningham, UNIMAN / MIoIR, paul.cunningham@manchester.ac.ukpaul.cunningham@manchester.ac.uk A. Gök, UNIMAN / MIoIR, abdullah.gok@manchester.ac.ukabdullah.gok@manchester.ac.uk