1. SNSF 31.03.2014 Research creates knowledge. Science Europe Working Group ‚Research Policy and Programme Evaluation‘ Work Strand C: ‚Uses of Evaluation Outputs‘ 12 November 2015

2. SNSF 31.03.2014 Research creates knowledge. What is Work Strand C about? Objectives Contribute to increasing the usability and use of evaluation outputs Exchange of experience among members of Working Group Identify conditions under which high quality, independent evaluations contribute to policy implementation and definition Explore whether policy guidance or recommendations can be provided on this basis Methods Case studies Stakeholder workshop

3. SNSF 31.03.2014 Research creates knowledge. Overview of case studies InstitutionResearch FieldInstrument/ Programme Evaluation of INRA-Research Units by HCERES (Katrin) Mecanical engineering in Sweden (Sten) DFG Collaborative Research Centres (Anke) Swiss Group for clinical cancer research (Katrin) Mathematics in Sweden (Sten) DFG Emmy Nöther Career Programme (Anke) Evaluation of Austrian Research Institute? (Klaus) Mathematics in Norway (Stig) Swedish Research in Biodiversity (Sten) LMT (Eugenijus) Plant & Soil Sciences in Estonia (Maarja-Liisa) SNSF Fellowship Programme (Katrin) One case study from FCT (Luisa) Engineering in Estonia (Maarja-Liisa) Case study from MRC-UK (Beverly) One case study from Jordi ? Evaluation of FWF Schrödinger Fellowships (Klaus) Education programmes (Stig) Nat. Research Progr.: Social challenges to national security (Eugenijus) Case study completed …in progress …planned …possible

4. SNSF 31.03.2014 Research creates knowledge. Methodological issues Case study selection Case studies must cover evaluation studies that… are finished, published are backward looking (interim or ex-post) cover evaluation objects at the level of the institution, research field, instrument/programme but not individual grants, persons or projects Case studies are selected to… cover at least 3 organisations per type of evaluation object (-> ensure critical mass/basis for comparison) cover various degrees of use, from extensive to non-use (-> enable exploration of potential factors influencing evaluation use)

5. SNSF 31.03.2014 Research creates knowledge. Methodological issues Guidelines for doing case studies Case studies are prepared in accordance with case study template tested and refined in pilot case studies They should rely on: a.information that is publically available, b.information from at least two different sources bringing in different perspectives with regard to the evaluation process.

6. SNSF 31.03.2014 Research creates knowledge. Methodological issues How to synthesize results? Triangulation of perspectives on evaluation use within case studies Collection of factors potentially influencing evaluation use Critical examination within Working Group… …and on the occasion of workshop with additional stakeholders Potentially derive guidance or policy recommendations

7. SNSF 31.03.2014 Research creates knowledge. Preliminary conclusions Hypotheses on factors influencing use Evaluation set-up Acceptance of evaluator and methods (e.g. reluctance towards outsiders/consultants), though perceived quality not always linked to use Form of the evaluation report, Scope and level of recommendations Institutional context Timing (fit with strategic cycles, transition of key players, lengthy processes conflict with impulse for change) Engagement of key players in relevant decision making processes Joint evaluation by different stakeholders Presentation by EU Commission highlights that very formal structure for evaluations may itself become an impediment for use if it is perceived as a bureaucratic apparatus. Research context Degree to which evaluands are accustomed to and prepared for being evaluated Policy context Definition of scope – e.g. delimitation of research field, research field v. programme?? Desire to promote or test an intended change, to fill a knowledge gap Fit of findings and recommendations with general strategic orientation and priorities Spotlight on a certain field, issue… may in itself produce an impact

8. SNSF 31.03.2014 Research creates knowledge. Preliminary conclusions General points Evaluation use may also manifest itself in non-action, if the evaluation prevents a planned or discussed ‘negative action’, for example an economic cutback of a certain research program or research field. Evaluations may be used beyond their immediate purpose, years later (‘sleeping beauties’), possibly with respect to other programmes. Evaluation use may also manifest itself in best-practice for future evaluation studies (evaluation design, methodology…). Excessive focus on use can also be problematic evaluation as a tool for change management? instrumentalisation?

9. SNSF 31.03.2014 Research creates knowledge. How to structure findings? Types of use Stakeholder analysis

10. SNSF 31.03.2014 Research creates knowledge. Preliminary recommendations Do not conduct evaluations unless you plan to use them! Establish a “program theory” or a “logic model” for the evaluation exercise itself.

11. SNSF 31.03.2014 Research creates knowledge. Validation Workshop (early June 2016) Discussion on conclusions with “sounding board” decision-makers evaluators experts of evaluation use Possible contacts for sounding board Limit participation from Working Group to ≈ 10, so as not to create imbalance

12. SNSF 31.03.2014 Research creates knowledge. Conference participations to share results Participation in FtEval Conference Special session for whole Working Group? Or participation by individual Work Strands? Participation in German Evaluation Society Conference …

13. SNSF 31.03.2014 Research creates knowledge. We are grateful for your feedback! Any comments on preliminary conclusions? How to get the most out of a Stakeholder Workshop? Which individuals or groups of actors could validate our findings? Who might not feel free to share experiences/views openly for political reasons?

14. SNSF 31.03.2014 Research creates knowledge. Coordination with Work Strands A & B Synergies with Work Strands A & B? Timeline for next Working Group meetings? Budget requirements ok from point of view of Work Strand C Assume there won’t be a joint final output/report by all Work Strands