1. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Activities of the ESF Member Organisation Forum on Research Evaluation

2. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 1. Objective of ESF Member Fora 2. ESF Member Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes 3. The new Forum on Evaluation of Publicly Funded Research Beispielfolie Inhaltsverzeichnis (mit Bild) ESF Member Fora Structure of the presentation

3. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 1. Objective of ESF Member Fora 2. ESF Member Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes 3. The new Forum on Evaluation of Publicly Funded Research Beispielfolie Inhaltsverzeichnis (mit Bild) ESF Member Fora Structure of the presentation

4. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 The instrument “ESF Member Organisation FORUM” was developed in response to clearly expressed needs from MOs in consultations on the Strategic Plan 2006-2010 Quoting the Strategic Plan: “MO Fora is an output-oriented, issue-related venue for MOs, involving others as appropriate, to exchange information and experiences and develop joint actions in science policy”. “ESF aims to be the actor of choice and preferred partner when national research organisations develop joint activities.” ESF Member Fora Objective of the instrument

5. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 ► Completed Member Fora ●Promoting Internationalisation of Social Sciences in Central and Eastern Europe (6/2006- 12/2008) ●Evaluation of Funding Schemes and Research Programmes (10/2007-11/2009) ●Research Careers (11/2007-4/2010) ► Current Member Fora ●Peer Review (10/2007-12/2011, extended) ●Research Integrity (9/2008-12/2010) ► New Member Fora ●Medium Sized Research Infrastructures (1/2010-2011) ●Science in Society Relationships (1/2010-9/2012) ●Evaluation of publicly funded Research (4/2010-1/2012) ●Indicators of Internationalisation (11/2009-11/2010) ●Research Careers (...) Completed, current and new Member Fora

6. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 1. Objective of ESF Member Fora 2. ESF Member Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes 3. The new Forum on Evaluation of Publicly Funded Research Beispielfolie Inhaltsverzeichnis (mit Bild) ESF Member Fora Structure of the presentation

7. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 ► Focus: “Post-Grant” Evaluation i.e. if the funding schemes or the research programmes achieve their stated aims. ► Objectives ●exchange and document experiences with current practices ●facilitate networking of science officers engaged in evaluation ●explore needs and possibilities for collaboration in future evaluation exercises. ► Participation: 38 Organisations ESF MOF on Evaluation of Funding Schemes and Research Programmes – Aims

8. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 1 st Workshop ► Evaluation of Funding Schemes and Research Programmes: Expectations, Practices and Experiences. Berlin, 22.-23. October 2007 (Host: DFG) 2 nd Workshop ► Quantitative Indicators in ex-post Evaluation of Funding Schemes and Research Programmes: Case studies and Rationales. Rome, 31. March -1. April 2008 (Host: INFN) 3 rd Workshop ► Quality of Quality Assurance. Striving for Highest Standards in ex-post Evaluation. Vienna, 6.-7. October 2008 (Host: FWF) 4 th Workshop ► Evaluating socio ‐ economic impact of research funding schemes. Budapest, 27.-28. April 2009 (Host: OTKA) 5 th Workshop ► Evaluation Strategies and Comparative Evaluation Techniques. Dublin, 16.-17. November 2009 (Hosts: HRB and SFI) ESF MOF on Evaluation of Funding Schemes and Research Programmes – Workshops

9. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 ► Action Items 1. Mapping study on evaluation of funding schemes –finalised 2. A pilot study on international benchmarks – will continue in a new forum ► Final Outcome 1. Report: Evaluation in National Research Funding Agencies: approaches, experiences and case studies can be downloaded: http://www.esf.org/activities/mo-fora/publications.htmlhttp://www.esf.org/activities/mo-fora/publications.html ESF MOF on Evaluation of Funding Schemes and Research Programmes – Outcome

10. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 1. Objective of ESF Member Fora 2. ESF Member Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes 3. The new Forum on Evaluation of Publicly Funded Research Beispielfolie Inhaltsverzeichnis (mit Bild) ESF Member Fora Structure of the presentation

11. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 ESF MO Forum Future ► New Forum on Evaluation of Publicly funded Research, approved by ESF Governing Council ●Second Forum „Indicators of Internationalisation“ ► Proposing Member Organisations: ●DFG, Germany; HRB, Ireland; NOW, Netherlands; FWF, Austria; FNR, Luxembourg; The Research Council of Norway; Swedish Research Council; Hungarian Scientific Research Fund. ► Towards good practices and joint evaluation exercises ► Alignment with EUROHORCs/ESF Road Map: ●Chapter 6. Develop common approaches to ex-post evaluation of funding schemes and research programmes ► Strengthen Workshop format

12. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 ESF MO Forum on Publicly Funded Research ► First meeting 10 th to 11 th May 2010 in Stockholm ► three more meetings within the next two years (next meeting Nov. 2010) ► set up three working groups that drafted their work plan: ●Assessing the Impact of Funding Schemes on Science and Society ●Ensuring and Improving quality in ex-post evaluation of Funding Schemes and Research Programmes ●Classification systems and Categorization of output data ► Envisaged outputs: ●a web-based repository of recent evaluation studies ●guidelines for evaluation – commenting on state of the art ●mapping the evaluation practices/the evaluation function in funding organisations ●overview of impact studies and identification of gaps ●recommendation on possible design of impact studies ●a pilot study on the classification and collection of output data ●a document on steps and pitfalls on the way towards a common classification

13. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Working Group on Classification systems and Categorization of output data ► Examine feasibility pilot for Health Research Classification System (HRCS) coding of portfolios in FNRS and DFG ► Final reports and data for statistical and evaluative purposes: Structuring final reports helps to gather data that allow to answer statistical questions on topics like male/female scientific staff in funded projects nationality of scientific staff-members (indicates internationality) age of staff-members (indicates „age of a field“) collaboration with national and international partners (by country) (indicates cluster and networking) Output (publications, patents, articles in proceedings etc.) Qualifications (dissertations of staff members)

14. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Final Reports: Method for achiving structured informations in the DFG ► DFG-Guidelines name the topics to be mentioned in a final report ► DFG plans to collect these data via a PDF-based standard-form ► Data within these forms will be extracted automatically and transferred into a database ► Some of these data are published via DFG-Internet (e.g. the publication list http://gepris.dfg.de ) http://gepris.dfg.de ► Others will be used only for statistical purposes (e.g. age and nationality of scientific staff members) Project Status DFG: ► Conceptual considerations on how to and which data to collect are finished ► Considerations on technical solutions (e.g. transfer to database) and legal framework in final stage ► In the beginning of 2011 the standardized collection of final reports of individual grants will be implemented

15. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Method for achiving structured informations in the FWF (Austrian Science Fund) ► Collection of project output data in the FWF Source: Collecting and using project output data at the FWF, Rudolf Novak, FWF, Austria

16. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Method for achiving structured informations in the FWF (Austrian Science Fund) - Publications ca. 10.000 Scientific Journals Source: Collecting and using project output data at the FWF, Rudolf Novak, FWF, Austria

17. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Working Group on Classification systems and Categorization of output data ► Next Steps: ●Produce short paper summarising approaches for the analysis of output data ●Circulate/share details of work ongoing in each research organisation which involves the gathering of information about research outputs/outcomes and impacts. ●Generate recommended “core” set of outputs that could/should be collected by research organisations, plus guidance on definitions etc. ●Plan pilot exercise in which data for “core” outputs is brought together across a number of research organisations. ► Discuss the results of each step in the whole MO Forum ► Contribute to the comparability of Research Funding!

18. euroCRIS Strategic Seminar, Katharina Fuß Brussels, 13.09.2010 Thank you for your attention! for further questions please contact: Anke Reinhardt Deutsche Forschungsgemeinschaft (DFG) -Informationsmanagement- D-53170 Bonn Tel. +49 (228) 885-2588 Fax +49 (228) 885-2777 Anke.Reinhardt@dfg.de http://www.dfg.de Laura Marin Science Officer MOs Relations and Partnerships Corporate Science Strategy Development European Science Foundation Tel.: +33 388 76 7138 lmarin@esf.org http://www.esf.org