Making Good Use of Research Evaluations Anneli Pauli, Vice President (Research)

Academy of Finland Main tasks

Academy of Finland Organisation

Academy of Finland Evaluation activities

Benefits of using international peer review method …to the Finnish Science Scientific quality guaranteed Easier to achieve objectivity, equal treatment and transparency Less negative feedback from researchers (small country) May enhance international collaboration More international visibility

Evaluation activitiesImpacts Applications Programmes Disciplines and fields of research Research system  Excellent scientific quality is a prerequisite for scientific as well as societal relevance  Feedback received on the quality of research, scientific and societal relevance, interdisciplinarity, national and international cooperation, programme management; Impact on the development of the field, identification of future research needs  Measures for further development of the field, prioritisation, possibly resource allocation  Knowledge on the structure and dynamics of science and innovation system for its further development

Peer review of applications Scientific rating by the peers Ranking by the Research Councils % of peers are foreigners, depending on the call Applications in English

Research programmes From planning to implementation Follow-up and evaluation plans included in early stages of planning A programme is evaluated against its objectives and funding volume Main components of the evaluation added value of the programme, ‘programmatic nature’ scientific results and impact societal, economic, technological, cultural impacts implementation and management of the programme Utilisation of evaluation after-care plans of the issues raised by the programme evaluation identifying future research needs and directions

Research fields: impact of evaluations Thorough planning of the evaluation Timing of the evaluation Commitment of research community to develop the field (realistic expectations) other stakeholders (research councils, other funding organisations, ministries etc.) Follow-up within the process of management by results (Finnish Ministry of Education and Science and Academy of Finland) cooperation of researchers, research institutions, and other stakeholders

CASE: Evaluation of Biotechnology in Finland (1) 2002: Evaluation of Biotechnology in Finland. Impact of Public Research Funding and Strategies for the Future. Commissioned by the Ministry of Education and Science Coordinated by the Academy of Finland Involved multiple stakeholders: ministries, Academy of Finland, National Technology Agency of Finland Tekes, and Sitra, industry Carried out by an international expert panel, chaired by Professor Fotis C. Kafatos, then dir.gen. of EMBL 2003: A follow-up working group appointed by the Ministry of Education and Science (MoE) Made a proposal for future actions and implementation of the recommendations in : MoE working group “Development programme for research and training in biotechnology in ” Follows and evaluates the implementation of recommendations and measures based on the Biotech 2002 Evaluation and the work of the follow- up working group 2003

CASE: Evaluation of Biotechnology in Finland (2) 2005, MoE working group: Questionnaire to universities and their biocentres, research institutes and funders of biotechnology: Information about resources and impact of research Implementation of the recommendations made in the BIOTECH 2002 evaluation Which recommendations, how and when the organisation has implemented? Which recommendations and when will be implemented in the future? Which recommendations cannot be implemented and why? Description of any other development measures which have been carried out in Which of the recommendations or measures addressed to other organisations have been of little importance and/or have been implemented inadequately?

Research system Reviews on the State, Quality and Impact of Scientific Research (1997, 2000, 2003) Next : 1. Bibliometrics Analyses of Finnish science (e.g. fields, organizations, cooperation) Comparison of Finland to OECD countries’ performance 2. Impact evaluation International benchmarking of evaluation methods and indicators Academy’s four Research Councils: Impact assessments of funded research  External expert panel: Impact evaluation of the Academy 3. Foresight Joint project of the Academy of Finland and the National Technology Agency Tekes 10 expert panels, important issues in the future

Challenges: from planning to implementation What is the motivation for carrying out an evaluation Cost-effectiveness Selection of evaluators of crucial importance Stakeholders: multiple views, commitment needed Long-term planning vs. short-term needs

Conclusions Different needs, levels, implementation and impacts Key issues of concern to research communities and institutions / policy makers / managers / others define the type of the evaluation Clear aims and criteria stated for each evaluation Right timing Results in a user-friendly format Combining evaluation with strategic planning / forward-looking / foresight Impacts can only be achieved if the findings of the evaluation results are utilised.