Presentation on theme: "SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH NOVEMBER 6, 2012 CHISINAU, MOLDOVA EVALUATION OF RESEARCH PERFORMANCE: A System’s."— Presentation transcript:
SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH NOVEMBER 6, 2012 CHISINAU, MOLDOVA EVALUATION OF RESEARCH PERFORMANCE: A System’s Approach (Klaus Schuch, ZSI)
Contents 10/05/2015 EVAL-INNO 2 General aspects Different levels of Science and Technology Policy Evaluation Project Evaluation Programme/Instrument Evaluation Institution Evaluation Policy Evaluation Situation in Southeast Europe The EVAL-INNO project
Public Interest 10/05/2015 EVAL-INNO 3 Democratic values: transparency and fairness S&T policy values: S&T policy has to be in a position to show how and why R&D investments are worthwhile a developed culture of evaluation is an integral part of a strategically oriented S&T policy Evaluations help to reduce the risk of future interventions to combat market and system failure Evaluations contribute external knowledge and opinions to policy- making and policy-delivery and also provide the necessary documentation of how policies are implemented But evaluations do not replace policy action!
Regulatory Framework – Example Austria 10/05/2015 EVAL-INNO 4 Public Procurement Law Opt for “economically most advantageous offer” and not always “best price” Include also knowledgeable experts from outside the country In Austria a soft law is in place that demands evaluations for each R&D programme Guidelines to support economic-technical research and technology development (2007 ) Recommendations of the Austrian Council for Research and Technology Development on the Evaluation and Monitoring of RTDI Programmes from 12 April 2005 Evaluation Standards in Research and Technology Policy issued by the Austrian Platform for Research and Technology Policy Evaluation (FTEVAL)
10/05/2015 EVAL-INNO 5
Utilisation of Evaluations 10/05/2015 EVAL-INNO 6 Create a demand for evaluation results by Ensuring the support of top-level decision-makers Raising realistic expectations Integrating those involved in and affected from Create an appropriate environment by Carrying out evaluations systematically Planning evaluations within the policy cycle Ensuring access and collecting all relevant data – and only that! Reserving sufficient funding for a sound evaluation Ensuring the methodological quality of evaluations Follow-up on the implementation of results Designing appropriate communication strategies etc. EVALUATIONS ARE NOT CARRIED OUT FOR THEIR OWN SAKE!
Functions of Evaluations EVAL-INNO Legitimisation: Information: Learning: Steering: Mediation: justifying the use of public funds to the public how funds are being used and to what effect for all stakeholders to do things better for establishing policy objectives, planning of measures etc. balancing interest and dialogue 10/05/2015 7
Evaluation Pyramid 10/05/2015 EVAL-INNO 8 Project evaluations Programme evaluations Institution evaluations Portfolio evaluations System evaluations Policy evaluations Instrument evaluations In a developed NIS evaluation occurs on different levels!
Project Evaluation 10/05/2015 EVAL-INNO 9 The basis of any evaluation system is a sound project evaluation. A project is an individual activity with its own fixed time plan and its own budget. Projects produce concrete results (output, outcomes). Ex-ante evaluations project proposals are of regular component of justifying funding decisions. They are either based on peer reviews or jury decisions (with external experts). They assess the relevance of the proposal, its likelihood of reaching the objectives (of the project and contribution to the programme, if any), the quality of its content and methodology (research design), the quality of its management, its efficiency in times of expected input- output ratio, its economic viability (if economic R&D), it potential impact etc. Larger projects should also be subject to interim or monitoring and terminal or ex-post evaluation.
Austrian Science Fund Applied Research (FFG) 10/05/2015 EVAL-INNO Only competition based – all fields of science compete (no quota) Only peer review based Only international peers invited Proposals mostly in English Effect on the Austrian participation in European RTDI Open and targeted (e.g. thematic calls) – often programme based Economic content requires discretion Mix of peer reviews and jury reviews involving external experts Proposals often in German language 10 Project Evaluation in Austria
Programme/Instrument Evaluation (1) 10/05/2015 EVAL-INNO 11 A programme is a combination of measures where the underlying intentions refer to each other and which aim at achieving one or several specific defined objectives. A programme usually exists for a fixed time period and has its own allocated and often centrally administered budget. An instrument describes a specific form of an intervention. Monetary instruments (funding, financing) are to be differentiated from non- monetary instruments (consultancy, awareness-raising measures, regulations, standards). Programmes can use one instrument or a combination of instruments (mix of instruments).
Programme/Instrument Evaluation (2) 10/05/2015 EVAL-INNO 12 Ex-ante concept evaluations review the mission, assumptions, fundamental hypotheses and framework conditions of any possible interventions. Design evaluations deal with the organisational structure of an intervention (timing, budgeting, conflict resolution), control whether the measures used are suited to meet the objectives and if it was possible to reach them. They are typical elements of programme evaluations at all stages (ex-ante, interim, ex-post). The trend goes from a mere “before and after” comparison (black-box evaluation) to process evaluation (typical for formative interim evaluations). Impact analyses (mostly ex-post) assess the extent to which the objectives of an intervention have been achieved and aim to identify all relevant directly or indirectly, intentional or unintentional caused effects in a summative way. They look at impacts, outcomes and outputs, scientific effects, social effects and economic effects.
Institution Evaluation 10/05/2015 EVAL-INNO 13 Institutions are mostly of permanent nature. Focus is on universities with their combination of research and teaching, research institutions, intermediaries, and funding institutions or agencies. Institution evaluations are often peer based. They should help to avoid lock-in phenomena and blind spots. When assessing the research performance the focus should be on internal, external and negotiated factors concerning knowledge generation knowledge utilisation knowledge diffusion The RECORD Benchmarking Manual provides a useful methodology for transition and catching-up countries. Example: Benchmarking of Nanotechnology Research Institutes in EECA
Policy Evaluation 10/05/2015 EVAL-INNO 14 Policy constitutes a set of activities (funding instruments, procedures, regulations etc.) which may be very different in type but share a common motivation or objective. In contrast to projects and programmes, policy is generally not limited by time nor budget. Policy evaluations are often carried out in form of international peer reviews (e.g. DAC peer reviews of OECD in the field of development assistance). In S&T policy initiated through CREST (now ERAC). They critically examine the existing S&T policy mix by using a “critical friends” approach, provide policy recommendations and engage in a dialogue with policy makers and S&T experts from the host country to increase mutual understanding and learning. A S&T Policy Mix Peer Review is rather a (mutual) learning than a rigid assessment exercise!
S&T Policy Mix Peer Review Moldova 10/05/2015 EVAL-INNO 15 Carried out 2012 within the INCO-NET EECA project by involving S&T policy experts from Austria, Belarus, Estonia, Germany and Greece. Hosted by the Academy of Science of Moldova Focus: Public research and innovation system Private sector RTDI Funding of RTDI Human Ressources Internationalisation and FP7 association Regional dimension
Situation in SEE 10/05/2015 EVAL-INNO 16 Number of R&D programmes, applied instruments etc. grows, but … Capacities to evaluate RTDI programmes, instruments, policies and organisations are underdeveloped lack of knowledge on professional tendering procedures (including public procurement laws) to obtain the best evaluation results Potential of evaluations for strategic intelligence building is not exploited (no policy cycle; no systematic approach) Difficulties to access comparable RTDI evaluation information and good practices Difficulties to access international databases with important output data (e.g. patents, publications) Lack of completed good practice RTDI programmes, institutions and policy evaluations Lack of systematic exchange with evaluators in EU and globally.
EVAL-INNO Objectives 10/05/2015 EVAL-INNO 17 Key objective is to strengthen regional as well as national evaluation capacities in order to improve the framework conditions for innovations (policies, programmes, projects, institutions). In particular, the project aims to promote the role of RTDI evaluation for a reflexive learning innovation system to develop the needed capacities for comprehensive RTDI evaluations to provide procedural and methodological know-how and tool-kits both on the side of evaluators and on the side of awarding authorities.
EVAL-INNO Core Activities 10/05/2015 EVAL-INNO mapping evaluation experts and RTDI programmes in SEE 2. training evaluators and policy delivery systems 3. strengthening the use of external evaluation through right- and meaningful public procurement 4. establishing a virtual regional RTDI evaluation platform 5. developing standards 6. running RTDI programme evaluations and benchmarking exercises 7. linking to international networks
EVAL-INNO Training Invitation 10/05/2015 EVAL-INNO Training Courses 2. Separate training courses for evaluators and policy delivery systems including programme owners Bulgaria: March 2013 Hungary: April 2013 Montenegro: June 2013 Serbia: October 2013
EVALUATORS POLICY DELIVERY AND PROGRAMME OWNERS 10/05/2015 EVAL-INNO Hermeneutics and theory of evaluation Rules and ethics Real evaluation cases from different policy spheres Concrete evaluation methods (qualitative & quantitative) Exercises Hermeneutics and theory of evaluation Rules and ethics Real evaluation cases from different policy spheres Public Procurement ToR construction Development of Monitoring Systems 20 Eval-INNO Training Invitation
Training Course for Programme Owners 10/05/2015 EVAL-INNO 21 MondayTuesdayWednesdayThursday Introduction to the course and overview of RTDI evaluations RTDI System Evaluation. A case study Evaluation Platforms. The Austrian example Constructing the Terms of References History of RTDI evaluation, definitions, types, levels, timing of evaluations A Structural Funds Operational Programme Evaluation. A case study Evaluating Economic Impacts Constructing the Terms of References Design of evaluation- logic charts Programme Evaluation. A case study Evaluation of social impact of research The EVAL-INNO Platform and Databases of Evaluators Rules and ethics for evaluators and commissioning institutes Research Institute Evaluation. A case study Overview of evaluation basic tools and methodologies Monitoring of evaluations from the perspective of the awarding authorities. Competence of evaluators and awarding authorities University Evaluation. A case study RTDI Public Procurement Legislation in ERDF and IPA countries real-case based group exercises Utilisation of evaluation results - Usefulness of evaluation Ministry/Research Agency/ Awarding authority Evaluation. A case study RTDI Public Procurement Legislation in ERDF and IPA countries real-case based group exercises
Training Course for Evaluators 10/05/2015 EVAL-INNO 22 MondayTuesdayWednesdayThursdayFriday Introduction to the course and overview of RTDI evaluations RTDI System Evaluation. A case study Evaluation Platforms. The Austrian example Bibliometrics and patent analysis expert panels, focus group, participatory evaluation approaches History of RTDI evaluation, definitions, types, levels, timing of evaluations A Structural Funds Operational Programme Evaluation. A case study Evaluating Economic Impacts RTDI IndicatorsForesight and technology assessment Design of evaluation- logic charts Programme Evaluation. A case study Evaluation of social impact of research Questionnaires, Interviews, and field/case studies Network Analysis Rules and ethics for evaluators and commissioning institutes Research Institute Evaluation. A case study Overview of evaluation basic tools and methodologies Benchmarking AnalysisAdditionality. Control Groups Competence of evaluators and awarding authorities University Evaluation. A case study real-case based group exercises Utilisation of evaluation results - Usefulness of evaluation Ministry/Research Agency/ Awarding authority Evaluation. A case study real-case based group exercises
Contact Klaus Schuch ZSI – Centre for Social Innovation Austria inno.eu 23 10/05/2015 EVAL-INNO