Evaluation of proposals Alan Cross European Commission

Evaluation: building on past experience…. - FP7 expert questionnaire Experts invited to participate Responses received Respondents finding the quality of the evaluation overall satisfactory to excellent (%) 96,197,6 97,498,2 98,4 Respondents rating the quality of the evaluation overall excellent (%) 22,126,52928,827,1 33 Respondents, having previously evaluated research proposals for national or international research funding schemes, finding the EU evaluation process better or much better (%) 52,661,361,060,863,9 65,7

…adapting to Horizon 2020 Less prescriptive calls; new types of proposal (multi-disciplinary and multi-sectoral; more emphasis on innovation and close-to- market) Greater spread of expertise needed; standard briefing on all key issues… Responding to 8 month time-to-grant imperative Proposals evaluated as submitted - no recommendations for substantial modifications Simplification: for applicants, experts, and for streamlined operations Simpler forms/page limits; improved facilities for remote evaluation (incl. consensus); faster data transfer Coherence across the progamme A single set of guidelines (Vademecum chapter) for all DGs and Agencies

HORIZON Overview of the Evaluation Process Evaluators Individual Evaluation Reports (Usually done remotely) Consensus Report (May be done remotely) Panel report Evaluation Summary Report Panel ranked list Eligibility check Allocation of proposals to evaluators Final ranked list

HORIZON Evaluation Process Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Consensus group Consensus Report Consensus Report Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Individual Evaluation Report Expert Minimum 3 experts Individual evaluation Consensus Proposal Eligible proposal

Award criteria (Research & Innovation; Innovation Actions) 1. Excellence  Clarity of the objectives;  Soundness of the concept, including transdisciplinary considerations;  Credibility of the proposed approach;  Ambition, progress beyond the state of the art. 2. Impact: […] contribution to:  The expected impacts listed in the work programme  Enhancing innovation capacity and integration of new knowledge;  Strengthening the competitiveness and growth of companies by developing innovations meeting the needs of European and global markets;  …measures to, disseminate and exploit the project results,… communication. 3. Quality and efficiency of implementation  Coherence and effectiveness of work plan, … allocation of tasks, resources;  Competences, experience and complementarity of the individual participants, as well as of the consortium as a whole;  Appropriateness of the management structures and procedures….risk management. First stage considerations in RED

7 Proposal scoring Each criterion is scored 0-5 half-scores allowed whole range should be considered scores must pass thresholds if a proposal is to be considered for funding Thresholds apply to individual criteria… Default threshold is 3 …and to the total score higher than the sum of the individual thresholds Default threshold is 10 Can vary from call-to-call! First stage proposals: first two criteria only threshold = 4

8 Interpretation of the scores 0 — The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information. 1 — Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses. 2 — Fair. The proposal broadly addresses the criterion, but there are significant weaknesses. 3 — Good. The proposal addresses the criterion well, but a number of shortcomings are present. 4 — Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present. 5 — Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

Weighting Unlike FP7, for Innovation Actions and SME instrument… Impact criterion weighted by factor of 1.5 'Impact' score given priority over 'excellence' when scores equal (the reverse applies in other types of action).

10 Selection of proposals All above threshold proposals in each topic are listed in descending order of overall score We select proposals starting from the top of the list, until the available budget is exhausted