Proposal evaluation process in FP7 Moldova – Research Horizon 29 January 2013 Kristin Kraav

Evaluation process Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Commission pre-actions Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Commission pre-actions Appointment of call coordinator and coordinators for the topics Selection of the independent experts for the Call FP7 experts database ( personal head-hunt Proposals eligibility check eligibility criteria may be budget, number of partners, duration … Formation of the expert panels for each topic of the call Assignment of proposals to chosen experts

Individual assessment Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Individual assessment Remote evaluation period ~4-6 weeks Everything happens in online system RIVET experts can access only assigned proposals Max 10 proposals per expert, min 3 experts per proposal Experts have to make their decisions based on: proposal Call background documents evaluation criteria Result – Individual Assessment Report experts personal opinion basis for the discussions in the next stage

The evaluation criteria Three evaluation criteria (Cooperation and Capacities – People and Ideas differ significantly): – Scientific and/or Technological excellence – Quality and efficiency of the implementation and the management – The potential impact through the development, dissemination and use of project results

The evaluation criteria and details ( all funding schemes) 1. Scientific and/or technological excellence (relevant to the topics addressed the call) 2. Quality and the efficiency of the implementation and the management 3. The potential impact through the development, dissemination and the use of project results All funding schemes Soundness of the concept and quality of objectives Appropriateness of the management structure and procedures Quality and relevant experience of the individual participants Contribution at the European (and/or international) level to the expected impacts listed in the work programme under the relevant topic/activity

The evaluation criteria CP, SA Collaborative projects Progress beyond the state-of-athe-art Quality and effectiveness of the S/T methodology and associated work plan Quality of the consortium as a whole (including complementarity, balance) Appropriateness of the allocation and justification of the resources to be committed (budget, staff, equipment) Appropriateness of measures for the dissemination and/or exploitation of project results, management of intellectual property Support action Quality and effectivness of the support action mechanisms and associated work plan Quality of the consortium as a whole (including complementarity, balance) [only if relevant]

The evaluation criteria (CA) Coordination actions Contribution to the co-ordination of high quality research Quality and effectivness of the co-ordination mechanisms and associated work plan Quality of the consortium as a whole (including complementarity, balance) Appropriateness of the allocation and justification of the resources to be committed (budget, staff, equipment) Appropriateness of measures for spreading the excellence, exploiting results and disseminating knowledge though engagement with stakeholders and the public at large

Scoring the proposals Each proposal is evaluated against all criteria and scored between 0 and 5 Threshold per criteria – 3 Overall threshold per proposal – 10 This means – if any of the criteria under 3 points or the total scoring of the proposal under 10 points, the proposal will be placed to the list of rejected proposals for reason  under the threshold

Interpretation of the scores 0 – The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information. 1 – Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses. 2 – Fair. While the proposal broadly addresses the criterion, there are significant weaknesses. 3 – Good. The proposal addresses the criterion well, although improvements would be necessary. 4 – Very good. The proposal addresses the criterion very well, although certain improvements are still possible. 5 – Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor. Slide 13

Consensus meetings Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Panel of experts

Consensus meetings 1 week for face-to-face meetings, 12 hour per day Consensus meeting 1.5 hour meeting per proposal gathering all experts who evaluated it, moderated by EC staff-member basically restart of evaluation has to end with consensus on scores and comments for each of the criteria → Consensus Report On the bases of discussions rapporteur (one of the evaluators) drafts the first version of Consensus Report

Consensus meetings → ESR First draft of CR turns into Evaluation Summary Report through: co-operation of all experts involved comments from Call coordinator and topic coordinator help of language editor ~4-5 day parallel process for all proposals submitted under the topic If the CM does not end with consensus: additional experts from the panel will evaluate it new CM is called If still no consensus → total restart of the process with new panel of experts

Panel meeting Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Panel meeting → proposed ranking list Drawing up the ranking list of the topic Comparison of scores and ESR Final marks and comments for each proposal Suggestions on order of priority, clustering, amendments, etc. Simple process if the ranking is clear Tricky process if many proposals have equal score and all of them can not be funded Priority order of criteria in case of equal scores If all scores equal, experts have to decide which score is stronger and support it with relevant arguments

Commission follow-up Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Commission follow-up Draw up final ranking lists decisions on proposals selected for funding decisions on rejected proposals Information and data to the Programme Committee PC formal agreement needed for financing proposals with budget over € Independent Observers’ report Invitation to Contract Negotiation and Evaluation Summary Reports sent to coordinators Contract negotiations

Evaluation process Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Evaluation process – lessons learned Individual evaluation Commission pre-actions Consensus meetings Panel meeting Commission Follow-up

Process of evaluation Evaluation is a deconstruction of proposals Comparison of different elements of proposal against each other overall aim → steps to achieve it (activities) → expertise (partners) for managing the process → allocated resources Everything should be written as clearly and shortly as possible – still all required parts of Part B have to be there “A picture is worth more than 100 words” – tables, graphs, schemes help immensely Do not expect evaluators to assume things

Remember: Evaluators are humans, they come from different professional and ethnical backgrounds; have different beliefs of right and wrong; speak different languages; usually are under time pressure Respect the evaluators: follow the structure edit your proposal to eliminate typos and other mistakes make the proposal legible: font and font size, structured text, test that all graphics can be read in b&w

Conclusions Fit to the call : read carefully the call topic Be outstanding on the Scientific and Technology point of view but do not underscore the other criteria Be credible Demonstrate an EU added value Respect the rules : read the Work Programme and Guide for Applicants