Presentation on theme: "The Evaluation Process, Tips from an Evaluator’s Point of View"— Presentation transcript:
1 The Evaluation Process, Tips from an Evaluator’s Point of View Leonardo PiccinettiEFB
2 OUTLINE Evaluation principles The experts, who are they ? Role of Commission staffFP7 evaluation processFP7 Evaluation Criteria
3 Basic principlesExcellence. Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls.Transparency. Funding decisions are based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals.Impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.
4 Basic principlesConfidentiality. All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence.Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework.Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award
5 The experts, who are they ? The Commission draws on a wide pool of evaluators (database) in all scientific fieldsc. 50,000 in FP6Experts/evaluators from ICPC are very welcome!Calls for “candidates”Call for applications from individuals; and from institutionsApplications via CORDIS (database of experts)A mass- ing of FP6 experts was sentA simple tick-box will ensure registration for FP7Commission invites individuals on a call-by-call basisNot self-selection!Expertise, and experience are paramountGeography, gender and “rotation” also considered
6 Independent expertsExpert evaluators are at the heart of the FP7 systemExpert provides independent, impartial and objective advice to the Commissionrepresents neither the employer, nor the country!Significant funding decisions will be made on the basis of expert adviceThe integrity of the process is crucialExperts have to read the Code of Conduct annexed to the appointment letter…and follow it!
7 Proposal submission and evaluation 27/03/2017Independent expertsExperts agree to terms and conditions of an “appointment letter”Typically, an individual will review 6-8 proposals “remotely”….…then spend a couple of days in BrusselsSome will participate in “hearings” with the consortiaTravel and subsistence reimbursedPlus €450 honorarium per dayExperts sign confidentiality and conflict of interest declarationNames published after the evaluationsPresentation IP - Heysel
8 Actors Confidentiality The content of proposals, or the evaluation results, can’t be discussed with anyoneThe sole exception: in the presence of the EC moderator with experts who are evaluating the same proposal in a consensus meeting group or final panelIs not possible to distribute any documents related to the evaluation of a proposal, or take any documents from the evaluation buildingNote: The Commission publishes names annually, but as a group – no link between expert and proposal
9 Conflicts of interest (2) Types of COI set out in appointment letterCheck the exact wording!Disqualifying COIInvolved in preparation of proposalStands to benefit directlyClose family relationshipDirector/trustee/partnerEmployee of a partner in a proposalMember of Advisory GroupAny other situation that compromises impartialityPotential COIEmployed within the last 3 years by a partner in a proposalInvolved in research collaboration with proposers in the previous 3 yearsAny other situation that casts doubt…or that could reasonably appear to do so…
10 Role of Commission staff Check the eligibility of the proposalsOversee work of expertsModerate discussionsOrganise the panel and its workEnsure coherence and consistency
11 Evaluation Peer-Review System Two-stage evaluation procedure Remote evaluationEvaluation on a non-anonymous basisUnless otherwise specified in call for proposalRegister as an Evaluatorhttps://cordis.europa.eu/emmfp7/
12 Role of Commission staff Commission staff may advise on:Background on previously supported or on-going projectsRelevant supplementary information (directives, regulations, policies, etc)Evaluation rulesKey points within the Work Programme, e.g. issues related to “Relevance”Commission staff may not introduce:New elements (cannot fill in “gaps” in proposals)Interpretations
17 Writing an FP7 proposal is NOT just a creative process for A NICE IDEA It requires to SHOW scientific, technological and depth knowledge of the subjectYou must present references, legislatures, previous work and experience
18 Be precise and to the point Use drawings to show your methodology at onceUse bold types in phrases that you would like to emphasizePresent cohesion and interaction among work packages
19 Overview of the Evaluation Process “remote”may be“remote”SubmissionIndividualreadingConsensusPanelFinalisationFull ProposalProposalformsFinal rankinglistEvaluatorsEvaluatorsEvaluatorsRejection listCriteriaCriteriaCriteriaProposals insuggestedpriority orderEligibilityCOMMISSIONCOMMISSIONRole of experts
20 FP7 evaluation process IAR 1 IAR 2 IAR 3 CONSENSUS MEETINGRECOMMENDANTIONFOR FUNDING OR NOTIAR 2CONSENSUS REPORT BY THE RAPORTERIAR 3PANNEL MEETING – RESULTS ARE SEND TO APPLICANTEVERY IAR IS 3 TO 4 HOURS0 - the proposal fails to address the issue under examination or can not bejudged against the criterion due to missing or incomplete information1 - poor2 - fair3 - good4 - very good5 - excellent
21 QUALITY OF COORDINATION RELEVANCEQUALITY OF THE CONSORTIUM1. CONCEPT – CAPABILITIES OF THE PARTNERSQUALITY OF COORDINATIONQUALITY OF MANAGEMENTMOBILISATION OF RESOURCES2. ACTIVITIES – AND CAPACITY TO DO THE TASKPOTENTIAL IMPACT3. THE TARGET GROUPS THAT BENEFIT
22 Process Evaluation - Individual reading (Will be done remotely) The experts:Evaluate the proposal individuallywithout discussing with the other evaluatorsCheck whether the proposal is ‘in scope’Complete an Individual Evaluation Report (IER) form giving comments and scores on all criteriaSign and date the formIERs should be checked and, if necessary, returned with a request to further justify the score givenScores must be in line with commentsThis stage can also take place remotely (at home/ workplace of the expert)…
23 Process Evaluation - Consensus (1) Built on the basis of the individual evaluationsThe aim is agreement on scores and commentsUsually involves a discussion1st part may be carried out remotely“Outlying” opinions need to be exploredNot just a simple averaging exerciseIt is quite normal for individual views to changeModerated by a Commission staff-memberhelps the group reach a conclusionprovides information if necessarydoes not contribute opinions
24 Process Evaluation - Consensus (2) A rapporteur is appointed, who is responsible for drafting the consensus report (CR)includes consensus marks and commentsThe quality of the CR is paramountIt is not often changed at the panel review stageThe aim is:a clear assessment of the proposal, with justificationclear feedback on weaknesses & strengthsTo be avoided:scores that don’t correspond with the commentsrecommendations in view of resubmission
25 The Panel ReviewEC ask some evaluators in each sub-activity to examine and compare the CRs of every proposal that passes all thresholdsKey function is to ensure consistencyThe Panel will recommend for a sub-activity a priority order including final marks and comments for each proposalEvaluation Summary Reports (ESR)Any new scores (if necessary) … should be carefully justifiedRanking of proposals with identical consensus scoresPrioritise certain criteria?Consider overall balance?Budget?Clear guidance for contract negotiation
26 Process Commission Follow-up Evaluation summary reports sent to applicants“initial information letter”Redress procedureDraw up final ranking listsInformation to the Programme CommitteeContract negotiationFormal consultation of Programme Committee (when required)Commission decisionsSurvey of evaluatorsIndependent Observers’ reportsNew forFP7
27 The evaluation criteria Criteria adapted to each funding scheme and each thematic areaspecified in the work programmeThree main criteria:S&T Quality (relevant to the topic of the call)Concept, objective, work-planImplementationIndividual participants and consortium as a wholeAllocation of resourcesImpactContribution to expected impacts listed in work programmePlans for dissemination/exploitation
28 Process FP7 Evaluation Criteria Applicable to ALL funding schemes1. S/T quality (in relation to the topics addressed by the call)2. Implementation3. ImpactSound concept, and quality of objectivesAppropriateness of the management structure and proceduresQuality and relevant experience of the individual participantsContribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity
29 Process Funding schemes Collaborative projects Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research. The size, scope and internal organisation of projects can vary from field to field and from topic to topic. Projects can range from small or medium-scale focused research actions to large-scale integrating projects for achieving a defined objective Projects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc.Template of the IER for your information (double click on the object)Guidance for evaluators – Call FP7-ENV
30 Process FP7 Evaluation Criteria Collaborative projects1. S/T quality (relevant to the topics addressed by the call)2. Implementation3. ImpactPrgress beyond the state-of-the-artQuality and effectiveness of the S/T methodology and associated work planQuality of the consortium as a whole (incl. complementarity, balance)Appropriate allocation and justification of the resources to be committed (budget, staff, equipment)Appropriateness of measures for the dissemination and/or exploitation of projects results, and management of intellectual property.
31 Template of the IERs for your information (double click on the object) ProcessFunding schemesCoordination & support actions Support to activities aimed at coordinating or supporting research activities and policies (networking, exchanges, trans-national access to research infrastructures, studies, conferences, etc). same scope and objectives as in FP6Template of the IERs for your information (double click on the object)Guidance for evaluators – Call FP7-ENV
32 Coordination & support actions 1. S/T quality (in relation to the topics addressed by the call)2. Implementation3. ImpactCA• Contribution to the co-ordination of high quality research• Quality and effectiveness of the co-ordination mechanisms, and associated work planQuality of the consortium as a whole (including complementarity and balance)Appropriate allocation and justification of the resources to be committed (budget, staff, equipment)Appropriateness of measures for spreading excellence, exploiting results, and dissemination knowledge, through engagement with stakeholders, and the public at large.SA• Quality and effectiveness of the support action mechanisms, and associated work plan
33 RELEVANCEExamines if the objectives of the work programme are metThe proposer must read in between linesIt must be clear what the commission wants from the callGood practice: attend info day, inside informationCovering the objectives u get a 4, additional objectives might give u a 5TIPS:Do not copy paste the objectives from the work programmeU must copy and specialize them to the concept of the proposalFIND THE KEY WORDS (ie networking, knowledge transfer) relating the key word to the objectiveTRY TO QUANTIFY OBJECTIVES ( i.E. Not just networking – but networking of three clusters)Always make a direct reference under each objective with the related work packages of how this objective will be achieved
34 Proposal relevance tips Identification of impact from CallDefinition of strategic goals for our proposal which clearly fulfill the expected.Definition of main actions/activities clearly related to the strategic goalsWork Packages for actions/activities.
35 QUALITY OF CONSORTIUMExamines the experience of the consortium in relation to the objectives and the work need to be done, complementarities between partners, suitability of work undertaken, and geographical coverageTIPSAlways include a description in the beginning of consortium description indicating the role of each partner and their added value to the project.It is preferred to have inside the consortium mix Member States, New member states and Associated states indicating transfer of good practices to new member and associated states.Under each partner description indicating the excellences (previous projects, research activities and experiences) and of the partner related to the proposal followed by the CVs of key persons involved and their role in the projectThese excellences should be summarized into an excellence key areas of the consortium in relation to the work need to be doneIndicate the logic behind consortium geographical and context selection (why these partners are selected)
36 The coordinatorThere is no limitation related to the size, nature, legal status and the years of existence for the proposed coordinator.In the management forms (A1, A2 and A3) the turnover is presented. The is a indirect but very important effect to the decision of the Commission.
37 QUALITY OF COORDINATION Examines coherency of the proposal. If there is a methodology that defines the interrelationships between the work packages and the work tasksThe plan must convince the evaluators that is a bottom down approach from the implementation plan to the work packagesTIPSStart with the relation of the objectives to the work need.Define your methodology of work mentioning entities (not yet work packages)Define the interrelationships of the WPs and the work tasks inside the work packagesYour methodology should have a clear description of impact creation in the target groups and European Added Value through the actions of the dissemination planDo not include management packages, just coordination packages this would be explained in another section.YOU MUST ALWAYS START WITH THE METHODOLOGY OF WORK AND THEN WRITE THE WORKPACKAGESWork packages should mirror the work in the implementation plan with CLEAR milestones and derivablesIn general you could exceed the number of pages asked from the commission
38 General tipsEvery text with more than three paragraphs must begin with a 2-3 lines abstract with a clear reference to what the remaining text presents.In most cases the opinion of the evaluator is coming from this abstract.Phrases 8 words, paragraphs max 3-4 lines.
39 QUALITY OF MANAGEMENTExamines the experience and management mechanisms of the management unitThe evaluator examines the capacity of the coordinator to handle the tasksThey like to have a strong centralized management and not very complex structuresThey like to see decision making mechanisms from steering committeesTIPSAlways start indicating the central management control indicating the experience of the coordinator to undertake the tasks in similar projectsIllustrate decision making mechanism (simple but partner inclusive)Show contingency plan with risks (i.e. what happens if partner withdraws)Show IPR managementShow quality assurance mechanismsShow knowledge management with in partners.For each of the above relate them with management instruments (web sites, consortium agreements, meetings, etc)
40 MOBILISATION OF RESOURCES Examines the coherence of the financial plan to handle the tasks, relating to the allocation of work in PM within work packages and among partners.The financial plan must be delivered with the proposal although there are no predefined forms from that (except A3, deliverable lists)TIPSALWAYS state the CVs of key persons involved with the project and their roleIllustrate other resources for each organization that are allocated to the project in the partnership description (ie equipment, previous work done etc)Subcontracting is always examined in detail (why and who will do it?)Avoid large differences in work allocation between partnersIf a partner gets more than 30% of the budget (management and coordination) is negativeManagement no more than 7%
41 POTENTIAL IMPACTExamines the impact (results) to target group defined, and the European Added value and mainly your dissemination planRelates to the previous work done in European and National LevelTIPSAlways start with the clear definition of the proposal target groups indicating the potential impact in quantitative terms if possibleUse structured bulleted writing and not abstract descriptionsThe dissemination plan should provide evidence of European added value in quantitative terms (not general instruments i.e. web page but exclusive actions such as connection with specific initiatives, work groups etc)Always illustrate impact in relation to previous work done – specific national and European programmes indicating the approach mechanisms (even it is obvious)The main evaluation criteria is the dissemination plan. The above issues should be described as work to be done in the dissemination work package
42 ABSTRACTShould start with objectives followed by the consortium logicThen should describe the implementation plan with the expected resultsand finish with the impact through the dissemination plan
43 Self-assessment process Most of failed proposals are taking a low mark in 1-2 evaluation criteria (relevance is the most usual).The pre-evaluation of the proposal by internal or external executives (definitely not involved in the proposal preparation process) based on the published evaluation criteria.
44 ConclusionBecome evaluator is the best way to learning to write proposalsUnderstanding how EC worksNetworkingWell paid