Presentation on theme: "Chisinau, November 6th, 2012 Dr Sebastiano FUMERO"— Presentation transcript:
1 The evaluation process in the 7th Framework programme for Research and Technological Development Chisinau, November 6th, 2012Dr Sebastiano FUMEROHead of Unit “FP7 support” European Commission Research Executive Agency
2 Overview How to apply The evaluation process: basic facts and figures Role of Commission/REA staffKey issues:Eligibility checkExpert selectionConflicts of interestThe criteriaThe observerRedress
3 FP7: how to apply Participant portal Calls for proposals Annual Work Budget, deadline, OJ ref., legal documentsLinks toSEPNCPAnnualWorkProgrammeYear NGuide for ApplicantsCall XFunding Scheme YSEPElectronic proposal submission system
7 To consider prior to submission Rules on submission and evaluationThe common reference for FP7Consistency vs flexibility!Guide for applicants (annexes 1 and 2)The work programmeThe topics and criteria against which the proposals will be judged (all criteria are important – consider sub-criteria - think as an evaluator)
8 Submission Drafting the Proposal Respect page limits specified in guide for applicantsEnsure you meet the minimum eligibility requirementsExcellent science is a condition but not enough. Consider also: - impact, dissemination and IPS - consider project implementation and management (role of coordinator is essential)Be precise, less is sometime more…Impartial view…ask your colleagues, friend to read it before…Start with SEP asap – a missed deadline implies proposal is not admissibleadministrative data (part A forms) should be consistent with info in part B
9 Evaluation process Submission Individual evaluation Consensus Panel May be“remote”May be“remote”SubmissionIndividualevaluationConsensusPanelFinalisationFull ProposalProposalformsFinal rankinglistEvaluatorsEvaluatorsEvaluatorsRejection listCriteriaCriteriaCriteriaProposals insuggestedpriority orderEligibilityCOMMISSIONCOMMISSIONExperts' role
10 Applicants are informed Evaluation processProposalIndividual EvaluationConsensusPanel reviewHearingsFor large projects(optional)ThresholdsEligibilityNegotiationCommission rankingCommission rejectiondecisionRemote or in BruxellesQuick Information LetterApplicants are informedof the CommissiondecisionCommission fundingdecision
11 Basic facts and figures Evaluation of proposalsBasic facts and figuresFunding decisions are based on peer review of research proposalsPeer review can also “add value” to projectsHigh quality evaluators are at the core of the systemOver experts registered for FP7 in the old databaseOver experts registered in the new database (Expert Area in the Participant Portal)Over independent experts engaged in 2011Approx proposals evaluated in 2011
13 Three References Evaluation of proposals RULESSubmission & EvaluationRules on submission and evaluationThe common reference for FP7Consistency vs flexibility!Guide for applicants (annexes 1 and 2)The work programmeThe topics and criteria against which the proposals will be judgedGuide for ApplicantsCall XFundingScheme YAnnualWorkProgramme
14 Some basic misconceptions clarified Evaluation of proposalsSome basic misconceptions clarifiedThe EU’s peer review system is not a political processLobbying has no influenceQuality of the proposal is the sole criterion for successHowever, “quality” involves a number of factors
15 Role of Commission/REA staff Evaluation of proposalsRole of Commission/REA staffCheck the eligibility of proposalsOversee work of expertsConduct briefingsModerate discussionsOrganise the panel and its workEnsure coherence and consistencyMay advise on:Background on previously supported or on-going projectsRelevant supplementary information (directives, regulations, policies, etc.)(Can even act as experts!)
16 Eligibility checks Evaluation of proposals Receipt of proposal before deadlineFirm deadlines (SEP)Minimum number of eligible, independent partnersAs set out in work programme and the callCompleteness of proposalPresence of all requested formsand readable, accessible and printable"In scope" vs "Out of scope"Others
17 Expert selection Evaluation of proposals Based on: A high level of expertiseAn appropriate range of competencesIf the above conditions can be satisfied, then also:Balance academic/industrialGenderGeographyRotationBut also, of course constrained by:AvailabilityAvoidance of conflicts of interestUncertainty over number and exact coverage of proposalsNot an easy process…!!!
18 FP7 Expert from this area Number of experts having supported the EC in FP7 by countryCountryNumber of Experts (participation)GreeceEL646RomaniaRO365HungaryHU311BulgariaBG165SloveniaSI163SlovakiaSK95CroatiaHR76SerbiaRS62MacedoniaMK8Bosnia and HerzegovinaBA5MoldovaMD3Number of experts registered in the EMPP by countryCountryMigratedValidBosnia and HerzegovinaBA8BulgariaBG204162CroatiaHU11589GreeceHR914687Hungary297MacedoniaMK2521MoldovaMD1612RomaniaRO464361SerbiaRS10178SlovakiaSK12194SloveniaSI166123
19 Conflicts of interest (1) Expert selectionConflicts of interest (1)More clarity in FP7Types of COI set out in appointment letterDisqualifying COIInvolved in preparation of proposalStands to benefit directlyClose family relationshipDirector/trustee/partnerEmployee (but, see exception…)Member of advisory groupAny other situation that compromises impartialityPotential COIEmployed in last 3 yearsInvolved in research collaboration in previous 3 yearsAny other situation that casts doubt… or that could reasonably appear to do so…
20 Conflicts of interest (2) Expert selectionConflicts of interest (2)Experts with a “disqualifying” COI cannot evaluateNeither in consensus group considering “problematic” proposalNor in final panelOne exception… if:The expert is employed in same organisation, but different department/lab/institute (e.g. CNRS)The constituent bodies operate with a high degree of autonomyJustified by the limited pool of qualified experts… then the Commission/REA might allow expert to participate in a panel reviewShould abstain if the specific proposal is discussedExceptionally (very rare!!!), might participate in consensus groupExperts with a “potential” COINeed to consider circumstances of case
21 The evaluation criteria 1. S/T quality (in relation to the topics addressed by the call)2. Implementation3. ImpactSound concept, and quality of objectivesAppropriateness of the management structure and proceduresQuality and relevant experience of the individual participantsContribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity
22 The evaluation criteria Some exceptionsMarie-Curie schemes for training and mobility of researchersInclude, e.g. quality of training programme, suitability of host institution, etc.European Research Council (ERC) grantsScientific quality only criterionExcellence!!!
23 Each criterion is scored 0-5 Proposal scoringEach criterion is scored 0-5Half-scores allowedWhole range should be consideredScores must pass thresholds if a proposal is to be considered for fundingThresholds apply to individual criteria:Default threshold is 3… and to the total scoreHigher than the sum of the individual thresholdsDefault threshold is 10Can vary from call to call!
24 Interpretation of the scores Proposal scoringInterpretation of the scores0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses.2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses.3 - Good. The proposal addresses the criterion well, although improvements would be necessary.4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible.5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.
25 Individual reading Evaluation process The experts evaluators first carry out individual readings (often done remotely)The experts:Evaluate the proposal individually (without discussing with the other evaluators)Check whether the proposal is “in scope” second check after the one done by the ECComplete an Individual Evaluation Report (IER) giving scores and comments on all criteriaScores should be in line with comments
26 IER Individual Evaluation Report Evaluation processMay be remoteProposal 1IER Individual Evaluation ReportExpert AConsensus meetingConsensus: Scores & commentsProposal 1IERExpert BProposal 1HEARING (optional)CRConsensus ReportIERExpert COne proposal can be evaluated by more than 3 experts
27 Consensus Evaluation process Build on the basis of the individual assessments of all the evaluatorsUsually involves a discussionModerated by a Commission/ REA representativeAgreement on consensus scores and comments for each of the criteriaOne expert acts as rapporteur
28 Consensus reports – key points Evaluation processConsensus reports – key pointsThe rapporteur is responsible for drafting the consensus report (CR)Includes consensus marks and commentsThe quality of the CR is paramountThe aim isA clear assessment of the proposal, with justificationClear feedback on weaknesses and strengthsTo be avoidedComments that do not correspond with the scoresRecommendations in view of resubmissionA proof reader might be appointed for quality control
29 Hearings Evaluation process Co-ordinators whose proposals have passed the thresholds are invited to BrusselsIntended to clarify any points raised by the experts in advanceNot an occasion to “improve” the proposalNot an occasion for a multi-media show!
30 The final Panel Review Evaluation process Key function is to ensure consistencyFinal marks and comments for each proposalEvaluation Summary reports (ESR)New scores (if necessary)… carefully justifiedClear guidance for contract negotiationSplit proposals with identical consensus scoresApproach is spelled out in WP and GFAResolve cases where a minority view was recorded in CR[Exceptionally] recommendations for combiningList of proposals for priority order
31 “Initial information” to applicants Sending of ESR Information to proposers“Initial information” to applicants Sending of ESRThe Commission/REA does not change the ESR, except if necessary to:Improve readability[Exceptionally] To remove factual errors or inappropriate comments that may have escaped earlier proof-readingThe scores are never changedThe ESR is sent to the proposal co-ordinator – no commitments at this stage regarding fundingThis is the public face of the evaluation!
32 Redress? Appeal In the past, complaints arrived haphazardly Handled at different levelsNo systematic treatmentNo common recordThe redress procedure introduced for FP7 does not give a new right of appeal…… but it ensures a consistent and coherent approach to complaintsEstablishes “due process”Uphold principles of transparency and equal treatment
33 Redress: Principles and guidelines AppealRedress: Principles and guidelinesRedress will not “stop the train”Non-contentious proposals negotiated and selected as normalComplaints must relate to shortcomings in the handling of proposal evaluationBefore a Commission decision has been madeThe procedure will not call into question the judgement of appropriately qualified experts
34 Independent observers MonitoringIndependent observersProvide assurance that the process is fairAnd can provide constructive adviceNot experts in the scientific area concernedTheir reports are made available to the Programme CommitteeIOs are convened annually to a Round TableWhat are the common issues?
35 Commission/REA follow-up Evaluation processCommission/REA follow-upEvaluation summary reports sent to applicantsDraw up final ranking listsInformation to the Programme CommitteeContract negotiationFormal consultation of Programme Committee (when required)Commission decisionsSurvey of evaluatorsIndependent Observers’ reports
36 Expert questionnaire Survey For every call, experts receive a message on returning homeInvited to complete an on-line surveyPersonal profileEvaluation processEvaluation criteriaOpinion on the task and the other evaluatorsLogisticsComments and recommendationsEarly results sent to call co-ordinator after one monthFull analysis at end of the year
37 Survey96% of the respondents found the quality of the evaluation overall 'satisfactory' to 'excellent'
38 Survey91% found the EU evaluation process similar or better than national or international schemes
39 Links EU research: http://ec.europa.eu/research/ 7th Framework Programme:Information on research activity and projects:Questions? Contact the Research Enquiry Service3939
40 Thank you for your attention! Dr. Sebastiano FUMEROHead of Unit “FP7 Support”European CommissionResearch Executive AgencyTel: