Presentation is loading. Please wait.

Presentation is loading. Please wait.

Policy REA - Research Executive Agency Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency The evaluation process.

Similar presentations

Presentation on theme: "Policy REA - Research Executive Agency Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency The evaluation process."— Presentation transcript:

1 Policy REA - Research Executive Agency Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency The evaluation process in the 7 th Framework programme for Research and Technological Development Chisinau, November 6 th, 2012

2 Overview  How to apply  The evaluation process: basic facts and figures  Role of Commission/REA staff  Key issues: Eligibility check Expert selection Conflicts of interest The criteria The observer Redress

3 SEP Electronic proposal submission system Annual Work Programme Year N Guide for Applicants Call X Funding Scheme Y FP7: how to apply Participant portal Calls for proposals Budget, deadline, OJ ref., legal documents Links to SEP NCP

4 The Participant Portal (PP)



7 7  Rules on submission and evaluation The common reference for FP7 Consistency vs flexibility!  Guide for applicants (annexes 1 and 2) The common reference for FP7  The work programme The topics and criteria against which the proposals will be judged (all criteria are important – consider sub-criteria - think as an evaluator) To consider prior to submission Submission

8 8 Drafting the Proposal Respect page limits specified in guide for applicants Ensure you meet the minimum eligibility requirements Excellent science is a condition but not enough. Consider also: - impact, dissemination and IPS - consider project implementation and management (role of coordinator is essential) Be precise, less is sometime more… Impartial view…ask your colleagues, friend to read it before… Start with SEP asap – a missed deadline implies proposal is not admissible administrative data (part A forms) should be consistent with info in part B Submission

9 Full Proposal Proposal forms Evaluators Eligibility Evaluators Final ranking list PanelSubmissionConsensus Individual evaluation Proposals in suggested priority order Rejection list Finalisation Criteria COMMISSION May be “remote” Experts' role Evaluation process May be “remote”

10 Applicants are informed of the Commission decision Commission funding decision Quick Information Letter Proposal Individual Evaluation Consensus Panel review Hearings For large projects (optional) Thresholds Eligibility Negotiation Commission ranking Commission rejection decision Remote or in Bruxelles Evaluation process

11 Basic facts and figures  Funding decisions are based on peer review of research proposals Peer review can also “add value” to projects  High quality evaluators are at the core of the system  Over 118.000 experts registered for FP7 in the old database  Over 15.000 experts registered in the new database (Expert Area in the Participant Portal)  Over 8 300 independent experts engaged in 2011  Approx. 27 000 proposals evaluated in 2011 Evaluation of proposals

12 Evaluation Process: basic principles Evaluation of proposals

13 Three References  Rules on submission and evaluation The common reference for FP7 Consistency vs flexibility!  Guide for applicants (annexes 1 and 2) The common reference for FP7  The work programme The topics and criteria against which the proposals will be judged Evaluation of proposals Annual Work Programme Guide for Applicants Call X Funding Scheme Y RULES Submission & Evaluation RULES Submission & Evaluation

14 Some basic misconceptions clarified  The EU’s peer review system is not a political process Lobbying has no influence  Quality of the proposal is the sole criterion for success However, “quality” involves a number of factors Evaluation of proposals

15 Role of Commission/REA staff  Check the eligibility of proposals  Oversee work of experts  Conduct briefings  Moderate discussions  Organise the panel and its work  Ensure coherence and consistency  May advise on: Background on previously supported or on-going projects Relevant supplementary information (directives, regulations, policies, etc.)  (Can even act as experts!) Evaluation of proposals

16 Eligibility checks  Receipt of proposal before deadline Firm deadlines (SEP)  Minimum number of eligible, independent partners As set out in work programme and the call  Completeness of proposal Presence of all requested forms and readable, accessible and printable  "In scope" vs "Out of scope"  Others Evaluation of proposals

17 Expert selection  Based on: A high level of expertise An appropriate range of competences  If the above conditions can be satisfied, then also: Balance academic/industrial Gender Geography Rotation  But also, of course constrained by: Availability Avoidance of conflicts of interest Uncertainty over number and exact coverage of proposals  Not an easy process…!!! Evaluation of proposals

18 Number of experts having supported the EC in FP7 by country Number of experts registered in the EMPP by country FP7 Expert from this area Country Number of Experts (participation) Greece EL646 Romania RO365 Hungary HU311 Bulgaria BG165 Slovenia SI163 Slovakia SK95 Croatia HR76 Serbia RS62 Macedonia MK8 Bosnia and Herzegovina BA5 Moldova MD3 Country MigratedValid Bosnia and Herzegovina BA 88 Bulgaria BG 204162 Croatia HU 11589 Greece HR 914687 Hungary HU 297204 Macedonia MK 2521 Moldova MD 1612 Romania RO 464361 Serbia RS 10178 Slovakia SK 12194 Slovenia SI 166123

19 Conflicts of interest (1)  More clarity in FP7  Types of COI set out in appointment letter  Disqualifying COI Involved in preparation of proposal Stands to benefit directly Close family relationship Director/trustee/partner Employee (but, see exception…) Member of advisory group Any other situation that compromises impartiality  Potential COI Employed in last 3 years Involved in research collaboration in previous 3 years Any other situation that casts doubt… or that could reasonably appear to do so… Expert selection

20 Conflicts of interest (2)  Experts with a “disqualifying” COI cannot evaluate Neither in consensus group considering “problematic” proposal Nor in final panel One exception… if:  The expert is employed in same organisation, but different department/lab/institute (e.g. CNRS)  The constituent bodies operate with a high degree of autonomy  Justified by the limited pool of qualified experts … then the Commission/REA might allow expert to participate in a panel review  Should abstain if the specific proposal is discussed Exceptionally (very rare!!!), might participate in consensus group  Experts with a “potential” COI Need to consider circumstances of case Expert selection

21 1. S/T quality (in relation to the topics addressed by the call) 2. Implementation3. Impact Sound concept, and quality of objectives Appropriateness of the management structure and procedures Quality and relevant experience of the individual participants Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity The evaluation criteria

22 Some exceptions  Marie-Curie schemes for training and mobility of researchers Include, e.g. quality of training programme, suitability of host institution, etc.  European Research Council (ERC) grants Scientific quality only criterion Excellence!!! The evaluation criteria

23 Proposal scoring  Each criterion is scored 0-5 Half-scores allowed Whole range should be considered Scores must pass thresholds if a proposal is to be considered for funding  Thresholds apply to individual criteria: Default threshold is 3  … and to the total score Higher than the sum of the individual thresholds Default threshold is 10  Can vary from call to call!

24 Interpretation of the scores 0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses. 2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses. 3 - Good. The proposal addresses the criterion well, although improvements would be necessary. 4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible. 5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor. Proposal scoring

25 Individual reading The experts evaluators first carry out individual readings (often done remotely) The experts:  Evaluate the proposal individually (without discussing with the other evaluators)  Check whether the proposal is “in scope” second check after the one done by the EC  Complete an Individual Evaluation Report (IER) giving scores and comments on all criteria Scores should be in line with comments Evaluation process

26 Proposal 1 IER Individual Evaluation Report Expert A IER Expert B IER Expert C Consensus meeting CR Consensus Report One proposal can be evaluated by more than 3 experts May be remote Consensus: Scores & comments HEARING (optional) Evaluation process

27 Consensus  Build on the basis of the individual assessments of all the evaluators  Usually involves a discussion  Moderated by a Commission/ REA representative  Agreement on consensus scores and comments for each of the criteria  One expert acts as rapporteur Evaluation process

28 Consensus reports – key points  The rapporteur is responsible for drafting the consensus report (CR) Includes consensus marks and comments  The quality of the CR is paramount  The aim is A clear assessment of the proposal, with justification Clear feedback on weaknesses and strengths  To be avoided Comments that do not correspond with the scores Recommendations in view of resubmission A proof reader might be appointed for quality control Evaluation process

29 Hearings  Co-ordinators whose proposals have passed the thresholds are invited to Brussels  Intended to clarify any points raised by the experts in advance  Not an occasion to “improve” the proposal  Not an occasion for a multi-media show! Evaluation process

30 The final Panel Review  Key function is to ensure consistency  Final marks and comments for each proposal Evaluation Summary reports (ESR) New scores (if necessary)… carefully justified Clear guidance for contract negotiation  Split proposals with identical consensus scores Approach is spelled out in WP and GFA  Resolve cases where a minority view was recorded in CR  [Exceptionally] recommendations for combining  List of proposals for priority order Evaluation process

31 “Initial information” to applicants Sending of ESR  The Commission/REA does not change the ESR, except if necessary to: Improve readability [Exceptionally] To remove factual errors or inappropriate comments that may have escaped earlier proof-reading  The scores are never changed  The ESR is sent to the proposal co-ordinator – no commitments at this stage regarding funding  This is the public face of the evaluation! Information to proposers

32 Redress?  In the past, complaints arrived haphazardly Handled at different levels No systematic treatment No common record  The redress procedure introduced for FP7 does not give a new right of appeal…… but it ensures a consistent and coherent approach to complaints Establishes “due process” Uphold principles of transparency and equal treatment Appeal

33 Redress: Principles and guidelines  Redress will not “stop the train” Non-contentious proposals negotiated and selected as normal  Complaints must relate to shortcomings in the handling of proposal evaluation Before a Commission decision has been made  The procedure will not call into question the judgement of appropriately qualified experts Appeal

34 Independent observers  Provide assurance that the process is fair And can provide constructive advice Not experts in the scientific area concerned  Their reports are made available to the Programme Committee  IOs are convened annually to a Round Table What are the common issues? Monitoring

35 Commission/REA follow-up  Evaluation summary reports sent to applicants  Draw up final ranking lists  Information to the Programme Committee  Contract negotiation  Formal consultation of Programme Committee (when required)  Commission decisions  Survey of evaluators  Independent Observers’ reports Evaluation process

36 Expert questionnaire  For every call, experts receive a message on returning home  Invited to complete an on-line survey Personal profile Evaluation process Evaluation criteria Opinion on the task and the other evaluators Logistics Comments and recommendations  Early results sent to call co-ordinator after one month  Full analysis at end of the year Survey

37 96% of the respondents found the quality of the evaluation overall 'satisfactory' to 'excellent' Survey

38 91% found the EU evaluation process similar or better than national or international schemes Survey

39 Policy REA - Research Executive Agency EU research: 7 th Framework Programme: Information on research activity and projects: Questions? Contact the Research Enquiry Service Links

40 Policy REA - Research Executive Agency Thank you for your attention! Dr. Sebastiano FUMERO Head of Unit “FP7 Support” European Commission Research Executive Agency Tel: +32-2-296 96 88

Download ppt "Policy REA - Research Executive Agency Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency The evaluation process."

Similar presentations

Ads by Google