1. The evaluation process in the 7th Framework programme for Research and Technological Development
Chisinau, November 6th, 2012
Dr Sebastiano FUMERO
Head of Unit “FP7 support” European Commission Research Executive Agency

2. Overview How to apply The evaluation process: basic facts and figures
Role of Commission/REA staff
Key issues:
Eligibility check
Expert selection
Conflicts of interest
The criteria
The observer
Redress

3. FP7: how to apply Participant portal Calls for proposals Annual Work
Budget, deadline, OJ ref., legal documents
Links to
SEP
NCP
Annual
Work
Programme
Year N
Guide for Applicants
Call X
Funding Scheme Y
SEP
Electronic proposal submission system

4. The Participant Portal (PP)

5. The Participant Portal (PP)

6. The Participant Portal (PP)

7. To consider prior to submission
Rules on submission and evaluation
The common reference for FP7
Consistency vs flexibility!
Guide for applicants (annexes 1 and 2)
The work programme
The topics and criteria against which the proposals will be judged (all criteria are important – consider sub-criteria - think as an evaluator)

8. Submission Drafting the Proposal
Respect page limits specified in guide for applicants
Ensure you meet the minimum eligibility requirements
Excellent science is a condition but not enough. Consider also: - impact, dissemination and IPS - consider project implementation and management (role of coordinator is essential)
Be precise, less is sometime more…
Impartial view…ask your colleagues, friend to read it before…
Start with SEP asap – a missed deadline implies proposal is not admissible
administrative data (part A forms) should be consistent with info in part B

9. Evaluation process Submission Individual evaluation Consensus Panel
May be
“remote”
May be
“remote”
Submission
Individual
evaluation
Consensus
Panel
Finalisation
Full Proposal
Proposal
forms
Final ranking
list
Evaluators
Evaluators
Evaluators
Rejection list
Criteria
Criteria
Criteria
Proposals in
suggested
priority order
Eligibility
COMMISSION
COMMISSION
Experts' role

10. Applicants are informed
Evaluation process
Proposal
Individual Evaluation
Consensus
Panel review
Hearings
For large projects
(optional)
Thresholds
Eligibility
Negotiation
Commission ranking
Commission rejection
decision
Remote or in Bruxelles
Quick Information Letter
Applicants are informed
of the Commission
decision
Commission funding
decision

11. Basic facts and figures
Evaluation of proposals
Basic facts and figures
Funding decisions are based on peer review of research proposals
Peer review can also “add value” to projects
High quality evaluators are at the core of the system
Over experts registered for FP7 in the old database
Over experts registered in the new database (Expert Area in the Participant Portal)
Over independent experts engaged in 2011
Approx proposals evaluated in 2011

12. Evaluation Process: basic principles
Evaluation of proposals
Evaluation Process: basic principles
EXCELLENCE
TRANSPARENCY
FAIRNESS &
IMPARTIALITY
CONFIDENTIALITY
ETHICAL & SECURITY
CONSIDERATIONS
EFFICIENCY &
SPEED

13. Three References Evaluation of proposals
RULES
Submission & Evaluation
Rules on submission and evaluation
The common reference for FP7
Consistency vs flexibility!
Guide for applicants (annexes 1 and 2)
The work programme
The topics and criteria against which the proposals will be judged
Guide for Applicants
Call X
Funding
Scheme Y
Annual
Work
Programme

14. Some basic misconceptions clarified
Evaluation of proposals
Some basic misconceptions clarified
The EU’s peer review system is not a political process
Lobbying has no influence
Quality of the proposal is the sole criterion for success
However, “quality” involves a number of factors

15. Role of Commission/REA staff
Evaluation of proposals
Role of Commission/REA staff
Check the eligibility of proposals
Oversee work of experts
Conduct briefings
Moderate discussions
Organise the panel and its work
Ensure coherence and consistency
May advise on:
Background on previously supported or on-going projects
Relevant supplementary information (directives, regulations, policies, etc.)
(Can even act as experts!)

16. Eligibility checks Evaluation of proposals
Receipt of proposal before deadline
Firm deadlines (SEP)
Minimum number of eligible, independent partners
As set out in work programme and the call
Completeness of proposal
Presence of all requested forms
and readable, accessible and printable
"In scope" vs "Out of scope"
Others

17. Expert selection Evaluation of proposals Based on:
A high level of expertise
An appropriate range of competences
If the above conditions can be satisfied, then also:
Balance academic/industrial
Gender
Geography
Rotation
But also, of course constrained by:
Availability
Avoidance of conflicts of interest
Uncertainty over number and exact coverage of proposals
Not an easy process…!!!

18. FP7 Expert from this area
Number of experts having supported the EC in FP7 by country
Country
Number of Experts (participation)
Greece EL
646
Romania RO
365
Hungary HU
311
Bulgaria BG
165
Slovenia SI
163
Slovakia SK 95
Croatia HR 76
Serbia RS 62
Macedonia MK 8
Bosnia and Herzegovina BA 5
Moldova MD 3
Number of experts registered in the EMPP by country
Country
Migrated
Valid
Bosnia and Herzegovina BA 8
Bulgaria BG
204
162
Croatia HU
115 89
Greece HR
914
687
Hungary
297
Macedonia MK 25 21
Moldova MD 16 12
Romania RO
464
361
Serbia RS
101 78
Slovakia SK
121 94
Slovenia SI
166
123

19. Conflicts of interest (1)
Expert selection
Conflicts of interest (1)
More clarity in FP7
Types of COI set out in appointment letter
Disqualifying COI
Involved in preparation of proposal
Stands to benefit directly
Close family relationship
Director/trustee/partner
Employee (but, see exception…)
Member of advisory group
Any other situation that compromises impartiality
Potential COI
Employed in last 3 years
Involved in research collaboration in previous 3 years
Any other situation that casts doubt… or that could reasonably appear to do so…

20. Conflicts of interest (2)
Expert selection
Conflicts of interest (2)
Experts with a “disqualifying” COI cannot evaluate
Neither in consensus group considering “problematic” proposal
Nor in final panel
One exception… if:
The expert is employed in same organisation, but different department/lab/institute (e.g. CNRS)
The constituent bodies operate with a high degree of autonomy
Justified by the limited pool of qualified experts
… then the Commission/REA might allow expert to participate in a panel review
Should abstain if the specific proposal is discussed
Exceptionally (very rare!!!), might participate in consensus group
Experts with a “potential” COI
Need to consider circumstances of case

21. The evaluation criteria
1. S/T quality (in relation to the topics addressed by the call)
2. Implementation
3. Impact
Sound concept, and quality of objectives
Appropriateness of the management structure and procedures
Quality and relevant experience of the individual participants
Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

22. The evaluation criteria
Some exceptions
Marie-Curie schemes for training and mobility of researchers
Include, e.g. quality of training programme, suitability of host institution, etc.
European Research Council (ERC) grants
Scientific quality only criterion
Excellence!!!

23. Each criterion is scored 0-5
Proposal scoring
Each criterion is scored 0-5
Half-scores allowed
Whole range should be considered
Scores must pass thresholds if a proposal is to be considered for funding
Thresholds apply to individual criteria:
Default threshold is 3
… and to the total score
Higher than the sum of the individual thresholds
Default threshold is 10
Can vary from call to call!

24. Interpretation of the scores
Proposal scoring
Interpretation of the scores
0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information
1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses.
2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses.
3 - Good. The proposal addresses the criterion well, although improvements would be necessary.
4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible.
5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

25. Individual reading Evaluation process
The experts evaluators first carry out individual readings (often done remotely)
The experts:
Evaluate the proposal individually (without discussing with the other evaluators)
Check whether the proposal is “in scope” second check after the one done by the EC
Complete an Individual Evaluation Report (IER) giving scores and comments on all criteria
Scores should be in line with comments

26. IER Individual Evaluation Report
Evaluation process
May be remote
Proposal 1
IER Individual Evaluation Report
Expert A
Consensus meeting
Consensus: Scores & comments
Proposal 1
IER
Expert B
Proposal 1
HEARING (optional) CR
Consensus Report
IER
Expert C
One proposal can be evaluated by more than 3 experts

27. Consensus Evaluation process
Build on the basis of the individual assessments of all the evaluators
Usually involves a discussion
Moderated by a Commission/ REA representative
Agreement on consensus scores and comments for each of the criteria
One expert acts as rapporteur

28. Consensus reports – key points
Evaluation process
Consensus reports – key points
The rapporteur is responsible for drafting the consensus report (CR)
Includes consensus marks and comments
The quality of the CR is paramount
The aim is
A clear assessment of the proposal, with justification
Clear feedback on weaknesses and strengths
To be avoided
Comments that do not correspond with the scores
Recommendations in view of resubmission
A proof reader might be appointed for quality control

29. Hearings Evaluation process
Co-ordinators whose proposals have passed the thresholds are invited to Brussels
Intended to clarify any points raised by the experts in advance
Not an occasion to “improve” the proposal
Not an occasion for a multi-media show!

30. The final Panel Review Evaluation process
Key function is to ensure consistency
Final marks and comments for each proposal
Evaluation Summary reports (ESR)
New scores (if necessary)… carefully justified
Clear guidance for contract negotiation
Split proposals with identical consensus scores
Approach is spelled out in WP and GFA
Resolve cases where a minority view was recorded in CR
[Exceptionally] recommendations for combining
List of proposals for priority order

31. “Initial information” to applicants Sending of ESR
Information to proposers
“Initial information” to applicants Sending of ESR
The Commission/REA does not change the ESR, except if necessary to:
Improve readability
[Exceptionally] To remove factual errors or inappropriate comments that may have escaped earlier proof-reading
The scores are never changed
The ESR is sent to the proposal co-ordinator – no commitments at this stage regarding funding
This is the public face of the evaluation!

32. Redress? Appeal In the past, complaints arrived haphazardly
Handled at different levels
No systematic treatment
No common record
The redress procedure introduced for FP7 does not give a new right of appeal…… but it ensures a consistent and coherent approach to complaints
Establishes “due process”
Uphold principles of transparency and equal treatment

33. Redress: Principles and guidelines
Appeal
Redress: Principles and guidelines
Redress will not “stop the train”
Non-contentious proposals negotiated and selected as normal
Complaints must relate to shortcomings in the handling of proposal evaluation
Before a Commission decision has been made
The procedure will not call into question the judgement of appropriately qualified experts

34. Independent observers
Monitoring
Independent observers
Provide assurance that the process is fair
And can provide constructive advice
Not experts in the scientific area concerned
Their reports are made available to the Programme Committee
IOs are convened annually to a Round Table
What are the common issues?

35. Commission/REA follow-up
Evaluation process
Commission/REA follow-up
Evaluation summary reports sent to applicants
Draw up final ranking lists
Information to the Programme Committee
Contract negotiation
Formal consultation of Programme Committee (when required)
Commission decisions
Survey of evaluators
Independent Observers’ reports

36. Expert questionnaire Survey
For every call, experts receive a message on returning home
Invited to complete an on-line survey
Personal profile
Evaluation process
Evaluation criteria
Opinion on the task and the other evaluators
Logistics
Comments and recommendations
Early results sent to call co-ordinator after one month
Full analysis at end of the year

37. Survey
96% of the respondents found the quality of the evaluation overall 'satisfactory' to 'excellent'

38. Survey
91% found the EU evaluation process similar or better than national or international schemes

39. Links EU research: http://ec.europa.eu/research/
7th Framework Programme:
Information on research activity and projects:
Questions? Contact the Research Enquiry Service 39 39

40. Thank you for your attention!
Dr. Sebastiano FUMERO
Head of Unit “FP7 Support”
European Commission
Research Executive Agency
Tel: