1. APRE Agency for the Promotion of European Research
Evaluation and submission procedures
in Seventh Framework Programme
Sabrina Bozzoli

2. Overview Information for proposers Writing your proposal Getting help
Submission
Selection
Evaluation
Eligible?
Information for proposers
Writing your proposal
Getting help
Submission of proposal
Eligibility checks
Evaluation (experts)
Selection
Poznan, 23rd January 2007

3. Information for proposers
Call fiche
Workprogramme
Guide for Applicants (now including the Guidance notes for evaluators, the evaluation criteria and the Background note on the funding scheme)
Model grant agreement
Rules for participation
Rules on submission of proposals, and the related evaluation, selection and award procedures
EPSS manual
Poznan, 23rd January 2007

4. When writing your proposal…(1)
Divide your efforts over the evaluation criteria
[Many proposers concentrate on the scientific element, but loose marks on project implementation or impact description]
a strong title, acronym
an interesting project summary (objectives, results, R&D-approach, partnership, utility of results, exploitation)
convincing technology background and state of the art
clear objectives, methods, results and deliverables
well designed work plan
appropriate management structures & procedures
detailed implementation & exploitation of results
realistic costs that lie within the budget of the Call
convincing consortium (roles, qualifications)
keep it clear and simple without loosing quality
The proposal is not a scientigic paper!!!
Poznan, 23rd January 2007

5. When writing your proposal…(2)
Think of the finishing touches which contribute to the quality of work
Use coincise standard English
Make text clear, well structured, easy to read
Add a table of contents, use short paragraphs, highlight key points in italics, use bullet points to break up lists
Include only relevant information
Make your proposal virtually attractive and inviting using graphic devices
Poznan, 23rd January 2007

6. Getting help with your proposal +
National Contact Points
practical information & advice on modalities and rules
National delegates in Programme Committees
background information on EU policies
Innovation Relay Centre (IRC)
EC Information desks (reachable by or phone. eg. IPR Help-desk)
Get early advice from NCP and EC Information Desk
Show your proposal to critical (experienced) collegues
Inform NCP and National delegates in Programme Committees
Submit complete proposal before deadline
Poznan, 23rd January 2007

7. Submission Fixed deadline calls
Electronic proposal submission system (EPSS) only
Proposal template given in the ‘Guide for Applicants’
Closely aligned to the evaluation criteria
Proposals are normally submitted and evaluated in a single stage
Two-stage submission of proposals
May be used for large, ‘bottom up’ calls
First stage
short proposal (about pages), dealing with main scientific concepts and ideas
use of limited set of criteria
successful proposers invited to submit complete proposals
Poznan, 23rd January 2007

8. Electronic submission
EPSS
[Electronic Proposal Submission System]
Online preparation only!
Improved validation checks before submission is accepted
FP6 Failure rate = + 1%
Main reason for failure - waiting till the last minute
Submit early, submit often!
Proposals must be submitted electronically, using the Commission's Electronic ProposalSubmission Service (EPSS) Proposals arriving at the Commission by any other means are regarded as ‘not submitted’, and will not be evaluated. All the data that you upload is securely stored on a server to which only you and the other participants in the proposal have access until the deadline. This data is encrypted until the close of the call.
You can access the EPSS from the call page on CORDIS. Full instructions will be found in the “EPSS preparation and submission guide”. This will be available from the CORDIS site early in 2007.
Poznan, 23rd January 2007

9. Proposal Part A A1 A2 A3 Title, acronym, objective etc. Free keywords
2000 character proposal abstract
Previous/current submission (in FP7) A2
Legal address/administrator address/R&D address
Clear identification as SME/Public body/Research centre/ Higher and Secondary education establishment A3
More cost details (direct/indirect costs distinguished)
Section A1 gives a snapshot of your proposal, section A2 concerns you and your organisation, while section A3 deals with money matters.
Please note:
• The coordinator fills in the section A1and section A3.
• The participants already identified at the time of proposal submission (including the coordinator) each fill in section A2.
• Subcontractors are not required to fill in section A2 and should not be listed separately in section A3.
• The estimated budget planned for any future participants (not yet identified at the time of the proposal) is not shown separately in form A3 but should be added to the coordinator’s budget. Their role, profile and tasks are described in Part B of the proposal.
Poznan, 23rd January 2007

10. Proposal Part B (pdf format only)
Part B format directly linked to evaluation criteria
Summary
S&T quality (bullet points = sections)
Implementation (idem)
Impact (idem)
Ethics
Section lengths recommended
Details of the work you intend to carry out will be described in part B
Poznan, 23rd January 2007

11. Elegibility checks
Date and time of receipt of proposal on or before deadline
Firm deadlines - except for Continuously Open Calls
Minimum number of eligible, independent partners
As set out in work programme/call
Completeness of proposal
Presence of all requested administrative forms (Part A) and the content description (Part B)
“Out of scope”
Others (eg. budget limits)
It is possible that a proposal is found to be completely out of scope of the call during the course of the individual evaluation, and therefore not relevant. If an expert suspects that this may be the case, a Commission staff member will be informed immediately, and the views of the other experts will be sought.
If the consensus view is that the main part of the proposal is not relevant to the topics of the call, the proposal will be withdrawn from the evaluation, and the proposal will be deemed ineligible.
Poznan, 23rd January 2007

12. Evaluation - FP7 No major change for FP7
But improved and streamlined, based on experience
Adapted to the new features of FP7 where necessary
What’s new?
Clearer page limits
Elegibility criteria (includes “scope”)
Evaluation criteria (3 instead of 5 or 6)
More clarity on conflicts of interest (Indipendent experts)
WHAT’S NEW?
Any page going behind the maximum indicated by the EC will be disregarded.
An extra eligibility criterion regarding the “scope of the project”
The clarity of conflict of interest will be assured through the appointment of indipendent experts. Conflicts of interest: Under the terms of the appointment letter, experts must declare beforehand any known conflicts of interest, and must immediately inform a Commission staff member if one becomes apparent during the course of the evaluation. The Commission will take whatever action is necessary to remove any conflict.
Poznan, 23rd January 2007

13. Evaluation – The criteria
Adapted to each funding scheme and each thematic area
Specified in the specific programme/work programme (Annex 2)
Divided into three main criteria:
S&T Quality (relevant to the topics addressed by the call)
quality of the objectives
progress beyond the state of the art
work plan
Implementation
individual participants and consortium as a whole
allocation of resources (budget, staff, equipment)
Impact
Contribution to expected impacts listed in work programme
Plans for dissemination/exploitation
Criteria generally marked out of 5
Criterion threshold 3/5
Overall threshold 10/15
S/T QUALITY= “Scientific and/or technological excellence (relevant to the topics addressed by the call)”
Soundness of concept, and quality of objectives
• Progress beyond the state-of the-art
• Quality and effectiveness of the S/T methodology and associated work plan
IMPLEMENTATION “Quality and efficiency of the implementation and the management”
Appropriateness of the management structure and procedures
• Quality and relevant experience of the individual participants
• Quality of the consortium as a whole (including complementarity, balance)
• Appropriateness of the allocation and justification of the resources to be committed (budget, staff, equipment)
IMPACT=“Potential impact through the development, dissemination and use of project results”
Contribution, at the European [and/or international] level, to the expected impacts listed in the work programme under the relevant topic/activity
• Appropriateness of measures for the dissemination and/or exploitation of project results, and management of intellectual property.
The relevance of a proposal will be considered in relation to the topic(s) of the work programme open in a given call, and to the objectives of a call. These aspects will be integrated in the application of the criterion "S/T quality", and the first sub-criterion under "Impact" respectively.
When a proposal is partially relevant because it only marginally addresses the topic(s) of the call, or if only part of the proposal addresses the topic(s), this condition will be reflected in the scoring of the first criterion. Proposals that are clearly not relevant to a call ("out of scope") will be rejected on eligibility grounds.
SCORES= for each criterion the maximum is 5, so in total it’s 15
Poznan, 23rd January 2007

14. Evaluation - The experts (1)
The Commission draws on a wide pool of evaluators
about in FP6
Call for candidates published on December 2006
Call for applications are addressed to individuals/organisations
Applications via CORDIS
FP6 experts have been invited to transfer to FP7 (with a request to update their information)
Commission invites individuals on a call-by-call basis
Not self-selection!
Expertise, and experience are paramount
Geography, gender and “rotation” is also considered
Poznan, 23rd January 2007

15. Evaluation - The experts (2)
Experts agree to terms and conditions of an “appointment letter”
Typically an individual will review 6-8 proposals “remotely”…
…then spend a couple of days in Brussels
Some will participate in “hearings” with the consortia
Experts sign confidentiality and conflict of interest declaration
Names published after the evaluations
Poznan, 23rd January 2007

16. Selection procedure Poznan, 23rd January 2007 PROPOSAL Eligibility
Individual
evaluation
Consensus
Security
Scrutiny
(if needed)
Thresholds
Applicants informed of
results
of expert evaluation
Panel review
with hearings
(optional)
Ethical
Review
(if needed)
Commission ranking
Negotiation
Commission rejection
decision
Applicants informed of
Commission decision
Consultation of programme committee
(if required)
Commission funding and/or rejection
decision
Poznan, 23rd January 2007

17. 1. From Individual assessment to Consensus
May be “remote”
Proposal X
Copy 1
IAR*
Expert 1
CONSENSUS
REPORT
3 experts
Proposal X
Copy 2
IAR
Expert 2
Consensus meeting
Each proposal will first be assessed independently by at least three experts, chosen by the Commission from the pool of experts taking part in this evaluation (Of course, there may be more than three evaluators for each proposal).
At this first step the experts are acting individually; they do not discuss the proposal with each other, nor with any third party. The experts record their individual opinions in an Individual Assessment Report (IAR), giving scores and also comments against the evaluation criteria.
Experts will assess and mark the proposal exactly as it is described and presented. They do not make any assumptions or interpretations about the project in addition to what is in the proposal. Concise but explicit justifications will be given for each score. Recommendations for improvements to be discussed as part of a possible negotiation phase will be given, if needed. The experts will also indicate whether, in their view, the proposal deals with sensitive ethical issues, or if it requires further scrutiny with regard to security considerations.
Signature of the IAR also entails a declaration that the expert has no conflict of interest in evaluating the particular proposal.
Proposal X
Copy 3
IAR
Expert 3
*IAR= Individual assessment report
Poznan, 23rd January 2007

18. 2. Consensus
Built on the basis of the individual assessments of all the evaluators
Usually involves a discussion
Moderated by a Commission representative
One expert acts as a rapporteur
Agreement on consensus marks and comments for each of the criteria
Once all the experts to whom a proposal has been assigned have completed their IAR, the evaluation progresses to a consensus assessment, representing their common views.
This entails a consensus meeting to discuss the scores awarded and to prepare comments. The consensus discussion is moderated by a representative of the Commission. The role of the moderator is to seek to arrive at a consensus between the individual views of experts without any prejudice for or against particular proposals or the organisations involved, and to ensure a confidential, fair and equitable evaluation of each proposal according to the required evaluation criteria.
The moderator for the group may designate an expert to be responsible for drafting the consensus report ("rapporteur"). The experts attempt to agree on a consensus score for each of the criteria that have been evaluated and suitable comments to justify the scores. Comments should be suitable for feedback to the proposal coordinator. Scores and comments are set out in a consensus report. They also come to a common view on the questions of scope, ethics and security.
If during the consensus discussion it is found to be impossible to bring all the experts to a common point of view on any particular aspect of the proposal, the Commission may ask up to three additional experts to examine the proposal.
The outcome of the consensus step is the consensus report. This will be signed (either on paper, or electronically) by all experts, or as a minimum, by the rapporteur and the moderator. The moderator is responsible for ensuring that the consensus report reflects the consensus reached, expressed in scores and comments. In the case that it is impossible to reach a consensus, the report sets out the majority view of the experts but also records any dissenting views.
Poznan, 23rd January 2007

19. 3. Panel review Panel Meeting Compare consensus reports
Examines proposals with same consensus score (if needed)
Final marks and comments for each proposal
Suggestions on order of priority, clustering, amendments, etc.
Hearings with proposers may be convened
Questions to the invited proposal coordinators
Small number of proposal representatives
PANEL REVIEW= The main task of the panel is to examine and compare the consensus reports in a given area, to check on the consistency of the marks applied during the consensus discussions and, where necessary, propose a new set of consensus scores. The panel is chaired by a representative of the Commission.
The outcome of the panel meeting is a report recording, principally:
• An evaluation summary report (ESR) for each proposal, including, where relevant, a report
of any ethical issues raised and any security considerations;
• A list of proposals passing all thresholds, along with a final score for each proposal passing the thresholds and the panel recommendations for priority order.
• A list of evaluated proposals having failed one or more thresholds;
• A list of any proposals having been found ineligible during the evaluation by experts;
• A summary of any deliberations of the panel;
• A record of the hearings (where applicable)
HEARINGS= Applicants whose proposals have been favourably evaluated are sometimes invited to Brussels to answer any specific questions raised by the experts.
Poznan, 23rd January 2007

20. 4. Commission Follow-up
Evaluation summary reports (ESR) sent to applicants (“Initial information letter”)
Draw up final ranking lists
Information to the Programme Committee
Commission decisions on rejected proposals
Contract negotiation
Formal consultation of Programme Committee (when required)
Commission decisions on proposals selected for funding
Survey of evaluators
INITIAL INFORMATION LETTER=EVALUATION SUMMARY REPORT
Poznan, 23rd January 2007

21. Thank you for your attention
Thank you for your attention! Sabrina Bozzoli APRE
Poznan, 23rd January 2007