Dir S ERC/European Commission RTD, Directorate S The European Research Council The evaluation of the ERC-2007-StG Call IDEAS Programme Committee, January 31, 2008

European Research Council │ 2 Overview 1.Status of evaluation report 2.Key data – both stages 3.The eligibility process 4.Budgets by domains and panels 5.The panel meetings and the ranked lists per panel 6.The consolidation by the Panel Chairs 7.Recommendations on grant levels 8.Feedback to applicants 9.Status of redress

European Research Council │ 3 Status of evaluation report Will be conform to that of other programmes Offering same facilities to PMC members We did not succeed having a fully quality checked document for this meeting. However, key data tables are distributed Formal E.R. statistical reporting will be based on the sample of 201 proposals in the main list – by convention Informal reporting, including public dissemination, is based on a sample of the top 300 proposals This is a reasonable estimate of the number that may be funded

European Research Council │ 4 Key data- number of proposals by evaluation step ineligible 5 withdrawn 8235 rejected 5 not submitted to second stage 2 passed away, 4 ineligible 1 withdrawn Submitted stage 1 Evaluated stage 1 Selected stage 1 Submitted stage 2 Evaluated stage in main list 116 in reserve list 113 reserve: reject for no budget 117 rejected: below thresholds

European Research Council │ 5 Key data: Evaluation process Stage 2: 559 proposals expected Eligibility and withdrawals Reception of proposals INDIVIDUAL ASSESSMENTS Panel membersReferees INTERVIEWS 20 panel- ranked lists PANEL MEETINGS PANEL CHAIR MEETING Single Consolidated list 1 day to 3 days

European Research Council │ 6 The eligibility process Eligibility decisions have been taken conform the standard FP7 methodology - and in accordance with its ‘case-law’ Some of the decisions have been complex, and have required discussion in parallel with the review We attempt to minimise the number of such cases However, given the numbers and the time-constraint, partial parallel processing of review and eligibility is necessary – and is in accordance with the Rules

European Research Council │ 7 Budgets by domains and panels, based on adopted / published call budget, 289.5M€ 80%20% 8 PE panels7 LS panels5 SH Reserve Nominal panel budgets, proportional to sum of stage-1 requested grant E.g. PE E. g. PE4 Fits in nominal panel budget, Score > 9, no ordering … Main listReject Candidate for reserve budget, Ranked in priority order, with scoring convention Reserve ……… M€ Additional budget contributions add to reserve Additional budget Not fundable, Score < 8

European Research Council │ 8 The stage-2 interviews / panel meetings Panels operated independently, typically 3 days Interviews with applicants perceived as extremely useful complement to individual assessments Panels identified four groups of proposals: 1.Main list proposals, inside nominal panel budget 2.Reserve list, serious candidate, priority ordering 3.Reserve list, good proposal, clearly outside budget 4.Rejected proposal, failing threshold Panels tagged inter-disciplinary, cross-panel / domain proposals For groups 1 and 2, panels recommended the grant level

European Research Council │ 9 The panel chair consolidation meeting Purpose of the meeting: to establish a consolidated ranking of the serious candidate reserve (group 2) proposals 1.With special emphasis on inter-disciplinary proposals 2.Given 20 panels across all scientific fields, not a trivial affair …. 3.Because no absolute excellence standard exists: small score differences are meaningless across panels To faciliate, a starting point ranking was needed 1.Not constraining the freedom of the panel chairs 2.‘Bureaucratically fair’: purely based on an algorithm 3.Accounting for different panel and proposal sizes 4.Without prejudice to any possible differences of excellence between panels

European Research Council │ 10 The consolidation method (simplified) By convention, all panels scored their serious reserve candidates at 8.9, and sub-ranked them: first, second, …. Accumulated grant Normalised accumulated grant Normalise each panel on its nominal budget Res 1 Res 2 Res 3 Res 4 Res 5 Res 6 Res 7 Res 8 Res 9 ….. Tagged as inter- disciplinary 1.0 Nominal budget Ordering by normalised accumulated grant

European Research Council │ 11 The final ranking by the panel chairs Regarding inter-disciplinary proposals, the panel chairs: 1.Considered that they did not have the resources to re-examine these in detail 2.Confirmed their confidence in the prior work of the panels – ‘mainstreaming’ Regarding the ranked list, the panel chairs unanimously adopted the starting point proposed by the Commission as a fair result This ranking has not been modified, neither by Commission nor by the Scientific Council

European Research Council │ 12 The issue of recommended grant levels - I Panels’ remit included providing recommendations on award levels Individual panels faced some difficulties in this respect: 1.Proposals not overly detailed, over-estimates by PI or host … 2.Incomplete understanding / interpretation of the Rules of Participation 3.Contradictory information during interviews Driven by this, and by different ‘needs of the field’, panels arrived at different solutions: 1.No grant reductions; reductions across the board; big reductions on some proposals 2.Some panels have removed the PI salary for PI’s with permanent positions – problematic in view of Rules and of grant mobility 3.Some panels have calculated the grant ‘bottom-up’

European Research Council │ 13 The issue of recommended grant levels - 2 The ‘reductions issue’ was discussed in the Panel chair meeting 1.In general, panel chairs confirmed the positions taken by their individual panels as fair and reasonable in the context of the field 2.A small number of corrections was introduced during the meeting 3.On the specific issue of the PI’s salary, panel chairs realised the difficulty of the situation and recommended that the Commission applies appropriate corrections The Commission has applied a correction to all cases where the PI’s salary was explicitly removed The Scientific Council has strongly endorsed the position of the panel chairs, and requested the Commission to award the grants accordingly, without negotiation

European Research Council │ 14 Towards granting The Commission has maintained the possibility for successful applicants to seek redress against the level of awarded grant, using the redress procedure In reality, no such request for redress has been received The Commission has been cautious in its feedback to applicants 1.Pending formal decisions on 44.5M€ of third-country contributions 2.In view of the pending redress Granting has started on the main list – 201 proposals 1.About 40 ethical reviews are ongoing 2.The expectation is that about 300 grants will be awarded

European Research Council │ 15 Transparency towards applicants Four messages to the four groups of applicants: proposals in main list: granting is imminent, started for serious reserve candidates; all have score 8.9; probability of a grant varies from 1 to 0 down the ranking good proposals but clearly outside budget; score between 8.0 and 8.8; clear information given proposals fail threshold; score < 8.0; clear information given Subject to individual disclaimers, list of all 430 proposals above threshold is now published on ERC web-site 1.Most of the 116 serious reserve candidates can make a reasonable assessment of their probability

European Research Council │ 16 Status of redress The ‘Redress’ procedure has worked well For Stage-1, 245 requests for redress received, three main areas: 1.Eligibility – none sustained 2.Factual errors by reviewers – 15 sustained 3.Scientific judgement – none sustained 15 stage-1 proposals were re-reviewed by panels 1.One applicant has been invited to submit a stage-2 proposal Stage-2 redress: has just started 1.27 requests were received

European Research Council │ 17 Details of stage-1 redress Scientific judgement of panels150 The review process20 1.Wrong panel6 2.Conflict of interest14 Very near to threshold53 Negative, offensive25 1.Due to editing / English18 2.Discriminatory7 Eligibility49 Factual errors by reviewers26 Rebut reviewers / complain size of proposal6 NOTE: a single redress request may address multiple categories

European Research Council │ 18 Main lessons from stage-2 Overall, the process worked well: consolidation effective Interviews successful, to be maintained for StG Review the framework in which panels operate 1.Respect autonomy, specificities of scientific fields 2.But more need for coherent decision-making The ‘mainstreaming’ approach to inter-disciplinarity has worked, but reflection needed: 1.It does not give much visibility 2.Difficult to achieve inter-panel coherence Reviewers comments in feedback to applicants: 1.Reveals some flawed judgements – redress 2.Transparency inevitably drives improvement