1 17/3/2009 European Commission Directorate General Information Society & Media Briefing for Remote Reading How to fill in the (IER) Individual Evaluation Report

2 17/3/2009 This tutorial gives guidelines on how to evaluate the proposals, i.e. it describes how to assess a proposal; how to fill in the Individual Evaluation Form (IER); scoring a proposal.

3 17/3/2009 When running the slide show, you can either go slide by slide, or navigate following the hyperlinks: access page return to previous menu Return to Contents page Tutorial Manual

4 17/3/ Evaluators responsibilities 2.How to evaluate / how to fill in the IER 3.Scoring 4.The evaluation criteria 5.The evaluation forms (Call 4, Obj.1.1) Contents IPSTREP

5 17/3/2009 You have signed a non-disclosure agreement Before, during & after the evaluation do not communicate any information about the proposals You are independent express your personal view (you do not represent your employer, country, …) You are the evaluator do not delegate your responsibilities do not discuss the proposals with 3 rd parties Do not contact the proposers for clarifications proposals are evaluated on the information that is presented there, any lack of information will downgrade the score Declare any potential conflict of interest in case you suspect you have a conflict, contact us immediately (see e.g. Call-4 Evaluation Handbook) Evaluators Responsibilities

6 17/3/2009 Give a fair and clear opinion on each proposal Give your personal views Evaluate proposals against the specific Objectives and Impact as defined in the Workprogramme Evaluate the proposal as written Consistently apply the same standard of judgment to each proposal Respect the code of conduct How to evaluate - Introduction

7 17/3/2009 Use the official evaluation criteria only Proposals are evaluated on three criteria Scientific and technical quality Implementation Impact Assess the proposals in terms of all 3 evaluation criteria. Each criterion is more fully defined by descriptive bullet points, adapted to the instrument type. These are shown on the evaluation forms. Provide a relevant comment on each of the bullet points These sub-criteria are not scored individually or separately. The Evaluation Criteria

8 17/3/2009 How to fill in IER - General Comments are confined only to the criterion concerned. Comments describe only the final view of the proposal. Start with drafting the comments, then adjust the score accordingly. Organise and structure your comment: Overall assessment Followed by two or three illustrations (especially if the score is very low or very high) Followed by any comments which mitigate/reduce the overall appreciation For negative arguments, a distinction should be made between severe shortcomings and recommendations for improvements.

9 17/3/2009 Comments are of adequate length (not just one sentence !) Do not be afraid to provide detailed comments. It is valuable to have sufficient information to be able to understand how you reached your conclusions. A detailed comment is helpful for the preparation of the Consensus Report and the discussion that follows. Poor comments merely echo the score – Good comments explain it: This proposal does not adequately advance the state of the art. This proposal fails to advance the state of the art in X or Y, it does not take Z into account. IER- Level of detail

10 17/3/2009 Comments are clear and provide clear justifications. Quote proposal text if useful. Poor comments are vague - Good ones are precise and final : We think the consortium management plan is probably inadequate given the duration of the project and the number of partners. The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners. Poor comments are ambiguous – Good comments are clear: The resources for the project are unrealistic. The resources in Workpackages 4 and 6 are seriously underestimated given the complexity of the activity involved. IER – Clear Messages

11 17/3/2009 Comments are substantial. Comments are facts not opinions not I think that.... but This proposal is... Do not write generic comments such as the proposal is not very innovative without specifying where exactly it fails. Poor comments include words like: Perhaps, think, seems, assume, probably, … Good comments include words like: Because, percent, specifically, for example, … IER – Factual Evidence

12 17/3/2009 IER – Avoiding Conflicts Poor comments provide an opening for a complaint - Good comments close the question: There is no discussion of dissemination activities. Dissemination activities are not adequately discussed. There are only two SMEs in the consortium. The consortium lacks a sufficient SME participation. The proposal coordinator is not adequately experienced. The proposal coordinator does not demonstrate in the proposal an adequate level of experience of work in this field.

13 17/3/2009 IER – Varying the Vocabulary Why say Poor when you can say: Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile… Why say Excellent when you can say: Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

14 17/3/2009 IER – Final Checks Have you fully explained the proposals strengths and weaknesses on all criteria ? Do scores match comments (high scores = positive comments, low scores = negative comments) ? Have you highlighted any points needing special attention ? Have you double-checked any matters-of-fact which you have quoted ? Have you written at adequate length ? Overall comment = any comment not covered already in individual criterion. If this was my proposal, would I find this report fair, accurate clear and complete?

15 17/3/2009 Priority should be given to the comments as they will justify the outcome of the evaluation. Scores should be adjusted to reflect the opinion expressed in the comments. The scores are only a mathematical tool allowing the comparison of the relative quality of different proposals. A threshold applies to each individual evaluation criterion and there is an overall threshold. A proposal fails if it is below at least one of the thresholds (incl. overall threshold). The overall criterion is not scored separately but automatically calculated from the individual ones. Scoring

16 17/3/2009 The Scoring Scale Use the full scale! Half marks may be given. 0 The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information. 1 Poor – The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses. 2 Fair – While the proposal broadly addresses the criterion, there are significant weaknesses. (can not be negotiations) 3 Good – The proposal addresses the criterion well, although improvements would be necessary. 4 Very good – The proposal addresses the criterion very well, although certain improvements are still possible. 5 Excellent – The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.