Evaluator’s view Borka Jerman-Blažič University of Ljubljana and Jožef Stefan Institute SLOVENIA

Acting as an evaluator Evaluation process is very demanding procedure which involves: Evaluation process is very demanding procedure which involves: commission officers and staff and commission officers and staff and number of evaluators number of evaluators

Acting as an evaluator Evaluators are usually people with expertise in the subject addressed by the call and are coming from different environments: Evaluators are usually people with expertise in the subject addressed by the call and are coming from different environments: Academia, industry, professional societies, research institutes, national institutions dealing with research etc Academia, industry, professional societies, research institutes, national institutions dealing with research etc

Invitation The selection of evaluators is based on the data in CORDIS data base The selection of evaluators is based on the data in CORDIS data base If you want to become an evaluator you must register in the respective data base of evaluators for particular program If you want to become an evaluator you must register in the respective data base of evaluators for particular program Sometimes, the selection process for evaluators is supported through additional channels Sometimes, the selection process for evaluators is supported through additional channels

Invitation Selection: you are selected according to your expertise and data records Selection: you are selected according to your expertise and data records You are selected because you have entered information about you in the CORDIS data base and because your expertise is in line with the scientific/technical objectives of the relevant call You are selected because you have entered information about you in the CORDIS data base and because your expertise is in line with the scientific/technical objectives of the relevant call So, be careful when entering data about your expertise, level of education and references So, be careful when entering data about your expertise, level of education and references Every call has different objectives and as a consequence the selection of the evaluators follows their expertise and in each call is different Every call has different objectives and as a consequence the selection of the evaluators follows their expertise and in each call is different After the selection an invitation is launched usually via After the selection an invitation is launched usually via

The evaluator appointment start with: An invitation An invitation Your agreement Your agreement Your correct response Your correct response Your on time provision of the requested data by the Commission Your on time provision of the requested data by the Commission Final selection/invitation comes later when all proposals are known, the call is closed and the number of needed evaluators is finally decided Final selection/invitation comes later when all proposals are known, the call is closed and the number of needed evaluators is finally decided The number of evaluators depends on the number of received proposals The number of evaluators depends on the number of received proposals

The information requested by the evaluator Availability Availability Legal entity form (each time you are asked to act as an evaluator) Legal entity form (each time you are asked to act as an evaluator) Bank information (if the procedures do not change one signed document by the bank and yourself is sufficient for at least one year) Bank information (if the procedures do not change one signed document by the bank and yourself is sufficient for at least one year) Copy of your ID document Copy of your ID document

Information requested You have to agree to the conditions of work as stated in the Commission documents sent to you You have to agree to the conditions of work as stated in the Commission documents sent to you Confidentiality agreement/commitment of non- disclosure and Confidentiality agreement/commitment of non- disclosure and Conflict of interest declaration Conflict of interest declaration Reimbursement of your personal cost and fee Reimbursement of your personal cost and fee

There are two options for an evaluator You can act as independent expert or You can act as independent expert or as employee of an organisation (if you are employed) as employee of an organisation (if you are employed) The conditions regarding payment are slightly different but the fee per day is same The conditions regarding payment are slightly different but the fee per day is same There are two forms of reimbursement: the fee per working day and reimbursement of travel and accommodation cost, both are specified in the accompanying documents There are two forms of reimbursement: the fee per working day and reimbursement of travel and accommodation cost, both are specified in the accompanying documents

The selection process After the decision is taken for your engagement as an evaluator the appointment procedure follows: After the decision is taken for your engagement as an evaluator the appointment procedure follows: you receive appointment letter where all conditions of work are specified as well your duties and commitments (confidentiality/non disclosure and conflict of interest declaration) you receive appointment letter where all conditions of work are specified as well your duties and commitments (confidentiality/non disclosure and conflict of interest declaration)

Appointment The appointment letter is a sort of contract with Annexes The appointment letter is a sort of contract with Annexes You must sign the letter and the enclosed documents and send them back to the Commission officer You must sign the letter and the enclosed documents and send them back to the Commission officer Instructions for your work follow as well Instructions for your work follow as well

On line and on site evaluation Travel and accommodation is organized by yourself Travel and accommodation is organized by yourself The place of evaluation is known (for FP6 this the building on Square Frere Orban) The place of evaluation is known (for FP6 this the building on Square Frere Orban) The date of evaluation is known as well The date of evaluation is known as well Some programs (e.g. Marie Curie) practice individual evaluation on distance at your site Some programs (e.g. Marie Curie) practice individual evaluation on distance at your site You receive evaluation guidelines for specific call that includes guidelines for your work, evaluation forms and guidelines how to fill them You receive evaluation guidelines for specific call that includes guidelines for your work, evaluation forms and guidelines how to fill them

Commitments and duties If you are evaluating on distance then you are provided with the proposal on line via secured access to the network If you are evaluating on distance then you are provided with the proposal on line via secured access to the network You have restricted time to do the evaluation (1-2 weeks) but you are more free in planning your time You have restricted time to do the evaluation (1-2 weeks) but you are more free in planning your time You may influence on the selection of proposals (by reading the abstracts) to be evaluated but this is not always possible as other evaluators are also pointing to the same proposals You may influence on the selection of proposals (by reading the abstracts) to be evaluated but this is not always possible as other evaluators are also pointing to the same proposals All your evaluation data are entered as well on line via secure route All your evaluation data are entered as well on line via secure route Consensus is achieved at site – in Brussels Consensus is achieved at site – in Brussels

Evaluation in FP6 All proposal are receipt, opened, acknowledged and their content entered into a data base by the Commission officials to support the evaluation process All proposal are receipt, opened, acknowledged and their content entered into a data base by the Commission officials to support the evaluation process Basic eligibility criteria for each proposal are also checked by the Commission staff before evaluation begins Basic eligibility criteria for each proposal are also checked by the Commission staff before evaluation begins Commission officials assign particular number of proposals to the panel of experts and to particular evaluator Commission officials assign particular number of proposals to the panel of experts and to particular evaluator Before you start to evaluate a briefing meeting is organized for all evaluators involved Before you start to evaluate a briefing meeting is organized for all evaluators involved

Evaluation in FP6: your role Commission staff do not influence the opinion of an independent expert, if asked they provide to you only additional information or assistance Commission staff do not influence the opinion of an independent expert, if asked they provide to you only additional information or assistance Each independent expert is assigned cca 5 to 6 proposals if they are IP or NoE, in case of STREPs that may be slightly more Each independent expert is assigned cca 5 to 6 proposals if they are IP or NoE, in case of STREPs that may be slightly more Your working time is from 9 to 17 but usually the evaluators work much more as the evaluation is very demanding process Your working time is from 9 to 17 but usually the evaluators work much more as the evaluation is very demanding process

The evaluation process in FP6 The working conditions are good and the staff is supporting and kind The working conditions are good and the staff is supporting and kind You are supposed NOT to talk to other experts about the proposals you are evaluating You are supposed NOT to talk to other experts about the proposals you are evaluating The place to do so are the consensus meetings where all appointed evaluators for particular proposal discuss for common evaluation report The place to do so are the consensus meetings where all appointed evaluators for particular proposal discuss for common evaluation report

The evaluation process in FP6 Your task is to read carefully the proposal document and to prepare an IAR – individual assessment report Your task is to read carefully the proposal document and to prepare an IAR – individual assessment report The report has several fields where you put your scores and justification The report has several fields where you put your scores and justification

Confidentiality/conflict of interest It is forbidden the evaluator to work closely with the proposer or to be involved in the preparation, the evaluator should not attend the panel meeting where such proposal is discussed It is forbidden the evaluator to work closely with the proposer or to be involved in the preparation, the evaluator should not attend the panel meeting where such proposal is discussed In case of indirect conflict of interest (proposal is coming from the institution the evaluator is employed) the evaluator should not be involved in evaluation of such proposal and have to indicate such case In case of indirect conflict of interest (proposal is coming from the institution the evaluator is employed) the evaluator should not be involved in evaluation of such proposal and have to indicate such case

Confidentiality/conflict of interest All data, documents, writings, comments etc must stay in the place of evaluation. All data, documents, writings, comments etc must stay in the place of evaluation. Under no circumstances may an evaluator attempt to contact proposal submitter on his own account, either during the evaluation or afterwards. Under no circumstances may an evaluator attempt to contact proposal submitter on his own account, either during the evaluation or afterwards. It is also NOT recommended to visit WEB references even if they are mentioned in the proposal especially if the information on the WEB are placed after the date of proposal submission It is also NOT recommended to visit WEB references even if they are mentioned in the proposal especially if the information on the WEB are placed after the date of proposal submission

Evaluation process: your role Assess and mark the proposal exactly as it is described and presented, do not make any assumptions or interpretations about the project in addition to what the proposers have written in their proposal Assess and mark the proposal exactly as it is described and presented, do not make any assumptions or interpretations about the project in addition to what the proposers have written in their proposal Keep to the evaluation criteria as stated in the evaluation forms Keep to the evaluation criteria as stated in the evaluation forms Give marks and comments and write them in readable shape Give marks and comments and write them in readable shape You should care the comments to be written in line with the the marks You should care the comments to be written in line with the the marks Maintain consistency in your scoring through the work Maintain consistency in your scoring through the work

Provide a brief but explicit justification of your marks, be honest, but correct. The language especially in case of low scores must be correct as the comments will be sent to the proposers, you may use short extracts from the proposal text to justify your opinion. Provide a brief but explicit justification of your marks, be honest, but correct. The language especially in case of low scores must be correct as the comments will be sent to the proposers, you may use short extracts from the proposal text to justify your opinion. Where justified you may give recommendations, but ensure that the scores given reflect the proposal as presented by the applicant Where justified you may give recommendations, but ensure that the scores given reflect the proposal as presented by the applicant It is recommendable before the start of the evaluation you to examine a number of proposals before signing off their first individual assessment forms. This will help to calibrate their scoring It is recommendable before the start of the evaluation you to examine a number of proposals before signing off their first individual assessment forms. This will help to calibrate their scoring

Evaluating: the good and the bad Good proposals usually have clearly written text, not excessive wording, but text that provide good information for the evaluator to understand the objectives and the presented work Good proposals usually have clearly written text, not excessive wording, but text that provide good information for the evaluator to understand the objectives and the presented work The objectives must be well set up and justified, very clearly The objectives must be well set up and justified, very clearly The methods to achieve the objectives should be sound, from technical and from management point of view The methods to achieve the objectives should be sound, from technical and from management point of view There must be a red line along the proposal from the beginning to the end There must be a red line along the proposal from the beginning to the end The proposal should look as compact package that is oriented to achieve common joint goals The proposal should look as compact package that is oriented to achieve common joint goals

Evaluating: the good and the bad The idea of the proposal (what exactly is planned to be done) described must innovative. The innovationmay have different aspects The idea of the proposal (what exactly is planned to be done) described must innovative. The innovationmay have different aspects Evaluators are not always very deeply involved in particular specific field, so good proposal contain short but convincing state of the art of the field or subject background that justify the need for specified and identified problem to be solved Evaluators are not always very deeply involved in particular specific field, so good proposal contain short but convincing state of the art of the field or subject background that justify the need for specified and identified problem to be solved Good proposal contain text or description that justify the project funding from public money: the proposer must present enough evidence why the proposed way of research or proposed research topic is carried out on European level and why deserve public funding Good proposal contain text or description that justify the project funding from public money: the proposer must present enough evidence why the proposed way of research or proposed research topic is carried out on European level and why deserve public funding

Evaluating: the good and the bad Each criteria from the IAR is important, so the proposal must show more than average merit in all subjects Each criteria from the IAR is important, so the proposal must show more than average merit in all subjects This implies that the proposal must be well thaught and well prepared in all aspects This implies that the proposal must be well thaught and well prepared in all aspects Scientific and technical value are most important, meaning that innovation must be present. Beside that, the constitution of the conzortium must be balanced, the proposal must show good balance of all relevant institutions, from industry, academia, SMEs, professional societies ect. Scientific and technical value are most important, meaning that innovation must be present. Beside that, the constitution of the conzortium must be balanced, the proposal must show good balance of all relevant institutions, from industry, academia, SMEs, professional societies ect. Distribution of the resources is also very important. The proposal must convince the evaluator that the allocation of the work and the resources are well prepared and that they give or show enough guarantee that the conzortium is capable to carry out the project up to the accomplishment of the goals Distribution of the resources is also very important. The proposal must convince the evaluator that the allocation of the work and the resources are well prepared and that they give or show enough guarantee that the conzortium is capable to carry out the project up to the accomplishment of the goals

Evaluating: the good and the bad All criteria from the IAR contribute to the overall ranking of the proposal, in that context All criteria from the IAR contribute to the overall ranking of the proposal, in that context management part must be also prepared in line with the type of the proposed work management part must be also prepared in line with the type of the proposed work Information about involved institution and the major staff must be provided as well. This information contribute to the trust building among the evaluators that there is enough capacity, knowledge and experience for successful accomplishment of the project objectives Information about involved institution and the major staff must be provided as well. This information contribute to the trust building among the evaluators that there is enough capacity, knowledge and experience for successful accomplishment of the project objectives The proposer must convince you during text reading that his idea is innovative, sound and the methods proposed for work are the most appropriate for the presented problem to be solved or idea implemented The proposer must convince you during text reading that his idea is innovative, sound and the methods proposed for work are the most appropriate for the presented problem to be solved or idea implemented This implies enough information about the conzortium expertise, skills, knowledge, previous engagement, published references or similar data. The project usually benefits from CVs of involved people and from good description of the institutions consisting the conzortium This implies enough information about the conzortium expertise, skills, knowledge, previous engagement, published references or similar data. The project usually benefits from CVs of involved people and from good description of the institutions consisting the conzortium

Evaluation in FP6: Consensus building One among the evaluators is appointed as a raporteur One among the evaluators is appointed as a raporteur NoE and IP have 5 evaluators + a raporteur NoE and IP have 5 evaluators + a raporteur STREPs have 3 evaluators STREPs have 3 evaluators The task of the raporteur is to accept and record the majority view of the other evaluators, together with the Commission officer may bring additional evaluator(s) (up to 3) in case no consensus is achieved The task of the raporteur is to accept and record the majority view of the other evaluators, together with the Commission officer may bring additional evaluator(s) (up to 3) in case no consensus is achieved The face to face consensus meeting is usually convened by the officer The face to face consensus meeting is usually convened by the officer

Evaluation FP6:consensus meeting The evaluators are usually presenting their scores and the justifications then they discuss in order to achieve common scores and comments The evaluators are usually presenting their scores and the justifications then they discuss in order to achieve common scores and comments If this is achieved then the raporteur write the report and all evaluators must agree with it and sign it, the raporteur is responsible for collecting the signatures If this is achieved then the raporteur write the report and all evaluators must agree with it and sign it, the raporteur is responsible for collecting the signatures Then the consensus report goes to the panel Then the consensus report goes to the panel The panel consist of all experts from particular area of the call e.g. BBAll, Security etc The panel consist of all experts from particular area of the call e.g. BBAll, Security etc

Evaluation in FP6: Consensus meeting If consensus has not been reached, the report sets out the majority view of the evaluators, but also records any dissenting views. In case any such disagreement has not been resolved a threshold score or average score given by the evaluators will be awarded for the proposal. If consensus has not been reached, the report sets out the majority view of the evaluators, but also records any dissenting views. In case any such disagreement has not been resolved a threshold score or average score given by the evaluators will be awarded for the proposal. All consensus reports are entered in the data base and joint list of proposals ranked according the overall scores is generated All consensus reports are entered in the data base and joint list of proposals ranked according the overall scores is generated This list is discussed by the panel This list is discussed by the panel The panel sometimes may agree to do some changes of the overall listing if this is explained and justified e.g. for some proposals covering missing area in the call etc. The panel sometimes may agree to do some changes of the overall listing if this is explained and justified e.g. for some proposals covering missing area in the call etc.

Hearings and your role The best ranked NoEs and IPs by the panel are called after few weeks to hearings The best ranked NoEs and IPs by the panel are called after few weeks to hearings Most of the experts panels are present at the hearings. Experts that have evaluated proposals presenting on the panel must be present at the hearings Most of the experts panels are present at the hearings. Experts that have evaluated proposals presenting on the panel must be present at the hearings These meetings are convened by the Commission official These meetings are convened by the Commission official Evaluators after the presentation of the proposal that include mainly answers on the questions set up by the evaluators can set additional questions Evaluators after the presentation of the proposal that include mainly answers on the questions set up by the evaluators can set additional questions Questions are prepared in written form and passed to the convenor Questions are prepared in written form and passed to the convenor The presenters have no knowledge who was asking particular questions The presenters have no knowledge who was asking particular questions Hearings are very helpful meetings as they clarify many aspects that are not always evident from the written text Hearings are very helpful meetings as they clarify many aspects that are not always evident from the written text

Decision making Outcomes of the hearings may somehow change the ordering of the highly ranked proposal on the list Outcomes of the hearings may somehow change the ordering of the highly ranked proposal on the list After the hearings, the panel convene and decide about final ranking After the hearings, the panel convene and decide about final ranking With this your task as evaluator is almost finished With this your task as evaluator is almost finished The further actions are connected with the reimbursement forms and the final payment for your services The further actions are connected with the reimbursement forms and the final payment for your services

The panel It is important the evaluators to have a look on the monitoring statistics before they discuss the proposals at the panel meeting It is important the evaluators to have a look on the monitoring statistics before they discuss the proposals at the panel meeting This will help to take in account the evaluators marking profile and that the other experts before revising the scores and the ranking This will help to take in account the evaluators marking profile and that the other experts before revising the scores and the ranking The evaluators may agree on some recommendations regarding the negotiation process The evaluators may agree on some recommendations regarding the negotiation process

CONCLUSION If you are part of the new ERA or if you preparing yourself to take part then being an evaluator is exciting experience If you are part of the new ERA or if you preparing yourself to take part then being an evaluator is exciting experience You meet with your colleagues and you contribute to the success of the FP6 You meet with your colleagues and you contribute to the success of the FP6 It is overall an exciting experience It is overall an exciting experience