Smart@Fire Evaluation Briefing How to fill in the Evaluation Scorecard

Agenda Intro Smart@Fire The challenge What has been done What’s coming up How to assess the proposals Scoring the proposals Filling in and returning the proposal score card

Confidentiality But first, remember your responsibilities You have signed a non-disclosure agreement Before, during and after this evaluation you do not disclose any information about the proposals which you have seen Keep any printed copies secure when not in use; bring them with you to Brussels You are the evaluator Do not pass this responsibility to anybody else Do not discuss the proposals with anybody else

Confidentiality Your responsibilities Do not contact the proposers Evaluate only the proposal as it was submitted to us on October 13 2014, without any later additions or clarifications from the proposer The identities of evaluating experts are never revealed to the proposers concerned Declare any potential conflict of interest If there is a situation which might prevent you evaluating a proposal impartially, mention in the general feedback on the evaluation sheet: “I cannot evaluate this proposal”

Introduction What has been done? Preliminary stage: Needs assessment Defining end-user requirements State-of-the-art First stage: Innovation Platform as market consultation instrument

Introduction The challenges To reduce the risks associated with firefighting, innovative ICT-solutions need to be developed and integrated in the smart Personal Protective System. The current ICT-solutions available on the market do not yet provide full satisfaction. Based on a large scale needs assessment of 961 fire brigades, interactive work sessions with end-users and work sessions with suppliers, the innovation potential from procurer’s and supplier’s side has been identified. To increase the safety of the fire fighters we are looking for innovative technologies, more in particular: PPS central nerve system: system architecture, communication, localization & interfaces Sweat absorbing multilayer underwear IT thermal hotspot detector HMD/HUD firefighter visualization sytem “BE SEEN” omnidirectional active illumination In consensus with the end-users and the European Commission, the priority challenge for the PCP tender scope covers primarily CHALLENGE 1, PPS central nerve system

Introduction PPS central nerve system The overall central nerve system architecture Communication network (indoor penetration, data pre-processing, update rate,..) Right architecture (distributed vs central processing, modularity, scalability) Limited integration with textile Limited integrative measures (pockets, velcro,..) Cabled and/or wireless integration Electromagnetic shielding Localization Engine A hybrid localization system (GPS/Inertial) Relative track&trace map Beacon-based solution for localization Intuitive user feedback For the intervention coordinating officer: Intuitive user interface dashboard For the fire fighter: Multimodal combination of audio, simple UI and Haptic belt Coupling via defined application interfaces (environmental temperature, explosive gas detector, physiological monitoring, …)

Introduction Design constraints Price of PPS Standardization / Guarnateeing safety High level functional requirements Configuration of the PPS system Autonomy Weight Speed of deployment Refresh rate of data transfer Accuracy of the localization system Standard interfaces between wirelessly connected devices on the firefighter Fraud proof Lyfecycle With minimal additional needed infrastructure Robustness under washing and exposure to specific substances

Introduction What is coming up? Second stage: Joint Pre-Commercial Procurement: development of innovative Smart@Fire Personal Protective Systems Perform solution exploration and design: during this stage of 4 months, a number of selected suppliers (or consortia of suppliers) further elaborate the detailed design of their proposed solution (or set of solutions). Develop joint-workable prototype(s): during this stage of typically 8 months, the chosen suppliers with the best solution design (as assessed by an evaluation committee) develop their own prototypes in parallel. Produce and test initial batch of finalized PPS prototypes: during this stage of typically 6 months, at least 2 remaining suppliers remain to ensure a future competitive market. Their final productized solution batch of 10 items is validated through field tests. Third stage: Final Joint Procurement of Smart PPS

Evaluation The evaluation package For the remote stage of this evaluation we are providing you with a comprehensive evaluation package containing: The tender document, the challenge brief and all tender proposals in PDF form + possible attachments An evaluation excel sheet existing out of the following tabs Overall timing indication: Gives an overview of the entire project from solution description until first batch production Process Evaluation round 1: Explains the first evaluation process and timing in detail Overview tender applicants: Provides a view on the final scoring per venture # Individual tender scoring tabs: These tabs will enable you to score the individual proposals

Evaluation The evaluation Criteria Proposals are evaluated on three criteria only Impact Quality Price Assess the proposal in terms of the first two criteria Each criterion is more fully defined by descriptive sub questions. These are shown on the evaluation sheet Provide a scoring and comment for each of the sub questions Overall feedback can be added on the bottom of each tender score sheet

Evaluation The scoring scale First develop your comments on each criterion ……then select scores in line with the scoring model provided with each question Only tenders with the following minimum scores are eligible for consideration for a contract: 60% of the maximum number of points for each of the categories: Impact (54 points from a maximum 90 available for Phase 1). Quality of the tender (66 points from a maximum 110 available for Phase 1). 60% of the maximum number of points for the combined Impact, Quality and Finance scores (138 points from a maximum 230 available for Phase 1) Failure to achieve the minimum score at any of the stages will result in the tender being excluded from further participation in the PCP.

Evaluation How to fill in your scorecard? Your comments Comments are confined only to the question concerned Comments describe only your final view of the proposal Comments are clear and unambiguous. Try to avoid obscure acronyms and technical terms Comments are of adequate length and are provided for all questions separately Comments provide full justification for the score given

Evaluation Scoring – Be factual Comments are substantial; do not write generic criticisms; be specific, explain Comments are facts not opinions, don’t show doubt or indecision not “I don’t understand why....” but “The proposers do not make clear why.. .” Poor comments include words like: “Perhaps, think, seems, assume, probably, …” Good comments include words like: “Because, percent, specifically, for example, …”

Evaluation Scoring – Give Clear Messages Poor comments are vague – Good ones are precise “I think the implementation plan is probably inadequate.” “The implementation plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.” “The resources for the project seem unrealistic.” “The resources in phase 2 and 3 are seriously underestimated given the complexity of the activities involved.”

Evaluation Scoring – Varying the Vocabulary Why say “Poor” when you can say: Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile… Why say “Excellent” when you can say: Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

Evaluation Scoring – Final Check Have you fully explained the proposal’s strengths and weaknesses on each of the criteria ? Do your scores match your comments (high scores = positive comments, low scores = negative comments)? Have you double-checked any matters-of-fact which you have quoted? Have you written at adequate length? If this was your proposal, would you find this report fair, accurate, clear and complete?

Evaluation Submitting your scores First, read over all your proposals to get an idea of the general standard and content When you have completed the evaluation of each proposal, submit your score for that proposal in the excel score sheet Once you’ve reviewed and scored all proposals, deliver your filled in excel score card before October 27, end of business day. Don’t worry if, after submitting your score card, you would like to add to or modify your comments. You will anyway get a chance to discuss your opinions with the other experts in the meetings in Brussels

Evaluation Finally Give a fair and clear opinion on each proposal. You are: Independent : you represent yourself, not your employer, not your country…..) Objective : you evaluate the proposal as written Accurate : you use the official evaluation criteria only Consistent : you apply the same standard of judgment to each proposal Incommunicado : you do not disclose to anybody the contents of the proposals which you see

- The Smart@Fire Project Team Finally Thank you very much for your help! - The Smart@Fire Project Team