2. Smart@Fire Evaluation Briefing
How to fill in the Evaluation Scorecard

3. Agenda Intro Smart@Fire The challenge What has been done
What’s coming up
How to assess the proposals
Scoring the proposals
Filling in and returning the proposal score card

4. Confidentiality But first, remember your responsibilities
You have signed a non-disclosure agreement
Before, during and after this evaluation you do not disclose any information about the proposals which you have seen
Keep any printed copies secure when not in use; bring them with you to Brussels
You are the evaluator
Do not pass this responsibility to anybody else
Do not discuss the proposals with anybody else

5. Confidentiality Your responsibilities Do not contact the proposers
Evaluate only the proposal as it was submitted to us on October , without any later additions or clarifications from the proposer
The identities of evaluating experts are never revealed to the proposers concerned
Declare any potential conflict of interest
If there is a situation which might prevent you evaluating a proposal impartially, mention in the general feedback on the evaluation sheet: “I cannot evaluate this proposal”

6. Introduction What has been done? Preliminary stage: Needs assessment
Defining end-user requirements
State-of-the-art
First stage:
Innovation Platform as market consultation instrument

7. Introduction The challenges
To reduce the risks associated with firefighting, innovative ICT-solutions need to be developed and integrated in the smart Personal Protective System. The current ICT-solutions available on the market do not yet provide full satisfaction.
Based on a large scale needs assessment of 961 fire brigades, interactive work sessions with end-users and work sessions with suppliers, the innovation potential from procurer’s and supplier’s side has been identified. To increase the safety of the fire fighters we are looking for innovative technologies, more in particular:
PPS central nerve system: system architecture, communication, localization & interfaces
Sweat absorbing multilayer underwear
IT thermal hotspot detector
HMD/HUD firefighter visualization sytem
“BE SEEN” omnidirectional active illumination
In consensus with the end-users and the European Commission, the priority challenge for the PCP tender scope covers primarily CHALLENGE 1, PPS central nerve system

8. Introduction PPS central nerve system
The overall central nerve system architecture
Communication network (indoor penetration, data pre-processing, update rate,..)
Right architecture (distributed vs central processing, modularity, scalability)
Limited integration with textile
Limited integrative measures (pockets, velcro,..)
Cabled and/or wireless integration
Electromagnetic shielding
Localization Engine
A hybrid localization system (GPS/Inertial)
Relative track&trace map
Beacon-based solution for localization
Intuitive user feedback
For the intervention coordinating officer: Intuitive user interface dashboard
For the fire fighter: Multimodal combination of audio, simple UI and Haptic belt
Coupling via defined application interfaces (environmental temperature, explosive gas detector, physiological monitoring, …)

9. Introduction Design constraints Price of PPS
Standardization / Guarnateeing safety
High level functional requirements
Configuration of the PPS system
Autonomy
Weight
Speed of deployment
Refresh rate of data transfer
Accuracy of the localization system
Standard interfaces between wirelessly connected devices on the firefighter
Fraud proof
Lyfecycle
With minimal additional needed infrastructure
Robustness under washing and exposure to specific substances

10. Introduction What is coming up?
Second stage: Joint Pre-Commercial Procurement: development of innovative Personal Protective Systems
Perform solution exploration and design: during this stage of 4 months, a number of selected suppliers (or consortia of suppliers) further elaborate the detailed design of their proposed solution (or set of solutions).
Develop joint-workable prototype(s): during this stage of typically 8 months, the chosen suppliers with the best solution design (as assessed by an evaluation committee) develop their own prototypes in parallel.
Produce and test initial batch of finalized PPS prototypes: during this stage of typically 6 months, at least 2 remaining suppliers remain to ensure a future competitive market. Their final productized solution batch of 10 items is validated through field tests.
Third stage: Final Joint Procurement of Smart PPS

11. Evaluation The evaluation package
For the remote stage of this evaluation we are providing you with a comprehensive evaluation package containing:
The tender document, the challenge brief and all tender proposals in PDF form + possible attachments
An evaluation excel sheet existing out of the following tabs
Overall timing indication: Gives an overview of the entire project from solution description until first batch production
Process Evaluation round 1: Explains the first evaluation process and timing in detail
Overview tender applicants: Provides a view on the final scoring per venture
# Individual tender scoring tabs: These tabs will enable you to score the individual proposals

12. Evaluation The evaluation Criteria
Proposals are evaluated on three criteria only
Impact
Quality
Price
Assess the proposal in terms of the first two criteria
Each criterion is more fully defined by descriptive sub questions. These are shown on the evaluation sheet
Provide a scoring and comment for each of the sub questions
Overall feedback can be added on the bottom of each tender score sheet

13. Evaluation The scoring scale
First develop your comments on each criterion
……then select scores in line with the scoring model provided with each question
Only tenders with the following minimum scores are eligible for consideration for a contract:
60% of the maximum number of points for each of the categories:
Impact (54 points from a maximum 90 available for Phase 1).
Quality of the tender (66 points from a maximum 110 available for Phase 1).
60% of the maximum number of points for the combined Impact, Quality and Finance scores (138 points from a maximum 230 available for Phase 1)
Failure to achieve the minimum score at any of the stages will result in the tender being excluded from further participation in the PCP.

14. Evaluation How to fill in your scorecard? Your comments
Comments are confined only to the question concerned
Comments describe only your final view of the proposal
Comments are clear and unambiguous. Try to avoid obscure acronyms and technical terms
Comments are of adequate length and are provided for all questions separately
Comments provide full justification for the score given

15. Evaluation Scoring – Be factual
Comments are substantial; do not write generic criticisms; be specific, explain
Comments are facts not opinions, don’t show doubt or indecision
not “I don’t understand why....”
but “The proposers do not make clear why.. .”
Poor comments include words like:
“Perhaps, think, seems, assume, probably, …”
Good comments include words like:
“Because, percent, specifically, for example, …”

16. Evaluation Scoring – Give Clear Messages
Poor comments are vague – Good ones are precise
“I think the implementation plan is probably inadequate.”
“The implementation plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.”
“The resources for the project seem unrealistic.”
“The resources in phase 2 and 3 are seriously underestimated given the complexity of the activities involved.”

17. Evaluation Scoring – Varying the Vocabulary
Why say “Poor” when you can say:
Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…
Why say “Excellent” when you can say:
Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

18. Evaluation Scoring – Final Check
Have you fully explained the proposal’s strengths and weaknesses on each of the criteria ?
Do your scores match your comments (high scores = positive comments, low scores = negative comments)?
Have you double-checked any matters-of-fact which you have quoted?
Have you written at adequate length?
If this was your proposal, would you find this report fair, accurate, clear and complete?

19. Evaluation Submitting your scores
First, read over all your proposals to get an idea of the general standard and content
When you have completed the evaluation of each proposal, submit your score for that proposal in the excel score sheet
Once you’ve reviewed and scored all proposals, deliver your filled in excel score card before October 27, end of business day.
Don’t worry if, after submitting your score card, you would like to add to or modify your comments. You will anyway get a chance to discuss your opinions with the other experts in the meetings in Brussels

20. Evaluation
Finally
Give a fair and clear opinion on each proposal. You are:
Independent : you represent yourself, not your employer, not your country…..)
Objective : you evaluate the proposal as written
Accurate : you use the official evaluation criteria only
Consistent : you apply the same standard of judgment to each proposal
Incommunicado : you do not disclose to anybody the contents of the proposals which you see

21. - The Smart@Fire Project Team
Finally
Thank you very much for your help!
- The Project Team