2. WP 7 - Regola TASK 7.4 : Pilot Application Validation D 7.4 Functional and Non-functional Evaluation Criteria for C2-SENSE Pilot Applications

3. WP 7.4 – Pilot application validation In this task, the testing environment for the C2-SENSE pilot application, which will be developed on top of the integrated C2-SENSE components, will be constructed. The assessment criteria will be defined as test case scenarios. The required tests that will be performed on the developed pilot application for the thoroughly examination of the system will be defined. The realized system for the pilot application scenarios will be tested as a whole in the test environment by the end users. The tests will be in parallel with the deployment of the pilot application. The feedback from the users will be collected through questionnaires which will be provided to the system developers to enhance the functionalities of the C2-SENSE final product.

4. WP 7.4 - concept Verification : technical test works. The product is produced right. Validation: User approve the functions. It’s the right product Acceptance test: is the final test before a delivery of oa product or a service Test case: How? Why? Wich purpose?

5. WP 7.4 - concept VALIDATION ACCEPTANCE TEST EU Research, Not a product technical aspects difficult to evaluate C2-SENSE VALIDATION This is the challenge

6. WP 7.4 - Evaluation Team Domain and IT Expert (REGOLA) User operator ( Regione Puglia, variuous org.) Technical expert of C2-SENSE components (Lutech, SRDC, SAGEM)

7. WP 7.4 - Object of Evaluation FUNCTIONAL REQUIREMENT: what C2-SENSE DO NON FUNCTIONAL REQUIREMENT: (simplifying) How C2-SENSE do it? It’s a good solution?

8. WP 7.4 - Evaluation Process Inspecion Analysis Questionnaire SCORE REPORT EVALUATION PILOT APPLICATION EXECUTION Function evaluation Non functional evaluation

9. WP 7.4 - Functional Evaluation FUNCTIONAL EVALUATION

10. WP 7.4 – Functional evaluation Architecture of the pilot application

11. WP 7.4 – Functional evaluation

12. WP 7.4 - Functional Evaluation The idea is: To evaluate C2-SENSE component indirectly (where possible) examining the effects on the user operation on the pilot applicatoin

13. WP 7.4 - Functional Evaluation - the scopes Scope 4.1.2.1 Situation Reporting 4.1.2.2 Mission Plan 4.1.2.3 Scheduling 4.1.2.4 Resource Management 4.1.2.5 Alert 4.1.2.6 Hospital Communication 4.1.2.7 Tracking of Citizens 4.1.2.8 Sensor Management 4.1.2.9 Enterprise User Authentication and Authorization (EUAA) 4.1.2.10 Audit Trail and Node Authentication (ATNA) 4.1.2.11 Emergency Situation Map 4.1.2.12 Situation Analysis 4.1.2.13 Permission Action Scope are the ‘functional area of investigation’ of the Functional Validation process The idea is to identify the ‘integration profile’ as Scope of investigation.

14. WP 7.4 - Functional Evaluation Objective Evaluation (the system work as espected) Subjective Evaluation (operator approve the behaviour of the solution) Global Evaluation

15. WP 7.4 - Functional Objective Evaluation Objective Evaluation High Level test case SCORE organized per ‘Action scope’

16. WP 7.4 - Functional Objective Evaluation Score are assigend according a pre-defined schema scoremeaning 0Function not covered by the solution at all. 1 Function covered minimally, but is not sufficient and cannot be accepted in the real world. 2 Function covered, but some critical/important issue was detected (Eg: cost too high, bad performance…). The IA can accept the issue temporarly with the promise of a fix. 3 Function covered and it is coherent with the reference. Some issue was identified, but a user can accept the solution. 4 Function covered, it is coherent with the reference and no issue was identified.

17. WP 7.4 - Functional Objective Evaluation Report contain annotation and a radar chart REPORT

18. WP 7.4 - Functional Evaluation FUNCTIONAL SUBJECTIVE EVALUATION

19. WP 7.4 - Functional Subjective Evaluation The idea is: TO ASK to the operator a subjective evaluation of the soluiton, using a pre-defined questionnaire organaized by scope BUT C2-SENSE is not a product: for this reason the interdisciplinary team is mandatory, in order to explain user to asnwer!! Regola (Domain and IT expert) will drive the survey. The scopes are the same of the functional objective evaluation

20. WP 7.4 - Functional Subjective Evaluation Example of subjective evaluation survey Scope Subjective evaluation criteria 4.1.2.1 Situation Reporting - the solution handles all the information you need? - the solution reduce the operator’s work load? - the solution decreases your response time in emergency management? - the solution decreases the loss of information? - overall impression from 0 to 4.? 4.1.2.2 Mission Plan - the solution handles all the information you need? - the solution reduces the operator’s work load? - the solution decreases your response time in emergency management? - the solution decreases the loss of information? - overall impression from 0 to 4? annotation: could be useful to verify with operator if C2-SENSE helps to share procedures or protocols among actors.

21. WP 7.4 - Functional Subjective Evaluation TO EVERY QUESTION WILL BE ASSIGNED A SCORE USING A PRE-DEFINED SCHEMA! scoremeaning 0I totally disagree - or – very bad 1I disagree – or - bad 2Neutral 3I approve – or - good 4I totally approve – or – very good

22. WP 7.4 - Functional Objective Evaluation Report contain annotation and a radar chart REPORT

23. WP 7.4 - Functional Evaluation FUNCTIONAL GLOBAL EVALUATION

24. WP 7.4 - Functional Evaluation Objective Evaluation (the system work as espected) Subjective Evaluation (operator approve the behaviour of the solution) Global Evaluation

25. WP 7.4 - NON Functional Evaluation NON FUNCTIONAL EVALUATION

26. WP 7 - NON Functional Requirement Non Functional Requirement: Accessibility Audit and control Availability (see service level agreement) Backup Capacity, current and forecast Certification Compliance Configuration management Dependency on other parties Deployment Documentation Disaster recovery Efficiency (resource consumption for given load) Effectiveness (resulting performance in relation to effort) Emotional factors (like fun or absorbing or has "Wow! Factor") Environmental protection Escrow Exploitability Extensibility (adding features, and carry-forward of customizations at next major version upgrade) Failure management Fault tolerance (e.g. Operational System Monitoring, Measuring, and Management) Legal and licensing issues or patent-infringement- avoidability Interoperability Maintainability Modifiability Network topology Open source Operability Performance / response time (performance engineering) Platform compatibility Price Privacy Portability Quality (e.g. faults discovered, faults delivered, fault removal efficacy) Recovery / recoverability (e.g. mean time to recovery - MTTR) Reliability (e.g. mean time between failures - MTBF, or availability) Reporting Resilience Resource constraints Response time Reusability Robustness Safety or Factor of safety Scalability (horizontal, vertical) Security Software, tools, standards etc. Compatibility Stability Supportability Testability Usability by target user community User Friendliness

27. WP 7 - NON Functional Requirement ISO 25000 series give a working framework on NON Functional Requirement

28. WP 7 - NON Functional Requirement Intrinsic Qualities Functional Suitability Performance Efficiency Compatibility Usability Reliability Security Maintainability Portability Usage Qualities Effectiveness Efficiency Satisfaction Freedom from Risk (Safety) Context Coverage (Usability Scope) External Qualities Service Cost Vendor Risk Mitigation Product Risk Mitigation

29. WP 7 - NON Functional Requirement Was defined a detailed list of NON funcional requirement Most of them will be evaluated thinking at the POTENTIAL C2SENSE final product deployed in production mode.

30. WP 7 - NON Functional Requirement EXAMPLE Intrinsic quality – Reliability – Recoverability Definition : The degree to which, in the event of an interruption or a failure, a product or system can recover the data directly affected and re-establish the desired state of the system will be evaluated:  Recovery Time Objective (RTO): It’s the time to recover full funcions after disarter. It must tend to 0 in Emergency management  Recovery Point Objective (RPO): Max time from data producing and data backup. It must be near to 0 in Emergency management  High availability architecture ready The evaluation will be performed on all C2-SENSE components deployable in production mode on a possible production architecture (not that usesd in pilot application)

31. WP 7 - NON Functional Requirement Evaluation -> SCORE - 0 : NFR not covered by the solution at all - 1 : NFR covered minimally but is not sufficient and cannot be accepted in the real world - 2: NFR covered but some important issue was detected (Eg: cost too high, bad performance, etc.). The IA can accept the issue temporarily with the promise of a fix. - 3 : NFR covered and it is coherent with the reference. Some issue was identified but a user can accept the solution - 4 : NFR covered, it is coherent with the reference and no issue was identified. Annotation: in order to identify issue it’s important to mix :  IA operator’s point of view  Technical point of view  State of the art of IT solutions

32. WP 7.4 - Functional Objective Evaluation Report contain annotation and a radar chart REPORT

33. Issue WP 7.4 - ISSUE

34. Pilot application need to be well defined technically WP 7.4 - ISSUE

35. Pilot application seems to be to complex and less effective Scope Defined testnote 4.1.2.1 Situation Reporting 35 too much 4.1.2.2 Mission Plan 10 4.1.2.3 Scheduling 1 4.1.2.4 Resource Management 14 4.1.2.5 Alert 13 4.1.2.6 Hospital Communication 2 4.1.2.7 Tracking of Citizens 5 4.1.2.8 Sensor Management 3 4.1.2.9 EUAA 0 correct- too technical 4.1.2.10 Audit Trail and Node Authentication (ATNA) 0 correct- too technical 4.1.2.11 Emergency Situation Map 5 4.1.2.12 Situation Analysis 7 4.1.2.13 Permission 0 configuration 2 correct- too technical WP 7.4 - ISSUE

36. NON FUNCTIONAL REQUIREMENT MUST TO BE EVALUATED BY OTHER PARTNER WP 7.4 - ISSUE

37. Thank you for your attention