1. Copyright Critical Software S.A. 1998-2006 All Rights Reserved. VAL-COTS Validation of Real Time COTS Products Ricardo Barbosa, Henrique Madeira, Nuno Silva Dependability & Embedded

2. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 2 Overview - VALCOTS Introduction Objectives and Goals Developed Techniques Conclusions Future Work

3. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 3 Introduction (1) VAL-COTS VALCOTS is an acronym for “Validation of Real Time COTS Products” COTS current trend Commercial Off-the-shelf products are being used broadly in space applications How to have full confidence in COTS components? (custom made components confidence can be controlled and assessed in-house)... sometimes one may doubt of certain dependability characteristics of COTS

4. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 4 Introduction (2) Why are COTS different from custom made software? Different requirements needs... COTS are designed to be general purpose (requirements such as timing and safety might not be given high priority) Wrong assumptions... Product engineers tend to assume that it is impossible to predict all possible component violations for each function Overlooked tests... Input tests are sometimes overlooked (insufficient) due to performance or schedule constraints

5. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 5 Objectives and Goals (1) Goals The project main goal is to develop techniques and toolsets for the assessment of real time COTS products based on automated fault injection techniques Fault Injection...? Fault injection techniques have shown to be an excelent approach in the validation of COTS products Mentioned in Galileo Software Standard as an advisable technique to be used for the validation of space systems Benchmarks used fault injection techniques for validating real time constraints [DBench] Can simulate effects of radiation on hardware (space environment)

6. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 6 Objectives and Goals (2) Introduce the new concepts on fault injection and testing in NASA’s IV&V philosophy as an agency’s value added research and expertise Improve NASA’s IV&V processes, tools and techniques on dynamic testing

7. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 7 Objectives and Goals (3) Exchange of software metrics (on COTS and others) Collaborative knowledge exchange on IV&V, dependability evaluations, dynamic testing and automated testing between CSW(European Space Industry) and NASA

8. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 8 Developed Techniques (1) First Technique Three different techniques were developed during the project First Technique is focused on COTS robustness and stress capabilities Based on application requirements, API fault injection is performed to assess the COTS component in the presence of in and out of bounds values This process is accelerated and tunned up by performing software automated fault injection in upstream components of the target COTS

9. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 9 mutex_change_priority ( TCB *p_tcb, UINT oldpri, UINT *p_newpri) Type Name Test Values INT-2147483648, -1, 0, 1, 2147483647 UINT0, 1, 4294967295 BOOLtrue, false FN-2147483648, -1, 0, 1, 2147483647 __xception__ = 0; m_p_tcb = mutex_change_priority ( p_tcb, oldpri, __xception__); __xception__ = 1; m_p_tcb = mutex_change_priority ( p_tcb, oldpri, __xception__); __xception__ = 4294967295; m_p_tcb = mutex_change_priority ( p_tcb, oldpri, __xception__); Developed Techniques (2) First Technique Automatic Test Cases Generation

10. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 10 Developed Techniques (3) Second Technique The second technique is double phased: (1) COTS component dependability is assessed individually at an earlier development stage (2) Software fault injection is performed inside the actual component to assess the component (and system) within the full integrated architecture, conducted at a latter developement stage Results may lead to architectural changes or component wrapping, or even COTS component replacement

11. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 11 Developed Techniques (4) Second Technique Earlier Stages of SDP Latter Stages of SDP

12. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 12 Developed Techniques (5) Third Technique A third technique was developed based on the second one To complement the technique, risk assessment calculations are also introduced to estimate failure probability of the component within the system Risk assessment calculations are based on software metrics from component and system complexity (e.g. cyclomatic complexity)

13. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 13 Conclusions The case studies conducted have shown that the applicaiton of any of the techniques lead to architectural and component improvements Thus, robustness and dependability improvements Results identify that COTS components have intrinsic faults that are hard to find with traditional testing Usefulness of fault injection Xception based automated testing as proven to be very usefull particularlly considering fault injection definition and injection automation Some difficulties where encountered while evaluating injection run results as it is important to improve results analysis automation

14. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 14 Future Work VAL-COTS II Improvement of the developed techniques Further testing to validate the techniques (hopefully on an on-going project) Adjust techniques for use in certification processes (follow specific standards like ISO 25051 or DO- 178B) Improve the techniques based on the newly developed software fault injection G-SWFIT G-SWFIT is a technique developed at Universidade de Coimbra for emulating real faults in fault injection Using this technique, no source code is needed for performing fault injection, only binary files

15. © Copyright Critical Software S.A. 1998-2006 All Rights Reserved. 15 The end! Thank You!! Any questions? rbarbosa@criticalsoftware.com hmadeira@criticalsoftware.com nsilva@criticalsoftware.com