1. DGTA-ADF Migrating to a Software Assurance Standard 2008 ADF Software Symposium FLTLT Patrick Redmond SCI-DGTA

2. DGTA-ADF Overview The Problem with Legacy Software Migrating to a Software Assurance Standard The Potential Challenges - Bounding the Change - Low-Level Requirements - Traceability - Structural Coverage Case Study - MIL-STD-498 to DO-178B - Considered throughout.

3. DGTA-ADF The Problem with Legacy Software Numerous ADF platforms acquired where no software assurance standard has been explicitly applied. Only MIL-STD-498 or DOD-STD-2167A. Development standards do not define how well software has to be constructed. During development, somebody made the decision to stop testing. - Why? During development, somebody made the decision to review source code. - Why? To what extent should the ADF rely on this software? To what extent does the ADF rely on this software?

4. DGTA-ADF Migrating to a Software Assurance Standard DGTA requires that a software assurance standard be applied to legacy software systems. All development methods apply a level of assurance, but is that level acceptable? Applying a software assurance standard to each modification of in-service software progressively increases the integrity. Applies ‘Cancer Theory’ to Legacy Software Two methods: - Service History Argument and Application of Recognised Standard  e.g. RTCA DO-178B - Software Assurance Task Matrix  Negotiate a custom software assurance “standard” with DGTA

5. DGTA-ADF How does migration work? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? Legacy Software of Unknown Integrity Known Integrity Software of Known(?) Integrity DGTA considers the software to be compliant with the applied software assurance standard from this point forward. Build x Area Impacted by Build x Build y

6. DGTA-ADF Service History Argument / Recognised Standard Establish a Service History Argument that demonstrates that the current software is acceptably safe. - Need to consider, among others:  Configuration Management  Problem Reporting  Change Control  Relevance of Product Service History  Operating Environments  Safety Related Problems  Design and Code Errors  Error Rates No need to create a service history argument for in-service software, ADF has already determined that it is acceptably safe. - Only applies to in-service, legacy software. Does not apply to acquisitions. - Does not apply if there is a substantial change of context in which the software is used. Apply a recognised software assurance standard to the next modification. DGTA then consider the software to be compliant with that standard.

7. DGTA-ADF Statement coverage. High-level requirements coverage. Software Assurance Task Matrix DesignCodeTest OFP B OFP C OFP D Source code is verifiable. Source code complies with low-level reqs. Low-level requirements are defined. High-level requirements are defined. Decision coverage. 178B Decision coverage. Negotiate with SCI Statement coverage plus untested code analysis.

8. DGTA-ADF The Potential Challenges Bounding the Change - How much “re-assurance” needs to be done? Traceability Low-Level Requirements Structural Coverage

9. DGTA-ADF The Easy Ones Planning Additional Considerations - Tool Qualification Development of High-Level Requirements, Source Code and Executable Object Code Verification of High-Level Requirements Configuration Management Quality Assurance

10. DGTA-ADF Bounding the Change Software assurance standard only needs to be applied to the scope of the modification. What is the scope of the modification? - The things that are changed, and - The things affected by the things that are changed. Change Impact Analysis: - Traceability - Memory Margin - Timing Margin - Data Flow - Control Flow - Input/Output - Development Environment and Process - Operational Characteristics - Certification Maintenance Requirements - Partitioning Analysis

11. DGTA-ADF Traceability Full traceability data was not recorded for many legacy software systems. Software assurance standards generally require traceability to source code. When applying a software assurance standard to a legacy software system, how much traceability data needs to be generated? DGTA Position: Trace down and up once. Trace down from new or modified high-level requirements to affected and new low-level requirements, to affected and new code. Trace up from affected and new code to low-level requirements and to high-level requirements.

12. DGTA-ADF Example – MIL-STD-498 to DO-178B This example will be considered a number of times. A legacy software system developed to MIL-STD-498 with typical artefacts. Can cause or contribute to Major hazards. Migrating to DO-178B Level C objectives.

13. DGTA-ADF HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 Traceability HLR2 LLR4 HLR1 LLR1LLR2LLR3 SCM1SCM2SCM3SCM4 SCM5 New High-Level Requirement New Low-Level Requirement Modified Low-Level Requirement New Source Code Modified Source Code

14. DGTA-ADF Low-Level Requirements Some legacy systems have no low-level requirements. Others have design descriptions that are not refined enough to be low-level requirements. Others have design descriptions that do not drive source code development (source code is developed from requirements). How many low-level requirements need to be defined or redefined? DGTA Position: Each affected low-level requirement and each low-level requirement identified by the down and up trace.

15. DGTA-ADF Low-Level Requirements HLR1HLR2 LLR1 LLR2 LLR3 LLR4 SCM1SCM2 SCM3SCM4SCM5 HLR2 LLR4 LLR3 SCM2 SCM3SCM4SCM5 LLR1

16. DGTA-ADF Low-Level Requirements Verify that low-level requirements LLR1, LLR3 and LLR4: - Comply with high-level requirements. - Are accurate and consistent. - Conform to standards. - Are traceable to high-level requirements. Ensure that low-level requirements LLR1, LLR3 and LLR4 are sufficiently refined to be directly translatable to source code. - May need to further refine one low-level requirement into several. Leave LLR2 as is.

17. DGTA-ADF Structural Coverage Software assurance standards generally require assessment of structural coverage in order to demonstrate that testing is complete. Purpose of Structural Coverage Objectives is to: - Identify shortcoming in requirements based test cases or procedures, - Identify inadequacies in software requirements, - Identify dead code, and - Identify deactivated code. How do these measures apply to modification of legacy software systems? DGTA Position: - For requirements based measures: each new, modified or affected requirement needs to be tested. - For structure based measures: each new or modified source code module.

18. DGTA-ADF Structural Coverage HLR1HLR2 LLR1 LLR2 LLR3 LLR4 SCM1SCM2 SCM3SCM4SCM5 HLR2 LLR4 LLR3 SCM2 SCM3SCM4SCM5 LLR1 HLR1 Statement Coverage Normal Range and Robustness Tests

19. DGTA-ADF Structural Coverage Normal Range and Robustness Tests - High-level requirements that have been added or modified. - High-level requirements where a dependent low- level requirement has been added or modified. - High-level requirements where the implementation (source code) of a dependent low-level requirement has been added or modified. - Low-level requirements that have been added or modified. - Low-level requirements where the implementation (source code) has been added or modified. Structural Coverage Objectives: - Structural coverage of all new or modified source code modules.

20. DGTA-ADF Additional Testing Also need to consider: - Data Dependencies  Has the change impacted data that other functions rely upon? - Control Flow Dependencies  Has the change inadvertently/adversely disrupted control flow or coupling? - Timing Dependencies  Has the change violated a timing constraint? - Memory Space Dependencies  Has the change violated memory constraints or used memory space assigned to other functions?

21. DGTA-ADF What about bug fixes? A large part of in-service support is the rectification of bugs. Bug fixes may not commence at the requirements level, may start with an identified fault in the source code. To what extent should software assurance standards be applied to bug fixes? DGTA Position: Trace up from the modified source code.

22. DGTA-ADF LLR7 LLR5 SCM8 HLR4 HLR3 What about bug fixes? LLR6 LLR8 SCM6SCM7 SCM9SCM10 Additional traceability data Structural Coverage Normal Range and Robustness Tests Low-Level Requirements

23. DGTA-ADF Conclusions DGTA requires the application of a software assurance standard to legacy software systems. All development methods provide a level of assurance, writing it down can identify gaps. A software assurance standard can be applied either by: - A Product Service History Argument and application of a recognised software assurance standard, or - A Software Assurance Task Matrix. The software assurance standard need only be applied to the current modification. - Determine extent through change impact analysis. For legacy software systems, a large number of assurance objectives are probably already being achieved. Meeting a recognised software assurance standard will probably require additional effort in the areas of: - Traceability, - Low-Level Requirements, and - Structural Coverage

24. DGTA-ADF Questions?