1. Presentation R. R. Lutz. Analyzing Software Requirements Errors in Safety-Critical Embedded Systems. In Proceedings of the IEEE International Symposium on Requirements Engineering, 1993. Phyllis Cheng 9/17/2002

2. Goal To proof that safety-related software errors tend to be produced by different error mechanisms than non-safety-related ones To proof that safety-related software errors tend to be produced by different error mechanisms than non-safety-related ones Then, the system safety can be directly enhanced by targeting the causes of safety-related errors Then, the system safety can be directly enhanced by targeting the causes of safety-related errors Finally, to reduce safety-related software errors Finally, to reduce safety-related software errors

3. Research Target 2 spacecrafts: Voyager and Galileo 2 spacecrafts: Voyager and Galileo reasons: reasons: - spacecrafts’ software is safety- critical - spacecrafts’ software is safety- critical - each spacecraft involves embedded software distributed on several different flight computers - each spacecraft involves embedded software distributed on several different flight computers

4. OVERVIEW OF CLASSIFICATION Program Faults Program Faults (Documented Software Errors) (Documented Software Errors) Human Error Human Error (Root Causes) (Root Causes) Process Flaws Process Flaws (Flaws in Control of System Complexity + Inadequacies in Communication or Development Method) (Flaws in Control of System Complexity + Inadequacies in Communication or Development Method)

5. Relationships between Program Faults, Root Causes and Process Flaws Program Faults Root Causes Process Flaws Effects Causes

6. PROGRAM FAULTS Internal Faults : Internal Faults : Syntax, Programming Language Semantics Syntax, Programming Language Semantics Interface Faults: Interface Faults: Interaction with other Software Components Interaction with other Software Components Interaction with hardware in the system Interaction with hardware in the system Functional Faults: Functional Faults: Operating Faults Operating Faults Conditional Faults Conditional Faults Behavioral Faults Behavioral Faults

7. HUMAN ERROR Coding or Editing Errors Coding or Editing Errors Communication Errors Communication Errors Within a Team Within a Team Between Teams Between Teams Requirements Requirements Errors in Recognition Errors in Recognition Errors in Deployment Errors in Deployment

8. PROCESS FLAWS Flaws in Inspection and Testing Methods Flaws in Inspection and Testing Methods Inadequate Interface-Specification & Communication Inadequate Interface-Specification & Communication Between Software Designers and ProgrammersBetween Software Designers and Programmers Between Software and Hardware EngineersBetween Software and Hardware Engineers Requirements Not Identified or Understood Requirements Not Identified or Understood Incomplete DocumentationIncomplete Documentation Missing RequirementsMissing Requirements Inadequate DesignInadequate Design

9. Results: Program Faults Program Faults Non-Safety-RelatedSafety-Related VoyagerGalileoVoyagerGalileo Internal Faults 1%3%0%2% Interface Faults 34%18%36%19% Function Faults 65%79%64%79% Operating Operating22%43%19%44% Conditional Conditional26%10%31%18% Behavioral Behavioral52%47%50%38% Both errors display similar proportions of program faults Functional Faults are the most common kind of software error Almost half of the errors are attributable to behavioral faults

10. Root Causes causing Interface Faults Interface Faults Non-Safety-RelatedSafety-Related Root Causes (Human Errors) VoyagerGalileoVoyagerGalileo Communication Within Teams 7%28%7%22% Communication between Teams : Misunderstood H/W Interface Spec Misunderstood H/W Interface Spec50%42%67%48% Misunderstood S/W Interface Spec Misunderstood S/W Interface Spec43%30%26%30% The primary cause of safety-related interface faults is Misunderstood H/W interface spec Communication error between development teams is the leading cause of interface faults

11. Root Causes causing Functional Faults Functional Faults Non-Safety-RelatedSafety-Related Root Causes (Human Errors) VoyagerGalileoVoyagerGalileo Requirement Recognition Error 47%62%62%79% Requirement Deployment Error 53%38%38%21% The primary cause of safety-related functional faults is errors in recognizing the requirements Non-safety-related functional faults are more often caused by errors in deploying implementing the requirements (Problem: small scale of statistic sample!!!)

12. Process Flaws causing Interface Faults Interface Faults Non-Safety- Related Safety-Related Process Flaws VoyagerGalileoVoyagerGalileo Flaws in Control of System Complexity: Interfaces not adequately identified or understood Interfaces not adequately identified or understood70%85%56%87% H/W behavior anomalies H/W behavior anomalies30%15%44%13% The most common complexity-control flaw is interfaces not adequately identified or understood for both non-safety-related and safety- related errors

13. Process Flaws causing Interface Faults Interface Faults Non-Safety- Related Safety-Related Process Flaws VoyagerGalileoVoyagerGalileo Flaws in Comm. Or Dev. Methods causing misunderstood S/W interfaces Interface Spec known but not documented or communicated Interface Spec known but not documented or communicated20%38%11%35% Interface design during testing Interface design during testing26%2%19%0% Flaws in Comm. Or Dev. Methods causing misunderstood H/W interfaces Lack of communication between H/W and S/W teams Lack of communication between H/W and S/W teams24%28%26%35% H/W behavior not documented H/W behavior not documented30%32%44%30% (Problem: small scale of statistic sample!!!)

14. Process Flaws causing Functional Faults Functional Faults Non-Safety- Related Safety-Related Process Flaws VoyagerGalileoVoyagerGalileo Flaws in Control of System Complexity: Requirements not identified: unknown, undocumented or wrong Requirements not identified: unknown, undocumented or wrong37534460 Requirements not understood Requirements not understood63475640 Requirements not identified Safety-related functional faults are more likely than non- safety-related functional faults to be caused by Requirements not identified (Problem: small scale of statistic sample!!!)

15. Process Flaws causing Functional Faults Functional Faults Non-Safety- Related Safety-Related Process Flaws VoyagerGalileoVoyagerGalileo Flaws in Comm. Or Dev. Methods causing errors in Recognizing Requirements Spec imprecise or unsystematic Spec imprecise or unsystematic16%28%21%38% Requirements missing from requirements documents Requirements missing from requirements documents31%35%42%42% Flaws in Comm. Or Dev. Methods causing errors in Deploying Requirements 53%37%37%20% The source of safety-related software errors lie in inadequate requirements, however, the source of non- safety-related errors more commonly involve inadequacies in the design phase (Problem: small scale of statistic sample!!!)

16. Comparison of Results with Previous Work 1. The role of Interface Specifications in controlling software was underestimated 2. Previous reports analyzed fairly simple systems in well-understood application domains

17. Comparison of Results with Previous Work (contd.) 3. Underestimate the impact of unknown requirements on the scope and schedule of the later stages of the S/W dev. Process 4. Distinction between causes of safety critical and non-safety critical S/W errors was not adequately investigated (?)

18. Recommendations (6) 1. Focus more on the interface between software and the system in analyzing the problem domain 2. Identify safety-critical hazards early in the requirements analysis

19. Recommendations (6) 3. Use formal specifications techniques in addition to natural-language software requirements 4. Promote informal communication among teams

20. Recommendations (6) 5. As requirements evolve, communicate the changes to the development and test teams 6. Include requirements for “defensive design”

21. Later Work (1993) Safety Checklist was developed to analyze software requirements, particularly with regard to interfaces and robustness Safety Checklist was developed to analyze software requirements, particularly with regard to interfaces and robustness It can be used as a first step towards specifying and checking safety constraints or as a supplement to the formal inspection of requirements spec. It can be used as a first step towards specifying and checking safety constraints or as a supplement to the formal inspection of requirements spec. Lutz, Robyn R. (1993) Targeting Safety-Related Errors During Software Requirements Analysis. Technical Report TR93-11, Department of Computer Science, Iowa State University. Lutz, Robyn R. (1993) Targeting Safety-Related Errors During Software Requirements Analysis. Technical Report TR93-11, Department of Computer Science, Iowa State University.Lutz, Robyn R. (1993) Targeting Safety-Related Errors During Software Requirements Analysis. Technical Report TR93-11, Department of Computer Science, Iowa State University.Lutz, Robyn R. (1993) Targeting Safety-Related Errors During Software Requirements Analysis. Technical Report TR93-11, Department of Computer Science, Iowa State University.

22. End of Presentation Phyllis Cheng 9/17/2002