1. SAFEWARE System Safety and Computers Chap18:Verification of Safety Author : Nancy G. Leveson University of Washington 1995 by Addison-Wesley Publishing Company, Inc. ISBN : 0-201-11972-2 Presentation by Yean-Ru Chen at ESL of CS, CCU, Taiwan 2005/09/08

2. Outline(1/1) Introduction Hazard analysis Dynamic Static Conclusions

3. Introduction(1/2) Most verification techniques focus on showing consistency between the code and the specification, but for safety, this approach is not enough. We need to examine not only the relationship between the computer inputs and outputs, but also that between the inputs and the effect of the outputs on system behavior.

4. Introduction(2/2) Because of considering the effect of the outputs on system behavior, “ Hazard analysis ” is needed to serves as the basis for this approach to safety verification.

5. Hazard analysis Hazard is a state or set of conditions of a system that, together with other conditions in the environment of the system, will lead inevitably to an accident. (Chapter 9) There are two main analysis ways: (1)Dynamic Analysis In this analysis, the code or a model of the code is executed and its performance is evaluated. (2)Static Analysis In static analysis, the code or model is examined without being executed.

6. Dynamic Analysis(1/6) Testing! Here we ’ ll focus on software and safety only. Process Considerations Limitations of Testing

7. Dynamic Analysis(2/6) Process Considerations: There are several aspects of the system needed to be considered when we use the “ Testing ” approach: Critical functions and variables. Boundary conditions. Special features, such as firewalls or safety kernels upon which the protection of the safety-critical features is based.

8. Dynamic Analysis (3/6) Reaction of the software to system faults and failures. The model of the system environment that guides critical control and safety decisions. The operator interface.

9. Dynamic Analysis (4/6) In general, there are also several aspects for the safety engineers to pay more attention : Review test plans Recommend tests based on the hazard analyses, safety standards, previous accidents, and so on. Review the test results for any safety-related problems that were missed in the analysis or in other testing. Ensure that testing feedback is integrated into the safety reviews and analyses that will be used in design modifications.

10. Dynamic Analysis (5/6) Limitations of Testing: The main limitation of testing is that testing is impossible in a real setting. Most testing must be done in a simulation mode, and there ’ s no way to guarantee that the simulation is accurate.

11. Dynamic Analysis (6/6) But these limitations do not mean that we should not test or include testing in a safety program. It merely means that there is a limit to the confidence that can be acquired through dynamic analysis and that testing may have to be augmented with static analysis if the limit is less than what is required.

12. Static Analysis Static analysis evaluates the software without executing it. Instead, it examines a representation of the software. There are two major means in this chapter : Formal Verification --- formal Software Fault Tree Analysis (SFTA) --- informal

13. Formal Verification (1/4) We mainly concern about the formal verification on software in this book. Formal verification essentially provides a proof of consistency between two formal specifications of a system. In safety, we have to use formal models and definitions of safety properties along with safety policies.

14. Formal Verification (2/4) A safety policy would describe the rules for determining whether a system is safe and would use the safety constrains as a basis for decision making. There are three general safety policies for which the specification could be checked: (Chapter 15)

15. Formal Verification (3/4) (I) There must be no paths to unplanned hazardous states. (II) Every hazardous state must have a path to a safe state. All paths from the hazardous states must lead to safe states. Time in the hazardous state must be minimized, and contingency action may be necessary to reduce risk while in the hazardous state.

16. Formal Verification (4/4) (III) If a safe state cannot be reached from a hazardous state, there must exist at least one path from that state lead to a minimum risk state. Major drawbacks of formal verification on software : Practicality and Feasibility. Major advantages of formal verification on software : Analytical,Mathematical and Formal.

17. Software Fault Tree Analysis (SFTA) (1/4) FTA focuses on hazards rather than on software errors in general, and it can be applied to assembly or high-level language. A leaf node of the software fault tree describes a behavior of the computer. SFTA traces that behavior into the logic of the code and determines whether a path exists through the code that could cause the hazardous output.

18. Software Fault Tree Analysis (SFTA) (2/4) SFTA uses Backward Reasoning approach. It follows the paths backward through the program from the hazardous outputs to identify any inputs that might cause that output.

19. Software Fault Tree Analysis (SFTA) (3/4) Here is an example of backward reasoning about variable values : X = 9 X =10 if a variable contains the value 10 after an instruction adds one to that variable, then the variable must have held the value 9 before the instruction was executed. X := X + 1

20. Software Fault Tree Analysis (SFTA) (4/4) SFTA uses fault tree templates to generate the software fault trees. Here are 4 templates to be introduced: Template for assignment Template for IF statement Template for WHILE statement Template for procedure call statement

21. Template for assignment Fig.18_4 Assignment causes event or Operand evaluation causes event Exception causes eventChange in value causes event

22. Ex. of assignment statements and simple IF ( A Tree !) (1) A := F(Y) (2) B := X-5.0 (3) IF A>B THEN FIRE FIRE called A > B F(Y) > X – 5.0 A > X – 5.0

23. Template for IF statement Fig.18_6 Event in IF- THEN-ELSE or Condition FALSE, ELSE-part causes event Condition evaluation causes event Condition TRUE, THEN-part causes event and Condition TRUE before IF-THEN- ELSE statement THEN-part causes event Condition FALSE before IF-THEN- ELSE statement ELSE-part causes event

24. Ex. of IF statements( A Tree !) analyze for the unsafe condition “ X>100 ” if a > b then X := f(X) ; else X := 100 ; X>100 caused by IF statement or a>b, X := f(X) causes X>100 and a > b before IF- THEN-ELSE X := f(X) causes X>100 a <= b before IF- THEN-ELSE X := 100 causes X >100 a 100

25. Template for WHILE statement Fig.18_8 WHILE statement causes event or Statement executed n times Statement not executed and Event before WHILE statement Condition false before WHILE n_th iteration causes event Condition true before WHILE

26. Ex. of WHILE statements( A Tree !) analyze for the unsafe condition “ z>100 ” (1/2) while b>x { b:=b-1; z:=z+10; } WHILE statement causes z>100 or Statement executed n times Statement not executed and z>100 before WHILE statement b <= x before WHILE z + 10(b-x) >100 before WHILE b > x before WHILE

27. Ex. of WHILE statements( A Tree !) analyze for the unsafe condition “ z>100 ” (2/2) Why z+10(b-x) > 100 will cause unsafe condition when “ while loop ” has been executed n times? Original value of z before loop : z Original value of b before loop : b The value of z after nth iteration :z ’ The value of b after nth iteration :b ’ z ’ = z+10n, b ’ = b – n Since iteration stopped, b ’ = x (at least) So, z ’ = z+10(b-x) If z ’ > 100,then it ’ s unsafe.

28. Template for procedure call statement Fig.18_9 F (p1,p2,p3,…) causes event or Failure of F causes event Execution of F with p1,p2,p3…causes event Evaluation of parameters causes event

29. Uses and Evaluation of SFTA(1/2) If SFTA is used with a system simulator, the critical software interfaces with the rest of the system can be identified and used to determine appropriate simulation states and events. However, the practicality of SFTA use on large-scale software has not been demonstrated.

30. Uses and Evaluation of SFTA(2/2) One of the reasons is that SFTA is basically a structured walkthrough of the code with an emphasis on hazardous behavior, so its practicality should be similar to that of other structured walkthrough techniques. However, the entire code is not usually safety-critical, maybe small part of it need to be examined in SFTA.

31. Conclusions Those using computers for safety- critical functions should be wary of relying too heavily on the techniques described in this chapter. They are necessary but not sufficient!!