1. EE551 Real-Time Operating Systems
Safety Critical Systems Analysis
Course originally developed by
Maj Ron Smith

2. Safety Critical Software Systems – ilities of Systems
Software safety is one of the “ilities” of that is part of non-functional requirements specifies criteria that can be used to judge the operation of a system, rather than specific behaviors
What separates safety critical software from other kinds of
software is the potential that it has to contribute to
hazardous system states.
Hazards in a system are contributing
factors in the risks of accidents that can lead to the following:

3. Safety Critical Software Systems – ilities of Systems
Execution Qualities
Usability and Operability
Security
Reliability
Safety
Fault Tolerance
Evolution Qualities
Maintainbility, Understandability and Modifiability
Supportability (Integrated Logistics Support)
Testability
Portability
Scalability and Extensibility
Integrity – often used to encompass other ilities
What separates safety critical software from other kinds of
software is the potential that it has to contribute to
hazardous system states.
Hazards in a system are contributing
factors in the risks of accidents that can lead to the following:

4. Safety Critical Software Systems – ilities of Systems
Safety and reliability are often misinterpreted
There is a school of thought that states that safety is a subset of reliability
What separates safety critical software from other kinds of
software is the potential that it has to contribute to
hazardous system states.
Hazards in a system are contributing
factors in the risks of accidents that can lead to the following:

5. Major RW Smith Software Reliability (part1) - 5
reliability, R(t) - the probability that, when operating under stated environmental conditions, a system will perform its intended function adequately for a specified interval of time.
a measure of the success with which a system conforms to some authoritative specification of its behavior
most frequent hardware metric - MTBF
failure rate is more universal in software
probability that a program will operate correctly in a specified environment for a specified length of time.
21-Apr-17
Major RW Smith Software Reliability (part1) - 5

6. Safeware: System Safety and Computers Nancy G. Leveson
Safety Critical Software Systems – Authoritative text
Safeware: System Safety and Computers Nancy G. Leveson
ISBN-10: | ISBN-13:

7. Safety Critical Software Systems
Potential of the software to lead to hazardous system states
Hazards can lead to accidents and:
Death
Serious Injuries
Damage to environment
Significant loss of material
Loss of strategic advantage
What separates safety critical software from other kinds of
software is the potential that it has to contribute to
hazardous system states.
Hazards in a system are contributing
factors in the risks of accidents that can lead to the following:

8. Safety Critical Systems

9. Safety Critical Systems

10. Safety Critical Systems

11. Safety Critical Systems

12. Examples of failures: Medical
Therac-25 ( )(extreme case)
Bloodbank software released over 1M “failed” plasma units on the market.
Pacemakers reset to unsafe parameters due to external radiation sources (antitheft devices, microwaves,…)
Infusion pumps delivering the wrong rate of medicine.
The literature is full of examples of accidents involving safety critical systems. Here these examples list a few examples studied by the FDA.

13. Safety Critical Software Systems
Safety Critical Software cannot be verified and validated using “traditional” methods to derive test cases
Must use risk management and hazard analysis techniques
Root Cause Analysis
It is important to know that Safety Critical Software cannot be verified and validated using
traditional V&V methods
We must use risk management and hazard analysis techniques to prove the system safe.

14. Safety Critical Software Systems
Hazard Analysis techniques
Hazard list from similar devices
Hazard and Operability (HAZOP) Analysis
Fault Tree Analysis (FTA)
Event Tree Analysis (ETA)
Failure Modes and Effects Analysis (FMEA)
Failure Modes, Effects and Criticality Analysis (FMECA)
Here is a list of hazard analysis techniques that have been successfully used on safety critical software in the past.
In the next few of slides, we will take a brief look at each of the methods.

15. Safety Critical Systems - Hazard Analysis – Hazard List
Known hazards lists or reports from previous similar devices
Lessons Learned DB (internal to companies)
Recall notices (general public – industry wide)
Food and Drug Administration Web Site (MAUDE)
Federal Aviation Agency
Transport Canada (CADORS)
The simplest hazard analysis method consists of looking at known hazards list from previous similar devices.
These lists can come from lessons learned, recall notices or obtained from the FDA
If there are no similar systems, then a brainstorming session can be conducted.
Generic lists such as the one found in Annex D of ISO can be also be used.

16. Safety Critical Systems - Hazard Analysis – Hazard List
Brainstorming session
Generic lists (ISO Annex D)
The simplest hazard analysis method consists of looking at known hazards list from previous similar devices.
These lists can come from lessons learned, recall notices or obtained from the FDA
If there are no similar systems, then a brainstorming session can be conducted.
Generic lists such as the one found in Annex D of ISO can be also be used.

17. Safety Critical Systems - Hazard Analysis – HAZOP
Hazard and Operability Study
Process oriented
is a structured and systematic examination of a planned or existing process or operation
to identify and evaluate problems that may represent risks to personnel, equipment or environment
Originates from Chemical Industry
The next method is HAZOP.
HAZOP is ideal for process based systems.
It has been widely applied for safety analysis of chemical plants.
Hazop is used to identify hazards that may occur while operating a system outside of its original intent.

18. Safety Critical Systems - Hazard Analysis – HAZOP
Analyze the behavior of a system based on operating deviations from original design or intent
Decomposition of system into sub-processes or items
(systems, subsystems, components)
Parameters
(flow, temperature, pressure,…)
Systematic qualitative analysis with Guide words
(less, more, inverse, too high, too low, before…)
Hazop decomposes a system into items or sub-processes. Each process is described by a set of parameters.
The expected behavior of the system is analyzed systematically with the use of defined guide words.

19. Safety Critical Systems - Hazard Analysis - FTA
Fault Tree Analysis is the most used of all Hazard analysis methods for safety critical software.
Each hazard is analyzed using a deductive or backward approach. You start with the hazard then you trace back to the possible events that could cause it.
The resulting analysis is in the form of a Boolean tree. The interesting part of this technique is that the leaves of the tree form a test case generation oracle.
The weakness of FTA is that there is no chronological ordering to the events displayed.

20. Safety Critical Systems - Hazard Analysis - ETA
Control measures
Event Tree Analysis use a forward or inductive approach to hazard analysis.
The ETA tree is similar to a binary decision tree.
The protection mechanisms are listed at the top of the tree.
The tree starts with an event that can lead to a hazard. At each level, the tree will branch for successful and failure of the protection measure.S

21. Safety Critical Systems - Hazard Analysis – FME(C)A
Item
Failure Mode
Causes
Effects
Criticality
Prob
Control measures
Registration
RMS error too large
a. Bad configuration
b. Markers too close
c. Handling errors
d. Tracking error
e. Transformation error
Cannot use IIGS
Critical
N/A
Operator training
Documentation
Failure Modes, Effects and Criticality Analysis can take the form of a tree or table. Tables representations are more common due to the compact representation.
Each component or subcomponent including software is analyzed. The question we are trying to ascertain is what is going to happen if this component fails.
The down side of such an analysis is that it only looks at single points of failure.

22. Safety Critical Software Systems
State Based Analysis methods
Markov Chain Models
Petri Nets
Software Cost Reduction Methods
David Parnas and Constance L. Heitmeyer
Formal mathematical approach to specifications
Apart from using hazard analysis methods,
We can also use techniques that use state or modes and transitions to perform a safety analysis.
I will not go into detail here due to time restrictions.