October 22, 2005 Parvaiz Ahmed Khand An Overview of Software Safety

2  Introduction  Concept of safety  Safety Terminology  Relationship between safety and reliability  Relationship between safety and security  Safety Analysis process  Summary  References ContentsContents

3 IntroductionIntroduction  Hardware backups, interlocks, and other safety devices are currently being replaced by software in many types of systems, including nuclear power plants, commercial aircrafts and weapon systems.  Software can affect system safety in two ways:  It can exhibit behavior in terms of output values and timing that contribute to the system reaching a hazardous state,  It can fail to recognize or handle hardware failures that it is required to control or to respond to in some way.  In recent years causes of some software related accidents were:  Though the requirements were specified correctly, but the behavior specified by the requirements was not safe from a system perspective  The requirements do not specify some particular behavior that is required for system safety  The software has unintended ( and unsafe) behavior beyond what is specified in the requirements

4 Concept of safety  Reduced Level of risk  E.g. Safety matches and safety razor  Not safe, only safer than their alternatives in their alternatives in a certain environment  Striking a match in a room filled with a combustible- air mixture  Shaving on a injured skin

5 Safety Terminology  System  A set of components that act together as a whole to achieve some common goal, objective, or end.  The components are all interrelated and either directly or indirectly connected to each other.  System state  At any point in time is the set of relevant properties describing the system at that time.  System environment  A set of components ( and their properties) that are not part of the system but whose behavior can effect the system state.

6  Safety critical software  Any software that can directly or indirectly contribute to the occurrence of a hazardous system state  Safety critical functions  Those system functions whose correct operation, incorrect operation (including correct operation at wrong time), or lack of operation could contribute to a system hazard.  Safety critical software functions  Those software functions that can directly or indirectly, in association with other system component behavior or environmental conditions, contribute to the existence of a hazardous state.  Accident  An undesired and unplanned or unintentional (but not necessarily un expected) event that result in (at least) a specified level of loss.  Incident  A near miss or incident is an event that involves no loss ( or only minor loss), but with potential for loss under different circumstances. Safety Terminology

7  Damage  A measure of the loss resulting from a mishap. Damage can range from many people killed as a result of an accident to minor injury or property damage.  Hazard  A state or set of conditions of a system (or an object) that, together with other conditions in the environment of the system (or object), will lead inevitably to an accident (loss event)  Hazard Characteristics  Severity: An assessment of the worst possible damage that could result from a particular hazard. Hazard severity can range from catastrophic where many people are killed to minor where only minor damage results.  Likelihood of occurrence  Hazard probability  The probability of the events occurring which create a hazard. Probability values tend to be arbitrary but range from probable (say 1/100 chance of a hazard occurring) to implausible (no conceivable situations are likely where the hazard could occur). Safety Terminology

8  Safety  The probability that conditions that can lead to a mishap (hazards) do not occur, whether or not the intended function is performed  Freedom from accidents or losses  Minimization of risks to an acceptable level Safety Terminology

9 Relationship between Safety and reliability  Software Reliability  The probability of failure free operation of a software (computer program) for a specified time in an specified environment  Reliability requirements are concerned with making a software failure free, whereas safety requirements are concerned with making it hazard free.

10 Relationship between Safety and security  Security is a system property that reflects the system’s ability to protect itself from accidental or deliberate external attack  Important in systems, which are networked, so that access to the systems through internet is possible  Damage from insecurity  Denial of service  Corruption of program or data  Disclosure of confidential information  Essential prerequisite for system availability, reliability and safety.

11 Ways in which computers are used in safety critical loops 1.To provide information or advice to human controller upon request 2.To interpret data and display it to controller, who makes the control decisions 3.Issuing commands directly, but with a human monitor of the computer actions providing a varying level of input. 4.Eliminating the human from control loop completely by using automatic control by computer

12 Software faults/failures  Types  Primary: They occur when the software errors result in the output, not meeting its specification  Secondary: They occur when the inputs differ from what was anticipated and designed for.  Command: They occur when the software responds to erroneous inputs that are expected but occur at wrong time or in the wrong order  Factors for severity classification  Cost impact  Human life impact  Service impact

13 Hazardous software behavior control  Identify and eliminate hazards through special safety analysis of the software requirements and code  Add hardware interlocks or human controls to the system design  Add special protection into the software itself such as:  Software interlocks  Fail safe software  Software monitoring or self checking mechanism

14 Safety Analysis process  Hazard Analysis  Identification of hazards and assessment of hazard level  In Development Phase: Identification and assessment of potential hazards, and conditions that can lead to them, so that hazards can be eliminated or controlled  In Operation Phase: Examination of an existing system to improve its safety and to formulate policy and operational procedures  In licensing Phase: Examination of a planned or an existing system to demonstrate acceptable safety to a regulatory authority  Risk Analysis  Identification and assessment of the environmental conditions along with exposure or duration

15 Stages of safety analysis  Preliminary Hazard Analysis (PHA)  Identification of safety critical areas and functions,  Identification and evaluation of hazards, and  Identification of the safety design criteria to be used.  Used in the earliest development life-cycle phases  System Hazard analysis (SHA)  It involves detailed studies of possible hazards created in the interfaces between subsystems or by the system operating as a whole, including potential human errors  started as design matures  Subsystem hazard Analysis (SSHA)  It examines subsystems or components in detail for the identification of hazards  started as soon as the subsystems are designed in sufficient detail  Operating and support hazard Analysis (OSHA)  It identifies hazards and risk reduction procedures during all phases of system use and maintenance

16 Hazard search strategy  Hazard Analysis techniques involve searching  The search strategy will depend upon the type of structure being searched including:  Basic elements of underlying model  Components (Physical or logical),  Events,  Conditions, or  Tasks  Relationship between those elements  Temporal (time or sequence related)  Structural (whole part)

17 Hazard search methods  Forward and backward  Useful when the underlying structure is temporal and the elements are events, conditions, or tasks.  Top down and bottom up  Useful when the relationships being investigated are structural  Combination of these two  Useful when the relationships when the relationships cannot be categorized according to these categories.

18 Hazard search methods  Forward Search (inductive)  Useful to look at the effect on the system state of both an initiating event and later events that are not necessarily caused by an initiating event  combinatorial explosion problem  Backward Search (deductive)  Useful to determine the paths that can lead to a particular hazard or accident  the most efficient method  Top down search  Used to examine the hazardous system behavior by examining the effect of all possible combinations of component behavior  Bottom up search  Used to examine the effect of individual component failures on the overall behavior of the system

19 Risk reduction procedures  Hazard Elimination  Eliminating the hazard either by eliminating the hazardous state itself or by eliminating the negative consequences (losses) associated with that state  Hazard Reduction  Reducing the occurrence of hazard by reducing the likelihood of hazard  Hazard Control  Reducing in the likelihood of hazard by minimizing the duration and exposure of hazard  Damage minimization  Reducing the consequences or losses by providing warnings and contingency actions

20 Safety Verification  Showing that a fault cannot occur, that is, 1. the software cannot get into an unsafe state and cannot direct the system into an unsafe state, or 2. showing that if a software fault occurs, it is not dangerous.  Verification Techniques  Dynamic Analysis: Code and model is executed and its performance is evaluated  Static Analysis: The code or model is examined without being executed  Formal verification  Software fault tree Analysis

21 SummarySummary  Use of computers in safety critical systems  Potential increase in safety  Decrease in safety margins  Hazards are not always caused by failures, and all failures do not cause hazards  Reliability requirements are concerned with making a software failure free, whereas safety requirements are concerned with making it hazard free.  Software security is an essential prerequisite for system availability, reliability and safety.  Preventing failures  increasing reliability  Preventing hazards  increasing safety  Preventing malicious intrusions  increasing security  Hazard Analysis techniques are used to identify the hazards; design techniques to control hazards and safety verification techniques to verify these processes.

22 1. Nancy G. Leveson, 1995, Safeware, “System Safety and Computers”, Addison Wesley Publishing company 2.Koo, Seo Ryong, 2005, “An integrated environment of Software development and V&V for PLC Based Safety Critical Systems, Doctoral Thesis, KAIST. 3.Ian Sommerville, 2004, Critical Systems Engineering Nancy G. Leveson, 1986, Software Safety: Why, What, and How, Computing Surveys, Vol. 18, No. 2, ReferencesReferences