1. 1 Safety Assessment February 2006

2. 2 SAFETY ASSESSMENT A Safety Assessment is essentially a process for finding answers to three fundamental questions: What could go wrong? What would be the consequences? How often is it likely to occur? Once we know the answers this automatically raises the next question: Is this acceptable? What can we do if not?

3. 3 SAFETY ASSESSMENT Consequently, the objective of Safety Assessments is to: ensure that the system operates normally and without exposing unacceptable risks to anyone; reduce and prevent incidents and accidents and; limit the consequences of any occurrence that might occur. The Scope of the Safety Assessments includes: Safety Assessment on Air Navigation Systems covering people, procedures and equipment; … does not address Air Navigation System certification issues; … does not address organisational and management aspects related to safety assessment.

4. 4 SAFETY ASSESSMENT Safety A condition in which the risk of harm or damages is limited to an acceptable level Risk The probable rate of occurrence of a hazard causing harm and the degree of severity of the harm Risk = Severity * likelihood Need to define severity and likelihood Need to define acceptability

5. 5 SEVERITY CLASSIFICATION Severity Classification Scheme 1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) 2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). 3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres 4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation 5 No immediate effect on safety No immediate direct or indirect impact on operations

6. 6 LIKELIHOOD CLASSIFICATION Likelihood Classification Scheme 1 Frequently Likely to occur frequently (often ) 2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year ) 3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year ) 4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years ) 5 Improbable Very unlikely to occur (1 occurrence per 20 years) 6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years )

7. 7 RISK CLASSIFICATION Likelihood

8. 8 AS LOW AS REASONABLE PRACTICABLE The risk is less than the pre-determined unacceptable limit, The risk has been reduced to a level which is as low as reasonable practicable (ALARP) and The benefits of the proposed system or changes are sufficient to justify accepting the risk All three of the above criteria should be satisfied before a risk is classed as tolerable

9. 9 SAFETY ASSESSMENT ICAO SEVEN STEP APPROACH Hazard Identification and Estimation steps Step 1 – System and Environment Description Step 2 – Hazard Identification Step 3 – Hazard Severity Step 4 – Hazard Likelihood Mitigation steps Step 5 – Risk Evaluation Step 6 – Risk Mitigation Documentation Step 7 – Safety Assessment Documentation

10. 10 STEP 1 - DESCRIPTION Before a safety assessment can be performed, we need to describe the ATM system being assessed. For that purpose we need (as a minimum): System Description; Operational Environment Description.

11. 11 STEP 1 - DESCRIPTION A detailed system description should include: the purpose of the system; how the system will be used; a description of system functions; the system boundaries and the external interfaces; where appropriate, the transition procedures from the previous system to the new system, including any hazards associated with the decommissioning of the previous system; description of contingency procedures and other procedures for non- normal operations; other input such as other safety assessment results, occurrence and investigation reports, lessons learnt etc.; regulatory framework and applicable standards.

12. 12 STEP 1 - DESCRIPTION A detailed operational environment description should include: traffic characteristics; weather characteristics & weather-related factors (e.g. average frequency of diversions due to severe weather); topography; aircraft performance and equipment; infrastructure modes and limitations including e.g. runway in use, closed taxiways etc; environmental constraints; characteristics of the users of the system; adjacent centre capabilities; …and other input concerning the environment in which the system is to be operated.

13. 13 HAZARD IDENTIFICATION AND ESTIMATION PROCESS

14. 14 STEP 2 – HAZARD IDENTIFICATION Purpose …to identify what could go wrong! (- or anticipate problems before they occur…) ….to identify the consequences (on safety) of the hazards A hazard is defined as any condition, event or circumstances which could induce an accident or incident (ICAO DOC 9422) The equipment (hardware and software); The operating environment; The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.

15. 15 STEP 2 – HAZARD IDENTIFICATION …to identify the consequences of the hazards on operation! A hazard consequence is defined as the potential effects on operation that a hazard may create The operational consequences list the effects the hazard will have on the operation and emphasise the impact / changes the hazard will introduce compared with normal operation. The safety consequences are derived from the operational consequences by deciding the impact on the safe provision of ATS. E.g. potential loss of separation. - increased receive/transmit - increased co-ordination - increased receive/transmit - increased co-ordination - potential loss of separation

16. 16

17. 17

18. 18 STEP 2 – HAZARD IDENTIFICATION The hazard identification step should consider all the possible sources of system failure. Depending on the nature and size of the system under consideration these could include: The equipment (hardware and software); The operating environment (including physical conditions, airspace and air route design); The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.

19. 19 STEP 2 – HAZARD IDENTIFICATION Methodologies Brainstorming; Vision Conferences; Historical Records of Incidents; Checklists; Other systematic methods.

20. 20 STEP 2 – HAZARD IDENTIFICATION Preferred Methodology Brainstorming because: Easy and straightforward process. No need to complicate or make too academic! Such group sessions are usually good at generating ideas and identifying issues – mutual inspiration; The interactions between participants with varying experience and knowledge tend to lead to broader, more comprehensive and more balanced consideration of safety issues.

21. 21 STEP 2 – HAZARD IDENTIFICATION Brainstorming Process interactive session facilitated by a moderator experts encouraged to bring forward any safety-related issue they can think of based upon pre-developed scenarios first step: identify hazards second step: identify consequences of the hazards

22. 22 STEP 2 – HAZARD IDENTIFICATION Participants participants should be chosen for their expertise in fields relevant to the project being assessed. Such experts usually include System users/operational experts: ATCOs and Flight Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective; System technical experts, to explain the system purpose, interfaces and functions; Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.

23. 23 STEP 2 – HAZARD IDENTIFICATION EXAMPLE

24. 24 STEP 3 – SEVERITY ASSESSMENT The severity expresses the impact on operation or the harm an individual may suffer. Severity Classification is a gradation, ranging from "worst case/accident" to "no safety impact" – expressing the magnitude of the consequence of the hazard. Thus, a severity is allocated each hazard consequence in accordance with the agreed severity classification scheme.

25. 25 STEP 3 – SEVERITY ASSESSMENT Severity Classification Scheme 1 Accident One or more catastrophic accident One or more mid-air collision One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) 2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). 3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres 4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation 5 No immediate effect on safety No immediate direct or indirect impact on operations

26. 26 STEP 4 – LIKELIHOOD ASSESSMENT The likelihood of occurrence expresses how often the consequence of a hazard is likely to occur. Likelihood Classification is a gradation, ranging from "frequently" to extremely improbable". Thus, a likelihood is allocated each hazard consequence in accordance with the agreed likelihood classification scheme.

27. 27 STEP 4 – LIKELIHOOD ASSESSMENT Likelihood Classification Scheme 1 Frequently Likely to occur frequently (often ) 2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year ) 3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year ) 4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years ) 5 Improbable Very unlikely to occur (1 occurrence per 20 years) 6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years )

28. 28 STEP 3 & 4 – SEVERITY AND LIKELIHOOD EXAMPLE

29. 29 STEP 5 & 6 – RISK EVALUATION AND MITIGATION Is this risk acceptable? We have a risk with a defined likelihood and severity Acceptable risks No Yes Not acceptable risks One of the causes training of Discussion of causes and failures What are the potential causes could be insufficient This consequence prevented if How can we resolve it? Discussion of Risk Mitigation could be reduced or Risk Mitigation Plan Mitigation will remove risk Mitigation will not remove risk Residual risk acceptable? Risk mitigation impracticable? Mitigation impracticable Open risks Discussion of acceptability

30. 30 STEP 5 – RISK EVALUATION Determine what is / is not acceptable Acceptable level of Safety Determine acceptability of identified risks Clearly unacceptable Clearly acceptable May be / may be not acceptable likelihood

31. 31 STEP 5 – RISK EVALUATION Performed by a small group System users/operational experts: ATCOs and Flight Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective; System technical experts, to explain the system purpose, interfaces and functions; Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards. May need to be extended with specialists in areas relevant for the ALARP assessment

32. 32 STEP 5 – RISK EVALUATION EXAMPLE

33. 33 STEP 6 – RISK MITIGATION Identify potential causes for a risk to occur Some causes are identified during the hazard identification Ensure that we have identified all causes Identify potential mitigation Remove the risk (remove the cause of the risk) Reduce the risk Reduce severity and/or probability Identify preferred mitigation approach

34. 34 likelihood STEP 6 – RISK MITIGATION

35. 35 STEP 6 – RISK MITIGATION Risk mitigation should be sought in any of the three components of a system: People Procedures Equipment The possible approaches to risk mitigation include: revision of the system (or airport) design; modification of operational procedures; changes to staffing arrangements; and training of personnel to deal with the hazard.

36. 36 STEP 6 – RISK MITIGATION To identify causes a number of techniques may be required Brainstorming sessions Fault tree analysis - Effect tree analysis Common cause failure identification (Single point failure) Task, Fail-Safe & Error Tolerance Analysis Failure Mode and Criticality Analysis Reliability, Availability and Maintainability Analysis Focus on components giving: Highest likelihood Highest degree of severity

37. 37 STEP 6 – RISK MITIGATION Performed by a small group System users/operational experts System technical experts Safety and human factors experts Different experts may be required to: Performed detailed studies of the causes of a risk Study system design to determine component potentially causing, e.g. loss of air situation display Study procedures to determine where e.g. misunderstandings can arise Ways to remove those causes

38. 38 STEP 6 – RISK MITIGATION SW Hazard S F S S F F Effect 1 Effect 2 Effect 3 Effect 4 P=Likelihood E = Severity PR P=Likelihood Failure Recovery Fault Tree and Effect Tree Analysis

39. 39 STEP 6 – RISK MITIGATION Procedure Assurance Level Procedure development effort should be proportional to the potential Risk associated with the Procedure. To achieve this, objective PAL should be determined and satisfied. PAL is setting some objectives to be met during the different phases of the procedure life cycle – Table 1. PAL objectives are applicable to the entire Procedure, not only to some part of it.

40. 40 STEP 6 – RISK MITIGATION LevelDefinitionDesign and validationImplementationTransfer in operations Operations 3 Other/own experience benchmarking Specification quality assurance Fast time simulation Qualitative risk assessment Pre-implementation trials Dedicated training Staff acceptance argumentation Quality assurance of implementation Competency argument for the staff to perform transfer Contingency plan Regular proficiency checks 4 Other/own experience benchmarking Specification quality assurance Fast time simulation Qualitative risk assessment Pre-implementation trials Quality assurance of implementation Contingency plan Regular proficiency checks Procedure Assurance Level

41. 41 STEP 6 – RISK MITIGATION Software Assurance Level Software development effort should be proportional to the potential Risk associated with the Software. To achieve this, objective SWAL should be determined and satisfied. SWAL is setting some objectives to be met during the different phases of the software life cycle. SWAL objectives are applicable to the software component is question (only some part of of the total software).

42. 42 STEP 6 – RISK MITIGATION Level Requirement 1234 37.3Unit, integration and system testing 37.3.1Unit and integration tests shall be conducted on individual units and on partially integrated units to demonstrate that the software is executable and that it produces the expected results for the specified test cases. MMMM 37.3.3Integration tests shall as a minimum demonstrate the correctness of all interfaces. J1J2MM M Mandatory requirement to the development process J1 Justification is to be provided if the clause or part of the clause is not followed J2 Justification for the omission or non-compliance is to be provided Extract from DEF-STAN-55

43. 43 STEP 6 – RISK MITIGATION Mitigation actions (safety requirements) should be carefully analysed: Will the mitigation remove the risk or reduce the risk (what will be remaining risk be) Will the implementation introduce any new hazards (repeat step 3, 4 and 5) Mitigation actions shall be documented Risk Mitigation Plan

44. 44 STEP 6 – RISK MITIGATION EXAMPLE

45. 45 STEP 7 - SAFETY ASSESSMENT DOCUMENTATION The purpose: To provide a permanent record of the final result of the safety assessment To provide the arguments and evidence demonstrating that the risks associated with the implementation of the proposed system or change: have been eliminated, or have been adequately controlled and reduced to a tolerable level.

46. 46 STEP 7 - SAFETY ASSESSMENT DOCUMENTATION Should contain a summary of: Methods used Safety criteria (the agreed safety levels) Results of the hazard identification process (including Hazard Logs) Risk mitigation required (safety requirements) Follow-up actions Evidence of compliance with safety requirements References should be included Evidence of validity of assumptions

47. 47 DIFFICULTIES – SAFETY ASSESSMENT General Complex, resource-demanding activity Target Levels of Safety (Severity and Likelihood) Complexity No guidelines or recommendation – in most cases not even statistics No guidelines to apportioning Safety Targets to lower levels No guidelines to who does what (Regulator Provider Supplier)

48. 48 DIFFICULTIES – SAFETY ASSESSMENT Risk Mitigation Very demanding concepts (software assurance levels, procedure assurance levels) Very demanding activities for risk mitigation Analyses required beyond reach for many organisation

49. 49 RECOMMENDATIONS Start with low level of ambition Even simple Safety Assessment provides quite efficient risk mitigation Introduce more advanced features once the simple version works Start with quantitative likelihood classification while data are collected to establish qualitative figures Make sure assumptions are well-defined and traced

50. 50 RECOMMENDATIONS Dont forget to design a follow-up system for (ICAO 2.26.5) Hazards (likelihood for different causes) Assumptions, e.g.: Capacity figures Reliability figures Should be extracted from the reporting system

51. 51 SUPPORTING SLIDES

52. 52 Target Level of Safety METNAV/EnrNAV/TermGroundTWRAPPACC Safety factor for Accidents (1,55 10 -8 per Flight hour) Mid-air collision ÷ Controlled flight into terrain ÷ Accident on ground with fatalities ÷ ÷÷ …… Safety Factors for Serious Incidents Separation minima infringement (less than 50%) ÷ Runway incursion with avoiding action ÷ ÷÷ ……