1. Integra Consult A/S Safety Assessment Karachi, January 2006

2. Integra Consult A/S SAFETY ASSESSMENT A Safety Assessment is essentially a process for finding answers to three fundamental questions: –What could go wrong? –What would be the consequences? –How often is it likely to occur? Once we know the answers this automatically raises the next question: Once we know the answers this automatically raises the next question: –Is this acceptable? –What can we do if not?

3. Integra Consult A/S SAFETY ASSESSMENT Consequently, the objective of Safety Assessments is to: Consequently, the objective of Safety Assessments is to: –ensure that the system operates normally and without exposing unacceptable risks to anyone; –reduce and prevent incidents and accidents and; –limit the consequences of any occurrence that might occur. The Scope of the Safety Assessments includes: The Scope of the Safety Assessments includes: –Safety Assessment on Air Navigation Systems covering people, procedures and equipment; –… does not address Air Navigation System certification issues; –… does not address organisational and management aspects related to safety assessment.

4. Integra Consult A/S SAFETY ASSESSMENT Safety Safety –A condition in which the risk of harm or damages is limited to an acceptable level Risk Risk –The probable rate of occurrence of a hazard causing harm and the degree of severity of the harm –Risk = Severity * likelihood –Need to define severity and likelihood –Need to define acceptability

5. Integra Consult A/S SEVERITY CLASSIFICATION Severity Classification Scheme 1 Accident One or more catastrophic accident One or more catastrophic accident One or more mid-air collision One or more mid-air collision One of more collisions on ground between two aircraft One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) 2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). 3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres 4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation 5 No immediate effect on safety No immediate direct or indirect impact on operations No immediate direct or indirect impact on operations

6. Integra Consult A/S LIKELIHOOD CLASSIFICATION Likelihood Classification Scheme 1 Frequently Likely to occur frequently (often ) 2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year ) 3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year ) 4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years ) 5 Improbable Very unlikely to occur (1 occurrence per 20 years) 6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years )

7. Integra Consult A/S RISK CLASSIFICATION Likelihood

8. Integra Consult A/S AS LOW AS REASONABLE PRACTICABLE The risk is less than the pre-determined unacceptable limit, The risk is less than the pre-determined unacceptable limit, the risk has been reduced to a level which is as low as reasonable practicable (ALARP) and the risk has been reduced to a level which is as low as reasonable practicable (ALARP) and the benefits of the proposed system or changes are sufficient to justify accepting the risk the benefits of the proposed system or changes are sufficient to justify accepting the risk All three of the above criteria should be satisfied before a risk is classed as tolerable

9. Integra Consult A/S SAFETY ASSESSMENT ICAO SEVEN STEP APPROACH Hazard Identification and Estimation steps Hazard Identification and Estimation steps –Step 1 – System and Environment Description –Step 2 – Hazard Identification –Step 3 – Hazard Severity –Step 4 – Hazard Likelihood Mitigation steps Mitigation steps –Step 5 – Risk Evaluation –Step 6 – Risk Mitigation Documentation Documentation –Step 7 – Safety Assessment Documentation

10. Integra Consult A/S STEP 1 - DESCRIPTION Before a safety assessment can be performed, we need to describe the ATM system being assessed. For that purpose we need (as a minimum): Before a safety assessment can be performed, we need to describe the ATM system being assessed. For that purpose we need (as a minimum): –System Description; –Operational Environment Description.

11. Integra Consult A/S STEP 1 - DESCRIPTION A detailed system description should include: A detailed system description should include: –the purpose of the system; –how the system will be used; –a description of system functions; –the system boundaries and the external interfaces; –where appropriate, the transition procedures from the previous system to the new system, including any hazards associated with the decommissioning of the previous system; –description of contingency procedures and other procedures for non-normal operations; –other input such as other safety assessment results, occurrence and investigation reports, lessons learnt etc.; –regulatory framework and applicable standards.

12. Integra Consult A/S STEP 1 - DESCRIPTION A detailed operational environment description should include: A detailed operational environment description should include: –traffic characteristics; –weather characteristics & weather-related factors (e.g. average frequency of diversions due to severe weather); –topography; –aircraft performance and equipment; –infrastructure modes and limitations including e.g. runway in use, closed taxiways etc; –environmental constraints; –characteristics of the users of the system; –adjacent centre capabilities; –…and other input concerning the environment in which the system is to be operated.

13. Integra Consult A/S HAZARD IDENTIFICATION AND ESTIMATION PROCESS

14. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION Purpose Purpose –…to identify what could go wrong! (- or anticipate problems before they occur…) –….to identify the consequences (on safety) of the hazards A hazard is defined as any condition, event or circumstances which could induce an accident or incident (ICAO DOC 9422) The equipment (hardware and software); The operating environment; The human operators; The human machine interface (HMI); Operational procedures; Maintenance procedures; External services.

15. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION …to identify the consequences of the hazards on operation! …to identify the consequences of the hazards on operation! – –A hazard consequence is defined as the potential effects on operation that a hazard may create The operational consequences list the effects the hazard will have on the operation and emphasise the impact / changes the hazard will introduce compared with normal operation. The operational consequences list the effects the hazard will have on the operation and emphasise the impact / changes the hazard will introduce compared with normal operation. The safety consequences are derived from the operational consequences by deciding the impact on the safe provision of ATS. E.g. potential loss of separation. The safety consequences are derived from the operational consequences by deciding the impact on the safe provision of ATS. E.g. potential loss of separation. - increased receive/transmit - increased co-ordination - increased receive/transmit - increased co-ordination - potential loss of separation

16. Integra Consult A/S

18. STEP 2 – HAZARD IDENTIFICATION The hazard identification step should consider all the possible sources of system failure. Depending on the nature and size of the system under consideration these could include: The hazard identification step should consider all the possible sources of system failure. Depending on the nature and size of the system under consideration these could include: –The equipment (hardware and software); –The operating environment (including physical conditions, airspace and air route design); –The human operators; –The human machine interface (HMI); –Operational procedures; –Maintenance procedures; –External services.

19. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION Methodologies Methodologies –Brainstorming; –Vision Conferences; –Historical Records of Incidents; –Checklists; –Other systematic methods.

20. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION Preferred Methodology Preferred Methodology –Brainstorming because: Easy and straightforward process. No need to complicate or make too academic! Easy and straightforward process. No need to complicate or make too academic! Such group sessions are usually good at generating ideas and identifying issues – mutual inspiration; Such group sessions are usually good at generating ideas and identifying issues – mutual inspiration; The interactions between participants with varying experience and knowledge tend to lead to broader, more comprehensive and more balanced consideration of safety issues. The interactions between participants with varying experience and knowledge tend to lead to broader, more comprehensive and more balanced consideration of safety issues.

21. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION Brainstorming Process Brainstorming Process –interactive session –facilitated by a moderator –experts encouraged to bring forward any safety-related issue they can think of –based upon pre-developed scenarios –first step: identify hazards –second step: identify consequences of the hazards

22. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION Participants Participants –participants should be chosen for their expertise in fields relevant to the project being assessed. Such experts usually include Such experts usually include –System users/operational experts: ATCOs and Flight Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective; –System technical experts, to explain the system purpose, interfaces and functions; –Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards.

23. Integra Consult A/S STEP 2 – HAZARD IDENTIFICATION EXAMPLE

24. Integra Consult A/S STEP 3 – SEVERITY ASSESSMENT The severity expresses the impact on operation or the harm an individual may suffer. The severity expresses the impact on operation or the harm an individual may suffer. Severity Classification is a gradation, ranging from "worst case/accident" to "no safety impact" – expressing the magnitude of the consequence of the hazard. Severity Classification is a gradation, ranging from "worst case/accident" to "no safety impact" – expressing the magnitude of the consequence of the hazard. Thus, a severity is allocated each hazard consequence in accordance with the agreed severity classification scheme. Thus, a severity is allocated each hazard consequence in accordance with the agreed severity classification scheme.

25. Integra Consult A/S STEP 3 – SEVERITY ASSESSMENT Severity Classification Scheme 1 Accident One or more catastrophic accident One or more catastrophic accident One or more mid-air collision One or more mid-air collision One of more collisions on ground between two aircraft One of more collisions on ground between two aircraft No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) No independent source of recovery mechanism, such as surveillance or ATC / Flight Crew procedure, can reasonably be expected to prevent the accident(s) 2 Serious Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). one or more aircraft deviating from their intended clearance, so that abrupt manoeuvre is required to avoid collision with another aircraft or with terrain (or when an avoidance action would be appropriate). 3 Major Incident large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. large reduction in separation (e.g. a separation of less than half the separation minima), without crew or ATC fully controlling the situation or able to recover from the situation. Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation, jeopardising the ability to recover without use of collision or terrain avoidance manoeuvres 4 Significant Incident Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Increased workload on ATCO or Flight Crew or slightly degrading capability of the CSN system Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation Minor reduction in separation (e.g. a separation of more than half the separation minima), without crew or ATC fully controlling the situation, or able to recover from the situation and fully able to recover the situation 5 No immediate effect on safety No immediate direct or indirect impact on operations No immediate direct or indirect impact on operations

26. Integra Consult A/S STEP 4 – LIKELIHOOD ASSESSMENT The likelihood of occurrence expresses how often the consequence of a hazard is likely to occur. Likelihood Classification is a gradation, ranging from "frequently" to extremely improbable". Thus, a likelihood is allocated each hazard consequence in accordance with the agreed likelihood classification scheme.

27. Integra Consult A/S STEP 4 – LIKELIHOOD ASSESSMENT Likelihood Classification Scheme 1 Frequently Likely to occur frequently (often ) 2 Probable Likely to occur several times during the life-time of the system (2-5 occurrences per year ) 3 Occasional Occurs sometimes during the life-time of the system (1 occurrence per year ) 4 Remote Unlikely to occur sometimes during the life-time of the system (1 occurrence per 5 years ) 5 Improbable Very unlikely to occur (1 occurrence per 20 years) 6 Extremely Improbable Extremely unlikely to occur (1 occurrence per 100 years )

28. Integra Consult A/S STEP 3 & 4 – SEVERITY AND LIKELIHOOD EXAMPLE

29. Integra Consult A/S STEP 5 & 6 – RISK EVALUATION AND MITIGATION Is this risk acceptable? We have a risk with a defined likelihood and severity Acceptable risks No Yes Not acceptable risks One of the causes training of Discussion of causes and failures What are the potential causes could be insufficient This consequence prevented if How can we resolve it? Discussion of Risk Mitigation could be reduced or Risk Mitigation Plan Mitigation will remove risk Mitigation will not remove risk Residual risk acceptable? Risk mitigation impracticable? Mitigation impracticable Open risks Discussion of acceptability

30. Integra Consult A/S STEP 5 – RISK EVALUATION Determine what is / is not acceptable Determine what is / is not acceptable –Acceptable level of Safety Determine acceptability of identified risks Determine acceptability of identified risks –Clearly unacceptable –Clearly acceptable –May be / may be not acceptable likelihood

31. Integra Consult A/S STEP 5 – RISK EVALUATION Performed by a small group Performed by a small group –System users/operational experts: ATCOs and Flight Crew (where necessary), to assess the consequences of hazard(s) from an operational perspective; –System technical experts, to explain the system purpose, interfaces and functions; –Safety and human factors experts, to guide in the application of the FHA methodology itself and to bring wider experience of the consequences of hazards. May need to be extended with specialists in areas relevant for the ALARP assessment May need to be extended with specialists in areas relevant for the ALARP assessment

32. Integra Consult A/S STEP 5 – RISK EVALUATION EXAMPLE

33. Integra Consult A/S STEP 6 – RISK MITIGATION Identify potential causes for a risk to occur Identify potential causes for a risk to occur –Some causes are identified during the hazard identification –Ensure that we have identified all causes Identify potential mitigation Identify potential mitigation –Remove the risk (remove the cause of the risk) –Reduce the risk Reduce severity and/or probability Reduce severity and/or probability Identify preferred mitigation approach Identify preferred mitigation approach

34. Integra Consult A/S likelihood STEP 6 – RISK MITIGATION

35. Integra Consult A/S STEP 6 – RISK MITIGATION Risk mitigation should be sought in any of the three components of a system: Risk mitigation should be sought in any of the three components of a system: –People –Procedures –Equipment The possible approaches to risk mitigation include: The possible approaches to risk mitigation include: –revision of the system (or airport) design; –modification of operational procedures; –changes to staffing arrangements; and –training of personnel to deal with the hazard.

36. Integra Consult A/S STEP 6 – RISK MITIGATION To identify causes a number of techniques may be required To identify causes a number of techniques may be required –Brainstorming sessions –Fault tree analysis - Effect tree analysis –Common cause failure identification (Single point failure) –Task, Fail-Safe & Error Tolerance Analysis –Failure Mode and Criticality Analysis –Reliability, Availability and Maintainability Analysis Focus on components giving: Focus on components giving: –Highest likelihood –Highest degree of severity

37. Integra Consult A/S STEP 6 – RISK MITIGATION Performed by a small group Performed by a small group –System users/operational experts –System technical experts –Safety and human factors experts Different experts may be required to: Different experts may be required to: –Performed detailed studies of the causes of a risk Study system design to determine component potentially causing, e.g. loss of air situation display Study system design to determine component potentially causing, e.g. loss of air situation display Study procedures to determine where e.g. misunderstandings can arise Study procedures to determine where e.g. misunderstandings can arise Ways to remove those causes Ways to remove those causes

38. Integra Consult A/S STEP 6 – RISK MITIGATION SW Hazard S F S S F F Effect 1 Effect 2 Effect 3 Effect 4 P=Likelihood E = Severity PR P=Likelihood Failure Recovery Fault Tree and Effect Tree Analysis

39. Integra Consult A/S STEP 6 – RISK MITIGATION Procedure Assurance Level Procedure Assurance Level – Procedure development effort should be proportional to the potential Risk associated with the Procedure. To achieve this, objective PAL should be determined and satisfied. – PAL is setting some objectives to be met during the different phases of the procedure life cycle – Table 1. – PAL objectives are applicable to the entire Procedure, not only to some part of it.

40. Integra Consult A/S STEP 6 – RISK MITIGATION LevelDefinition Design and validation Implementation Transfer in operations Operations 3 Other/own experience benchmarking Other/own experience benchmarking Specification quality assurance Specification quality assurance Fast time simulation Fast time simulation Qualitative risk assessment Qualitative risk assessment Pre- implementation trials Pre- implementation trials Dedicated training Dedicated training Staff acceptance argumentation Staff acceptance argumentation Quality assurance of implementation Quality assurance of implementation Competency argument for the staff to perform transfer Competency argument for the staff to perform transfer Contingency plan Contingency plan Regular proficiency checks Regular proficiency checks 4 Other/own experience benchmarking Other/own experience benchmarking Specification quality assurance Specification quality assurance Fast time simulation Fast time simulation Qualitative risk assessment Qualitative risk assessment Pre- implementation trials Pre- implementation trials Quality assurance of implementation Quality assurance of implementation Contingency plan Contingency plan Regular proficiency checks Regular proficiency checks Procedure Assurance Level

41. Integra Consult A/S STEP 6 – RISK MITIGATION Software Assurance Level Software Assurance Level – Software development effort should be proportional to the potential Risk associated with the Software. To achieve this, objective SWAL should be determined and satisfied. – SWAL is setting some objectives to be met during the different phases of the software life cycle. – SWAL objectives are applicable to the software component is question (only some part of of the total software).

42. Integra Consult A/S STEP 6 – RISK MITIGATION LevelRequirement1234 37.3 Unit, integration and system testing 37.3.1 Unit and integration tests shall be conducted on individual units and on partially integrated units to demonstrate that the software is executable and that it produces the expected results for the specified test cases. MMMM 37.3.3 Integration tests shall as a minimum demonstrate the correctness of all interfaces. J1J2MM M Mandatory requirement to the development process J1 Justification is to be provided if the clause or part of the clause is not followed J2 Justification for the omission or non-compliance is to be provided Extract from DEF-STAN-55

43. Integra Consult A/S STEP 6 – RISK MITIGATION Mitigation actions (safety requirements) should be carefully analysed: Mitigation actions (safety requirements) should be carefully analysed: –Will the mitigation remove the risk or reduce the risk (what will be remaining risk be) –Will the implementation introduce any new hazards (repeat step 3, 4 and 5) Mitigation actions shall be documented Mitigation actions shall be documented –Risk Mitigation Plan

44. Integra Consult A/S STEP 6 – RISK MITIGATION EXAMPLE

45. Integra Consult A/S STEP 7 - SAFETY ASSESSMENT DOCUMENTATION The purpose: The purpose: –To provide a permanent record of the final result of the safety assessment –To provide the arguments and evidence demonstrating that the risks associated with the implementation of the proposed system or change: have been eliminated, or have been eliminated, or have been adequately controlled and reduced to a tolerable level. have been adequately controlled and reduced to a tolerable level.

46. Integra Consult A/S STEP 7 - SAFETY ASSESSMENT DOCUMENTATION Should contain a summary of: Should contain a summary of: – Methods used – Safety criteria (the agreed safety levels) – Results of the hazard identification process (including Hazard Logs) – Risk mitigation required (safety requirements) – Follow-up actions – Evidence of compliance with safety requirements References should be included References should be included –Evidence of validity of assumptions

47. Integra Consult A/S DIFFICULTIES – SAFETY ASSESSMENT General General –Complex, resource-demanding activity Target Levels of Safety (Severity and Likelihood) Target Levels of Safety (Severity and Likelihood) –Complexity –No guidelines or recommendation – in most cases not even statistics –No guidelines to apportioning Safety Targets to lower levels –No guidelines to who does what (Regulator Provider Supplier)

48. Integra Consult A/S DIFFICULTIES – SAFETY ASSESSMENT Risk Mitigation Risk Mitigation –Very demanding concepts (software assurance levels, procedure assurance levels) –Very demanding activities for risk mitigation –Analyses required beyond reach for many organisation

49. Integra Consult A/S RECOMMENDATIONS Start with low level of ambition Start with low level of ambition –Even simple Safety Assessment provides quite efficient risk mitigation –Introduce more advanced features once the simple version works –Start with quantitative likelihood classification while data are collected to establish qualitative figures Make sure assumptions are well-defined and traced Make sure assumptions are well-defined and traced

50. Integra Consult A/S RECOMMENDATIONS Dont forget to design a follow-up system for (ICAO 2.26.5) Dont forget to design a follow-up system for (ICAO 2.26.5) –Hazards (likelihood for different causes) –Assumptions, e.g.: Capacity figures Capacity figures Reliability figures Reliability figures –Should be extracted from the reporting system

51. Integra Consult A/S SUPPORTING SLIDES

52. Integra Consult A/S Target Level of Safety METNAV/Enr NAV/Ter m GroundTWRAPPACC Safety factor for Accidents (1,55 10 -8 per Flight hour) Mid-air collision ÷ Controlled flight into terrain ÷ Accident on ground with fatalities ÷÷÷ …… Safety Factors for Serious Incidents Separation minima infringement (less than 50%) ÷ Runway incursion with avoiding action ÷÷÷ ……