1. System Certification (Safety Assurance) of WAAS
Deane Bunce
SBAS Approval Workshop
21-22 June 2005

2. System Certification Verify Performance Requirements Met
Accuracy
Integrity
Continuity
Availability
GAO Report of 2000
FAA Underestimated Complexity of Proving the Integrity Requirement Satisfied
FAA Did Not Closely Monitor the Contractor’s Efforts to Demonstrate Integrity
Recommendations
Establish Checkpoints To Determine Contractor’s Approach For Meeting Performance Requirements
Have External Organization Evaluate Agency Progress At These Checkpoints
WAAS is a Complex System whose Safety Cannot be Demonstrated Solely by Test
Failures in WAAS can Result in Hazards to Users
Must Verify System Design Mitigates Hazards or Reduces to Acceptable Risk

3. Approach for Assuring WAAS Integrity
Elements to Consider
WAAS Hazard Assessment
WAAS Architecture
Need for Technical Oversight (Independent Verification)

4. WAAS Hazard Assessment
Category
Severity
Causes
Probability Limit
HMI
Hazardous/ Severe-Major (Cat 1 PA)
Major (ER/NPA)
Failure to Detect Broadcast of Misleading Information (MI)
Failure to Alarm Detected Broadcast of MI Within the Specified TTA
1E-07
LOC
Minor
Hardware/Software Failure Resulting in Loss of Signal in Space
Corrections or Monitor Trip Resulting in Transition or IGP to NM or DU State
Detection of MI Resulting in User Alarm
x 10-5 per approach
Personnel
Minor to Catastrophic (per MIL-STD-882D)
Miscellaneous (Electrical, Hazardous Materials, Physical Hazards)
N/A

5. Corrections Processor
WAAS Architecture
Satellite Signals
Corrections Processor
Safety Processor
Receiver
Uplink/GEO
User
Monitors Data
(Level B)
Generates data (Level D)
CRC protects data
Correct – Then – Monitor Architecture
Trusted Processing for HMI Risk Limited to Safety Processors
Data Verification and Integrity Monitoring Algorithms Designed to Detect and/or Bound Sources of Data Errors
Must Verify System Architecture/Design Mitigates HMI Hazards or Reduces to Acceptable Risk
Safe Behavior Cannot Be Verified Completely Through Testing for Complex/Non-Deterministic Algorithms
System Integrity Performance Is Not Completely Quantifiable
Process-Based Assurance Method Required to Assure all Threats to Integrity were Addressed

6. Process Based Assurance Method
Safety Assurance: Assure that WAAS Design is capable of Meeting System Safety Requirements (Integrity and Continuity)
Method of Assurance Based on Aircraft Certification Standards / Prior Navaid Applications
SAE ARP 4754/4761, RTCA DO-178B
Consists of Audits/Reviews of Contractor System Development and Safety Assessment Activities
Process Captured in Safety Assurance Process Requirements Document [AVR/ATS]

7. Safety Assurance Process
Assurance Process: The Planned and Systematic Actions Necessary to Provide Adequate Confidence That a Product or Process Satisfies Given Requirements
Safety
Standards
Mission Need Statement
Safety
Assurance
MIL-STD-882
DO-178B
Program
Requirements
Safety
Assurance
Requirements
AVR
System
Development
Plan
Process Input
ARP 4761/4754
System Safety Program Plan
Assurance
Guidance
Integrity Threats
ARP 4754
System
Development
Safety
Assessment
ARP 4761
Requirements
Definition and
Maintenance
System
Design
Design
Implementation
and
Integration
Verification
Hazard
Analysis
and
Identification
Requirements,
Design, and
Code
Review
Verification
Review
Design
Requirements, Design , and Implementation
Verification Methods and Results

8. System Development Activities
System and Component Development Activities to Assure Flow-down of Safety Requirements into the WAAS Implementation
System Engineering, Specification, and Architecture Design
Component Design Specification
Implementation and/or Procurement
Engineering Processes of Quality Assurance (QA), Configuration Management (CM) and Verification for All Development Activities

9. Safety Assessment Activities
System Hazard Assessment
Fault Tree Analysis
Algorithm P(HMI) Analysis
Safety Processor Input Analysis
Safety Directed Analysis
Exposure Time/Latency Analysis
Failure Modes and Effect Analysis (FMEA)
Common Cause Analysis
Safety Risk Assessment – HMI Hazards Sufficiently Controlled

10. Safety Assurance Process Requirements
Plans and Standards
System and Component Specifications, Requirements, and Design Capture
System Safety Assessments
Component Implementation
System Integration
Acceptance
Evidence Required
Artifacts
Audits/
Reviews
Vendor
Vendor/FAA
Baseline
Reports/
Deliverables

11. Assurance Approval Process
Safety
Assurance Complete
Safety Design
Approval
WIPP
Approval
Safety Assessments
& Requirements
Verification
Establish Baseline of
Safety Artifacts
Monitor Development
& Validation Complete

12. Assurance Approval
WAAS Program Office Establishes a WAAS Integrity Performance Panel (WIPP)
Role is to Establish Technical Approach and Solution for Meeting Integrity
WIPP Comprised of WAAS Experts … (FAA, Academia, Industry)
WIPP Meetings with Contractor Held Monthly to Review and Evaluate Contractor’s Progress

13. WIPP Tasks Target Level of Safety Algorithm Contribution to HMI
Coding
Operating System
Computer HW
etc.
Algorithm
Contribution to HMI
A Great Stew of Feared Events Each Requiring:
Threat Model
Detection Algorithm
P(HMI) Analysis!
SV 19
Level D SW
WRS Clocks
Iono
Tropo
L1/L2
Biases
WRS RFI,
Multipath
Cycle Slips
GPS
Ephemeris

14. Define Analysis Methodology
WIPP Role
Algorithm Development
Algorithm
Definition
Document
(ADD)
Coding Spec
HMI Plan &
Prelim Results
Threat
Model
Monitor
Design &
Trades
WIPP
Approval
WIPP
Approval
WIPP Guidance
Evaluate
with Field
Data +
Threat
Prototype
& Validate
Operational
Software
Sys.
Integ.
Ready for
Test
Provability
Assessment
HMI Analysis
WIPP
Approval
WIPP
Approval
WIPP Guidance
Collect
Data/ Develop
Tools
Define Analysis Methodology
Data-
Driven
Analysis
HMI Analysis
Report
Prelim.
Eval.
Theoretical
Analysis 3 1

15. WAAS Design Assurance
WAAS Design Assurance can be Broken down into Three Major Areas
Software Assurance
Hardware Assurance
Algorithm (Monitor) Assurance
Assurance that Hazards were Sufficiently Mitigated was Evaluated by Means of a Fault-Tree Analysis (FTA)
Events in Fault Tree Broken Down by Hardware, Software and Algorithm (Monitor) Contributions

16. Software Assurance Approach
Assurance Level Requirement Assigned According to RTCA/DO-178B Guidelines
Assigned According to the Hazard Severity Caused by Software Failure Based on the WAAS FTA
HMI – Assurance Level B (Where Software Failure is a Single Point Cause of HMI Without Mitigation or Control; or Software Function Provides Mitigation/Control of an HMI Failure Condition)
LOC – Minimum Assurance Level D (Where Software Failure is a Single Point Cause of LOC Without Mitigation or Control)
Software PHMI Contribution is Assigned Based on Assurance Level Achieved
HMI – Assurance Level B Software is Assigned a Failure Probability of Zero (0); Assurance Level D Software is Assigned a Failure Probability of One (1)
LOC – Assurance Level B or D Software is Assigned a Failure Probability of Zero (0)
Monitor Algorithms Failure to Detect Probability is Assessed Based on Probabilistic Analysis

17. Software Assurance – Alternate Means
Some Level D Functions Required Assurance
Developed a Process called Safety Directed Analysis (SDA)
Software Tested and Analyzed to Assure Output Always What is Expected
Result was Assignment of a Zero Probability for Failure in Fault Tree
Process and Results were Coordinated with FAA National Resource Specialist (NRS) for Software

18. Hardware Assurance
Assurance Completed at a Functional Level by means of a Failure Modes and Effects Analysis
Failure Rates of Hardware Utilized were Based on Several Years of Empirical Data
Exposure Times for Failures Captured as Part of Latency Analysis and Rolled into the Fault-Tree
Un-trusted Communications Assured by Means of Cyclical Redundancy Checks (CRCs)

19. Algorithm Assurance
Algorithms Developed to Detect Both Internal and External Faults
Internal Faults:
Data Provided to Monitors from Un-trusted Components was Assumed to be Corrupted
Required Analysis of all Inputs to the Safety Processor
Analysis Approach called Safety Processor Input Analysis (SPIA) Considered Corrupted Data, Absent Data and Data Timeliness and the Algorithms (Monitors) ability to Detect Such Conditions
External Faults or Environmental Conditions Evaluated by Means of an HMI Analysis Methodology

20. COTS Use on WAAS Applied to Level D (or Lower) WAAS Functions
IBM AIX OS
Network Management Software
Compilers, CM Tools
Required to Meet All Applicable RTCA/DO-178B Objectives
Assured in Accordance with Allocated Functional Requirements Within the WAAS Application

21. WAAS Network FTA
Any combination of active data at user antenna causes MI and WAAS FTD within TTA
LNAV-001
WAAS GEO broadcasts a calculated underbound UDRE/GIVE leading to MI
WAAS GEO broadcasts TSWM that got corrupted between WMS CMP and GUS WMP
WAAS GEO broadcasts TSWM that got corrupted after Tandem CRC decoding
WAAS GEO broadcasts TSWM that got corrupted after 24-bit CRC encoding and SLB FTD
WAAS GEO broadcasts ranging signal that is MI
WAAS GEO broadcasts valid UDRE that becomes MI
LNAV-100
LNAV-200
LNAV-300
LNAV-400
LNAV-500
LNAV-700

22. WAAS Network FTA GPS Satellite GEO Satellite 32 BIT CRC 24 BIT CRC
LNAV-700
RFU
SGS (see figure D for details)
WSG
WMP2
WMP1
GUSP
GUS
TCN
ANH-
ROUTER
BCN
TCS
ETHERNET
HUB
SWITCH
CP1
COMPARATOR
SP1
SP2
CP2
WMS (see figure C for details)
WRS x 25 (see figure A1 for details)
DCP
WRE-A (see figure A2 for details)
WAAS RECEIVER
GPS ANTENNA
WRE-B
32 BIT CRC
24 BIT CRC
GEO Satellite
GPS Satellite
LNAV-500
LNAV-100
LNAV-200
LNAV-300
LNAV-400

23. WAAS Network FTA – Data Transmission
WAAS GEO sent TSWM that got corrupted after 24-bit CRC encoding
WAAS GEO broadcasts TSWM that got corrupted after 24-bit CRC encoding and SLB FTD
WMPs short loopback (SLB) FTD TSWM that got corrupted after 24-bit CRC encoding
LNAV-401
LNAV-400
LNAV-402
G= 7.95E-23
G= 2.12E-04
G= 3.75E-19

24. WAAS Network FTA – SW Failure
Bias error exists and Range Domain Monitor (RDM) FTD
LNAV-114
G= 4.50E-08
Bias error exists leading to MI
RDM FTD bias errors
LNAV-192
LNAV-193
G= 1.00E+00
G= 4.50E-08
Note: LNAV-192 in this example is the result of any one of satellite L1-L2 bias error, receiver L1-L2 bias error, or receiver clock bias error. Since the error can result from untrusted software or hardware in any of these sources, the worst-case probability is used.

25. Safety Assurance Process Compliance Verification
FAA and Contractor Assure System Safety Program Plans (SSPPs) Address Safety Assurance Process Requirements
Contractor SSPP
Captures When and How Requirements are Satisfied
FAA SSPP
Schedule of Reviews Contingent on Contractor Plan
Establish Audit/Review Teams and Define Roles/Responsibilities
Seek Plan Approval
Safety Assurance Audit and Review Activity
Contractor Coordinates Artifact Schedule and Availability
FAA Verifies Artifact Compliance with SAPR
Completion of Activity Results in Safety Artifact Baseline

26. Safety Assurance Documentation Baseline
Established to Provide Evidence of Compliance with the WAAS Safety Assurance Process Requirements
Includes:
Program Planning and Standards Definition
System Specification, Requirements Definition, and Design
Component Specification, Requirements Definition and Design
System Safety Assessment
Component Implementation
System Integration
Configuration Management
Quality Assurance
Safety Assurance Process Compliance

27. Program Plans and Standards
System Engineering: System Engineering Management Plan, Reliability Program Plan
Software Engineering: Software Development Plan, PSAC, SAS and Supporting Process and Standards Notebooks
System Test: Master Test Plan, Development Test Plan, Production Acceptance Test Plan
System Safety: System Safety Program Plan, Supporting Process/Procedure Notebooks, Audit Records

28. System Specification and Design
System Safety Architecture Description
Safety Requirements Traceability Matrix

29. Component Specification and Design
Prime Item Development Specifications
Software Requirement Specifications
Software Design Documents
Software Product Specifications
Algorithm Description Documents
WIPP Minutes

30. System Safety Assessment
System Hazard Analysis Report
Fault Tree Analysis
Algorithm Contribution to HMI
Safety Processor Input Analysis
Safety Directed Analyses
Qualitative Assessments
Failure Modes and Effects Analyses/Summaries (FMEAs/FMESs)
Failure Rate Source Documentation
Common Cause Analysis
Hazard Closure Records

31. Component Implementation
Hardware Test Plans, Test Procedures, and Test Reports
Software Test Plans, Test Procedures, and Test Reports
Software Test Folders/Regression Test Folders

32. System Integration Design Qualification Test Procedures and Reports
Reliability Test Procedures and Report

33. Configuration Management
CM Plan
Configuration Audit Plan
Change Control Records
System Configuration Index

34. Quality Assurance
Quality Control Program Plan (or Quality System Plan)
Quality Audit/Review Logs

35. Safety Assurance Process Compliance
System Safety Program Plan
Safety Assurance Audit Process
Safety Audit Records
WAAS Safety Assurance Configuration Item List (SACIL)
Identifies List of Artifacts Required for Proof of SAPR Compliance
Defines Artifact Association to SAPR Requirements
Asserts Configuration Control over Artifacts Under Review
Creates and Maintains an Artifact Baseline Required for Formal Approval

36. SACIL Excerpt

37. Maintenance of WAAS Safety Baseline
Rationale for Safety Requirements a Must
Requirements Traceability
Baseline Definition and Control

38. Questions?

39. Backup Slides

40. System Development and Safety Assurance Process Model
Program Plans and Standards Safety Assurance Reviews:
Plans and Standards
Verification Procedures
Verification Results
Configuration Conformance
QA Results Audits
Program Plans and Standards:
Configuration Control Plan
Quality Assurance Plan
Development Plans
Verification Plans
Acceptance Plan
Development Standards
Verification and Validation Reviews
= Contractor Activities
= Safety Team Safety Assurance
Activities
= Contractor Verification and/or Validation Activities
= Intersecting Paths are Connected
= Intersecting Paths are not Connected
Legend:
Angled Arrows indicate data required for DO-178B Review
= Contractor Configuration Control and Quality Assurance Activities
System and Component Safety Assurance Reviews:
Specifications
System Architecture
Requirements
Designs
SW DO-178B Compliance
Procedures
Verification Procedures and Results
Validation Criteria
Operating and Maintenance Procedures
Component Test Safety Assurance Reviews:
Test Procedures
Test Results
System Integration Safety Assurance Reviews:
Safety Assessment Safety Assurance Reviews:
Qualitative/Quantitative FTAs
Common Cause Analysis
FMEAs, FMESs, Failure Rates, Exposure Times
System and Component Specification, Requirement, and Design Capture:
System Specifications
HW and SW Requirements
HW and SW Designs
Component Verification Procedures
Validation and Verification Procedures
Component Implementation:
As-Built Components
System Integration:
As-Built Ssytem
SystemTest Procedures
System Safety Assessments:
Procedure Assessments
Acceptance:
Deliverables
Conformance
Demonstration
Delivery
Configuration Control
Quality Assurance

41. Overview of Safety Assurance Activities
End
Qualitative
Design
Level B SW
HMI FT structure
assurance
assurance review
Quantitative HMI FTA
DO-178B
Safety directed analysis & test
assurance review
reviews
compliance
assurance review
review
VV/CC/QA
VV/CC/QA
Verif/Valid/CC/QA
Qualitative
Quantitative HMI FTA
Safety directed analysis & test
HMI FT structure
of critical level D SW functions
Verif / Valid/CC/QA
VV/CC/QA
SW component
data
Procedural
mitigation
Other monitoring
assessment
check data
System architecture & functional descriptions
Algorithm data
Procedural
mitigation
cert review
CRC data
HW component
data
Verif / Valid/CC/QA
HW HMI failure rates
CRC failure rates
Algorithm failure
HMI FT base event
from FMEA
rates
exposure times
Start
HMI FT base event
HW HMI FMEA
CRC failure rates
Alg failure rates assurance review
exposure times
assurance review
assurance review
assurance review
VV/CC/QA
Legend:
HMI FT base event
probability calculations
= Data flow - intersecting pathes
are
connected.
= Safety team safety assurance activites.
HMI FT base event
= Data flow - Intersecting pathes are
not
connected.
= Raytheon activites.
probability calculations
assurance reviews
= Raytheon verification, validation, config control, & QA activities
(placed behind corresponding Raytheon development activities)

42. PHMI Analysis - What Is It?
Probability of Monitor Missing an Error that would cause HMI has to be less than the assurance level (on order of 10-7)
DO-178B Level D inputs can NOT cause HMI when integrity monitor is operating as designed
Test 30 Days of WAAS Data (UDREI & GIVEI) for Overbounding
Evaluate Worst Case Convolutions of UDREI and GIVEI Histograms

43. PHMI Analysis - How Does it Support FTA

44. PHMI Analysis - How Does it Support FTA
P (MD error > cause hazard error enters monitor) < Allocation to Algorithm HMI
From UDRE, GIVE
message data determine
protection level
P(fail = 1)
~10-8
Output of Algorithm PHMI Analysis Provides the Probability of Missed Detection based on a Fault free WAAS
SPIA, SDA, FMECA and Latency Analysis Provide Probability of Missed Detection based on Faulted WAAS (SDA and SPIA qualitative)
10-7  P(HMIE) + P(HMIEC)

45. SPIA - What is it?
Analysis that Determines the Erroneous Data Inputs that Could Result in HMI being Output from the Safety Processor (SP)
Define Inputs to each Monitor
Identify any Input or Combination of Inputs that Could result in HMI
Input data Conditions include: corrupted data, absent data or data timeliness
Determine if the fault condition(s) is covered by existing monitors
Determine the Dependency or Relation to Other Monitors
Backtracks the contributing data to the input source
Assess the likelihood for the contributing data being corrupted

46. SPIA - How does it support FTA?
SPIA Complements and Validates the FT
Identifies Events that Cause HMI
Verifies Events are Captured by FT; or
Contributes to FT Completeness
New Failure Event Identification
Qualitative Event Probability Assessment
Contributes to Resolution
No Action for Extremely Low Event Probability; or
Specific Identification of Level D Software Functions Which may be Assured by SDA; or
Need for Design Change to Mitigate Failure Event

47. SDA - What is it?
Qualitative Analysis of DO-178B Level D critical software functions identified in the WAAS fault tree
Critical Level D software functions are defined as those that prevent satisfaction of WAAS safety performance requirements
For Fault Tree Analysis, Level D software has a failure probability of 1
Safety Directed Analysis is applied to the Level D software that performs these critical functions
The SDA demonstrates through analysis and test that software threats can be assigned a failure probability of 0
Ensures that the Level D software is reliable enough to support the WAAS System safety performance

48. SDA - How Does it Support FTA?
Successful SDA Eliminates Software Failure Events from the Fault Tree
Failed SDA assessments will identify Level D functions that must be mitigated
Software design change will be implemented to monitor the Critical Level D function or rehost it to Level B
Fault Tree will be modified to incorporate the updated design and reduced PHMI

49. FMEA/FMES - What is it? Failure Modes and Effects Analysis/Summary
Systematic, bottom-up method of identifying:
Failure modes of a system, component, or function and effects on next higher architecture level
Failure rate or probability of occurrence of failure modes at each level (limited to hardware on WAAS)
Also contains ability to detect severity of failure
May be performed at any architectural level
For WAAS most done at LRU level

50. FMEA/FMES - How Does it Support FTA?
ES hardware failures cause
TSWM to be corrupted
NPA054ES
FTA
ENB
(also doc’d in FMECA database)
Event failure rate is the sum of the failure rates
of the failure modes that map to it. Event
probability is calculated from the event failure rate.
Mapping of failure modes to events
- implied ‘OR’ gate, sums failure rates
for modes. Mapping documented
in ENB
Failure mode failure rates are the
sum of all the cause failure rates
that map to the failure mode.
Corrupted or intermittent data
from port 1 of ES
Corruption/noise in all
external interfaces . . .
Estimated MI failure rate
from HPR database . . .
FMECA
Failure Modes
Events in the FTA
have one or more
failure modes mapped
to it. A group of such
failure modes is called a
Failure Mode Cluster.
This cluster is CVES_WM:MI.
Data extracted from ENB C
Figure 3.1-1

51. Exposure Time/Latency Analysis - What is it?
Analysis of Exposure Time:
e.g. 150 secs, PA; 1 Hr NPA
System element under analysis known good at beginning of exposure time.
Latency accounts for components not continuously tested
The estimate of how long a failure mode can be present in an equipment or function before it is mitigated (or removed).
Requires failure detection, reporting, and mitigation
Ground rules: Monitors must be in Level B software and not already included in FTA at higher level.

52. Exposure Time/Latency Analysis - How Does it Support FTA?
Lowest FTA element requires Probability of Failure
Reliability (Probability of Success) = e-FR*t
Where FR = Failure Rate = 1/MTBF
t = Exposure Time or Latency
Exposure time for PA = 150 seconds, other phases 1 hour
Probability of Failure (Pf) = 1- Probability of Success
Safety Computer Comparator Example:
Safety Computer (SC) failure rate = 5.08 x10-5 Failures/Hour If testing were continuous, t = 150 seconds (during PA), Pf = 2.11 x10-6 Accounting for SC MTBF (19,704 Hours) Latency, Pf = Accounting for C&V MTBF (872 hours), Pf = 0.043
Source of failure rates (ENB )

53. SSAD Provide System Architectural and Design Details
Describes the Safety Architecture from System to Component to Capability Level
Provides Requirements Rationale and Tracing
Identifies Analysis to Validate and Verify Requirements at System and Component Level
System Satisfies Integrity Requirements
Failure Modes and Effects Mitigated

54. System Safety Architecture Description
SSAD
A121
System Safety
Requirement
(PHMI, Cont’y)
System Safety Analyses
FTA
Hazards
CCA
HEA
Operator
Safety Procedures
System Safety Architecture
Components
Operational Sequence
Diagrams
Algorithm Design Documents
SAC Analyses
Alg PHMI
SPIA
IETM/TIB Verification
IETM/TIB
Safety
Features
Software
Safety
Features
Component
Capabilities
Verification Methods and Results
Feature
Requirements
(SRSs)
Capability
Requirements
(CIs, SRSs)
SDFs
System Test Procedures
System Test Results