Safety Critical Systems and Certification Issues DO-178B Airborne Standard

SC190 / WG-52 Application Guidelines For RTCA DO-178b/ED-12b EUROCAE SC-167 WG-12 DO-178B ED-12B CNS/ATM Community Avionics Industry Cast Position Papers SC-190 / WG-52 RTCA DO-178B is the joint development of: RTCA SC-167 and EUROCAE WG-12 It’s history is: RTCA SC-145 -> EUROCAE WG-12 ED-35 developed in parallel. ED-12 was withheld and RTCA and E. worked jointly on SC-145. 1982 => SC-145 produced the DO-178 document. 1985 => SC-152 produced DO-178A 1992 => EUROCAE WG-12 & RTCA SC-167 produced DO-178B WG = Working Group SC = Special Committee RCTA = Radio Tech. Comm. On Aeronautics EUROCAE = E. Org. for Civil Aviation Equipment CAST (Certification Authority Software Team) SC-190 Products 3

DO-178B in a Nutshell Highly Process Oriented Requires Traceability Requirements High Level Design Detailed Design Source Code Test Procedures Test results Test & Test & Test & Test and ….. 4

DO-178B Safety Levels Level A Level B Level C Level D Catastrophic Failure Prevents Continued Safe Flight and Landing Level B Hazardous/Severe-Major Potential Fatal Injuries to a Small Number of Occupants Level C Major Discomfort to Occupants or Possible Injuries Level D Minor Increased Crew Workload 5

Certification Evidence Development Team What is COTS? How can objectives of DO-178B be satisfied, using COTS? There is much variation in applicants for COTS certification credit Is DO-178B clear on the interpretations? Certification Evidence Available as COTS Product is COTS + Together these satisfy safety objectives 6

Testing without Source Code Wrappers to Validate Parameters Commercial O/S Application This cannot be trusted unless O/S is verified 7

Special Considerations Use of Service History for Certification Air Traffic Control system Worked Correctly in US for years Transferred to U.K. Actual Plane 2 Plane 1 Displayed Plane 2 Greenwich Meridian 8

Use of Service History Developed under a less stringent standard (Military?) Used for 4 years Problems tracked Quality Good! Safety Critical System Residual Error Dead Code(Unintended Function) Is this system safe for the next 4 years? At Level A, B, C? We can bound inputs, but we cannot check internal states without “looking inside” 9

Black Box Testing No single failure should prevent “Continuous safe flight and landing.” Statistical testing cannot show absence of a single state that will cause a failure Software has discontinuities Software does not follow Gaus/Normal Distribution There is no foundation for statistical reasoning about software faults or safety 10

Verification Team examples of issues What are low level requirements? How can they be tested Data and Control flow coupling Use of higher level test results for lower level requirements What is the intent of structural coverage? Traceability of source to object code for structural coverage What is statement, decision, condition and MCDC coverage testing (Modified Condition/Decision Code) Verification tool qualification etc.. MCDC = Modified Condition/Decision Code 11

Coverage Analysis at Level B and C Statement Coverage Decision Coverage Entry Points Exit Points All Decisions All Outcomes 12

Fixing anomalies example for level B/Library Filter B := 3; A := Filter (B); X := X + A; Compiler Object Code Source level coverage required Even for Filter 13

Boundary Level testing not enough! A := B; -- A and B are packed Boolean arrays Run-time call to: Bit_Block_Move (From, To, Size); -- size in bits Min, Mid, Max in combination gives 27 (useless) test cases 5 Bits size 32 Bit size 67 Bit Size From overlaps To To overlaps From Interesting test cases based on actual code i.e. White Box Testing 14

Coverage at Level A Coverage required at Machine Code level or Show source to object code traceability and test at source level or Use different language/compilers and use voting system MCDC testing required each condition must have effect on outcome Tools which modify source for traceability problem at level A Mitigation method : use 3 different compilers (Now Look At Conditional Statements) 15

Conditions/Decisions if A=B and C or D<3 then Decision Boolean Variable Boolean Operators 16

What are Conditions/Decisions if (A=B and C) or D<3 then Ada : if ((A==B) & C ) | (D<3) then C : if ((A==B) * C) + (D<3) then C : if ((A==B) and C) or (D<3) then C++ : MCDC Coverage Requires all Branches AND all Conditions Be Covered 17

More Boolean Conditions X := (A=B and C) or D<3; if X then -- X is Boolean Ada : Cannot hide from Testing Obligations X = ((A==B) * C) + (D<3)); if X then /* X can be any Integer C : ‘*’ and ‘+’ are Boolean Operators! 18

Condition coverage X := (A=B and C) or D<3; if X then -- X is Boolean Ada : Coverage of “Basic-Block” may not capture condition results 19

Avoiding MCDC Testing Use Ada’s short-circuit conditions: Modified Condition/Decision Code Use Ada’s short-circuit conditions: if A=0 and then B< 2 and then C>5 then Or in C write: if A== 0 && B < 2 && C < 5 { 20

Why short-circuit conditions eliminate MCDC if A=0 and then B< 2 and then C>5 then if A=0 then if B<2 then if C>5 then P; end if; MCDC not required for this code 21

Testing strategy must evaluate conditions if A=0 and then B< 2 and then C>5 then if A=0 then if B<2 then if C>5 then P; end if; BUT !!! Testing must show that each ‘then’ part has been tested True and False MCDC not required for this code 22

Inlining code If decisions/conditions introduced Decisions must be identified and verified (level B) Conditions must be identified and verified (level A) Verification may be done by analysis Traced to derived requirements ensure safety is not compromised Code may be “deactivated” As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements Intent - absence of unintended funtion Dead code not allowed 23

Use of Tools Tool Qualification is required if tool replaces a step of development process Development tools Examples - Compiler, Design to code generator May introduce an error In general - NOT qualified, not trusted Verification tools Examples - Coverage analyser May conceal an error May be qualified - Trusted for verification purposes Additional verification process required if the tool is not trusted 24

CNS/ATM Process Integration Information matrix Regulators Committees Standards Bodies Standard Software Integrity Assurance Standards vs. Software Development Standards Relationships between DO-178B and IEC 61508 25

Ground Based Community Communications/ Navigation/Surveilance and Air Traffic Management - in the loop Ground Based Systems affect airborne software DO-178B addresses airborne only Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee) Standards tightening up 26

Certification Standards are Improving “Holes” in documents are being fixed Understanding of Certification Requirements is spreading Industry and Certification Authorities are collaborating on Guidance materials It will get more difficult to “shop around” for a more lenient signature 27

( Legal) Safety Systems Laws Regulations Standards Guidelines Case Law Precedence Interpretations Standards Guidelines PROCESS Visibility Traceability EVIDENCE / RECORD Confidence / Safety 28

When Is Software Safe 29

When Is Software Safe We Don’t Know !! 30

What is our best guess about the safety When applicable processes have been followed When we have verified the code “from within” When this has been checked and checked 31

The FAA/JAA Rules are Strict To Date: “no fatalities have been attributed to Software Failure” Have we been lucky? Have we been safe? 32