Presentation is loading. Please wait.

Presentation is loading. Please wait.

Safety Critical Systems and Certification Issues DO-178B Airborne Standard.

Similar presentations


Presentation on theme: "Safety Critical Systems and Certification Issues DO-178B Airborne Standard."— Presentation transcript:

1

2 Safety Critical Systems and Certification Issues DO-178B Airborne Standard

3 3 SC190 / WG-52 Application Guidelines For RTCA DO-178b/ED-12b DO-178B ED-12B RTCAEUROCAE SC-167 WG-12 CAST (Certification Authority Software Team) Cast Position Papers SC-190 / WG-52 SC-190 Products CNS/ATM Community Avionics Industry

4 4 DO-178B in a Nutshell nHighly Process Oriented nRequires Traceability  Requirements  High Level Design  Detailed Design  Source Code  Test Procedures  Test results nTest & Test & Test & Test and ….. nHighly Process Oriented nRequires Traceability  Requirements  High Level Design  Detailed Design  Source Code  Test Procedures  Test results nTest & Test & Test & Test and …..

5 5 DO-178B Safety Levels nLevel A Catastrophic Catastrophic Failure Prevents Continued Safe Flight and Landing Failure Prevents Continued Safe Flight and Landing nLevel B Hazardous/Severe-Major Hazardous/Severe-Major Potential Fatal Injuries to a Small Number of Occupants Potential Fatal Injuries to a Small Number of Occupants nLevel C Major Major Discomfort to Occupants or Possible Injuries Discomfort to Occupants or Possible Injuries nLevel D Minor Minor Increased Crew Workload Increased Crew Workload nLevel A Catastrophic Catastrophic Failure Prevents Continued Safe Flight and Landing Failure Prevents Continued Safe Flight and Landing nLevel B Hazardous/Severe-Major Hazardous/Severe-Major Potential Fatal Injuries to a Small Number of Occupants Potential Fatal Injuries to a Small Number of Occupants nLevel C Major Major Discomfort to Occupants or Possible Injuries Discomfort to Occupants or Possible Injuries nLevel D Minor Minor Increased Crew Workload Increased Crew Workload

6 6 Development Team nWhat is COTS? nHow can objectives of DO-178B be satisfied, using COTS? nThere is much variation in applicants for COTS certification credit nIs DO-178B clear on the interpretations? nWhat is COTS? nHow can objectives of DO-178B be satisfied, using COTS? nThere is much variation in applicants for COTS certification credit nIs DO-178B clear on the interpretations? Product is COTS Certification Evidence Available as COTS + Together these satisfy safety objectives

7 7 Testing without Source Code Commercial O/S Wrappers to Validate Parameters Application This cannot be trusted unless O/S is verified

8 8 Special Considerations nWorked Correctly in US for years nTransferred to U.K. nWorked Correctly in US for years nTransferred to U.K. Air Traffic Control system Plane 1 Actual Plane 2 Displayed Plane 2 Greenwich Meridian Use of Service History for Certification

9 9 Use of Service History Safety Critical System Developed under a less stringent standard (Military?) Used for 4 years Problems tracked Quality Good! Dead Code(Unintended Function)Residual Error Is this system safe for the next 4 years? At Level A, B, C? We can bound inputs, but we cannot check internal states without “looking inside”

10 10 Black Box Testing nNo single failure should prevent “Continuous safe flight and landing.” nStatistical testing cannot show absence of a single state that will cause a failure nSoftware has discontinuities nSoftware does not follow Gaus/Normal Distribution nNo single failure should prevent “Continuous safe flight and landing.” nStatistical testing cannot show absence of a single state that will cause a failure nSoftware has discontinuities nSoftware does not follow Gaus/Normal Distribution There is no foundation for statistical reasoning about software faults or safety There is no foundation for statistical reasoning about software faults or safety

11 11 Verification Team examples of issues What are low level requirements? How can they be tested What are low level requirements? How can they be tested Data and Control flow coupling Data and Control flow coupling Use of higher level test results for lower level requirements Use of higher level test results for lower level requirements What is the intent of structural coverage? What is the intent of structural coverage? Traceability of source to object code for structural coverage Traceability of source to object code for structural coverage What is statement, decision, condition and MCDC coverage testing ( Modified Condition/Decision Code) What is statement, decision, condition and MCDC coverage testing ( Modified Condition/Decision Code) Verification tool qualification Verification tool qualification etc.. etc.. What are low level requirements? How can they be tested What are low level requirements? How can they be tested Data and Control flow coupling Data and Control flow coupling Use of higher level test results for lower level requirements Use of higher level test results for lower level requirements What is the intent of structural coverage? What is the intent of structural coverage? Traceability of source to object code for structural coverage Traceability of source to object code for structural coverage What is statement, decision, condition and MCDC coverage testing ( Modified Condition/Decision Code) What is statement, decision, condition and MCDC coverage testing ( Modified Condition/Decision Code) Verification tool qualification Verification tool qualification etc.. etc..

12 12 Coverage Analysis at Level B and C nStatement Coverage nDecision Coverage Entry Points Entry Points Exit Points Exit Points All Decisions All Decisions All Outcomes All Outcomes nStatement Coverage nDecision Coverage Entry Points Entry Points Exit Points Exit Points All Decisions All Decisions All Outcomes All Outcomes

13 13 Fixing anomalies example for level B/Library Compiler B := 3; A := Filter (B); X := X + A; Filter Object Code Source level coverage required Even for Filter

14 14 Boundary Level testing not enough! nMin, Mid, Max in combination gives 27 (useless) test cases 5 Bits size 5 Bits size 32 Bit size 32 Bit size 67 Bit Size 67 Bit Size From overlaps To From overlaps To To overlaps From To overlaps From nMin, Mid, Max in combination gives 27 (useless) test cases 5 Bits size 5 Bits size 32 Bit size 32 Bit size 67 Bit Size 67 Bit Size From overlaps To From overlaps To To overlaps From To overlaps From A := B; -- A and B are packed Boolean arrays Run-time call to: Bit_Block_Move (From, To, Size); -- size in bits Interesting test cases based on actual code i.e. White Box Testing Interesting test cases based on actual code i.e. White Box Testing

15 15 Coverage at Level A Coverage required at Machine Code level or Coverage required at Machine Code level or Show source to object code traceability Show source to object code traceability and test at source level or Use different language/compilers Use different language/compilers and use voting system MCDC testing required MCDC testing required  each condition must have effect on outcome Tools which modify source for traceability Tools which modify source for traceability problem at level A  Mitigation method : use 3 different compilers (Now Look At Conditional Statements) Coverage required at Machine Code level or Coverage required at Machine Code level or Show source to object code traceability Show source to object code traceability and test at source level or Use different language/compilers Use different language/compilers and use voting system MCDC testing required MCDC testing required  each condition must have effect on outcome Tools which modify source for traceability Tools which modify source for traceability problem at level A  Mitigation method : use 3 different compilers (Now Look At Conditional Statements)

16 16 Conditions/DecisionsConditions/Decisions if A=B and C or D<3 then Boolean Operators Boolean Variable Conditions Decision

17 17 What are Conditions/Decisions if (A=B and C) or D<3 then Ada : if ((A==B) & C ) | (D<3) then C : if ((A==B) * C) + (D<3) then C : if ((A==B) and C) or (D<3) then C++ : MCDC Coverage Requires all Branches AND all Conditions Be Covered

18 18 More Boolean Conditions X := (A=B and C) or D<3; if X then -- X is Boolean Ada : X = ((A==B) * C) + (D<3)); if X then /* X can be any Integer C : Cannot hide from Testing Obligations ‘*’ and ‘+’ are Boolean Operators!

19 19 Condition coverage X := (A=B and C) or D<3; if X then -- X is Boolean Ada : Coverage of “Basic-Block” may not capture condition results

20 20 Avoiding MCDC Testing Use Ada’s short-circuit conditions: if A=0 and then B 5 then Or in C write: if A== 0 && B < 2 && C < 5 { Modified Condition/Decision Code

21 21 Why short-circuit conditions eliminate MCDC if A=0 then if B<2 then if C>5 then P; end if; if A=0 then if B<2 then if C>5 then P; end if; if A=0 and then B 5 then MCDC not required for this code

22 22 Testing strategy must evaluate conditions if A=0 then if B<2 then if C>5 then P; end if; if A=0 then if B<2 then if C>5 then P; end if; if A=0 and then B 5 then MCDC not required for this code BUT !!! Testing must show that each ‘then’ part has been tested True and False Testing must show that each ‘then’ part has been tested True and False

23 23 Inlining code If decisions/conditions introduced If decisions/conditions introduced  Decisions must be identified and verified (level B)  Conditions must be identified and verified (level A) Verification may be done by analysis Verification may be done by analysis  Traced to derived requirements  ensure safety is not compromised  Code may be “deactivated” As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements Intent - absence of unintended funtion Intent - absence of unintended funtion Dead code not allowed Dead code not allowed If decisions/conditions introduced If decisions/conditions introduced  Decisions must be identified and verified (level B)  Conditions must be identified and verified (level A) Verification may be done by analysis Verification may be done by analysis  Traced to derived requirements  ensure safety is not compromised  Code may be “deactivated” As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements As inlined code depends on local state it may be very hard to test the conditions in accordance with standards requirements Intent - absence of unintended funtion Intent - absence of unintended funtion Dead code not allowed Dead code not allowed

24 24 Use of Tools Tool Qualification is required if tool replaces a step of development process Tool Qualification is required if tool replaces a step of development process Development tools Development tools  Examples - Compiler, Design to code generator  May introduce an error  In general - NOT qualified, not trusted Verification tools Verification tools  Examples - Coverage analyser  May conceal an error  May be qualified - Trusted for verification purposes Additional verification process required if the tool is not trusted Additional verification process required if the tool is not trusted Tool Qualification is required if tool replaces a step of development process Tool Qualification is required if tool replaces a step of development process Development tools Development tools  Examples - Compiler, Design to code generator  May introduce an error  In general - NOT qualified, not trusted Verification tools Verification tools  Examples - Coverage analyser  May conceal an error  May be qualified - Trusted for verification purposes Additional verification process required if the tool is not trusted Additional verification process required if the tool is not trusted

25 25 CNS/ATM Process Integration nInformation matrix Regulators Regulators Committees Committees Standards Bodies Standards Bodies Standard Standard nSoftware Integrity Assurance Standards vs. Software Development Standards nRelationships between DO-178B and IEC and IEC nInformation matrix Regulators Regulators Committees Committees Standards Bodies Standards Bodies Standard Standard nSoftware Integrity Assurance Standards vs. Software Development Standards nRelationships between DO-178B and IEC and IEC 61508

26 26 Ground Based Community Communications/ Navigation/Surveilance and Air Traffic Management - in the loop Communications/ Navigation/Surveilance and Air Traffic Management - in the loop Ground Based Systems affect airborne software Ground Based Systems affect airborne software DO-178B addresses airborne only DO-178B addresses airborne only Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee) Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee) Standards tightening up Standards tightening up Communications/ Navigation/Surveilance and Air Traffic Management - in the loop Communications/ Navigation/Surveilance and Air Traffic Management - in the loop Ground Based Systems affect airborne software Ground Based Systems affect airborne software DO-178B addresses airborne only DO-178B addresses airborne only Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee) Guidance being prepared to encompass needs of CNS/ATM community (SC-190 committee) Standards tightening up Standards tightening up

27 27 Certification Standards are Improving “Holes” in documents are being fixed “Holes” in documents are being fixed Understanding of Certification Requirements is spreading Understanding of Certification Requirements is spreading Industry and Certification Authorities are collaborating on Guidance materials Industry and Certification Authorities are collaborating on Guidance materials It will get more difficult to “shop around” for a more lenient signature It will get more difficult to “shop around” for a more lenient signature “Holes” in documents are being fixed “Holes” in documents are being fixed Understanding of Certification Requirements is spreading Understanding of Certification Requirements is spreading Industry and Certification Authorities are collaborating on Guidance materials Industry and Certification Authorities are collaborating on Guidance materials It will get more difficult to “shop around” for a more lenient signature It will get more difficult to “shop around” for a more lenient signature

28 28 ( Legal) Safety Systems Laws Regulations Standards Guidelines Case Law Precedence Interpretations Standards Guidelines Visibility Traceability PROCESS EVIDENCE / RECORD Confidence / Safety

29 29 When Is Software Safe

30 30 When Is Software Safe We Don’t Know !!

31 31 What is our best guess about the safety When applicable processes have been followed When applicable processes have been followed When we have verified the code “from within” When we have verified the code “from within” When this has been checked When this has been checked and checked and checked and checked When applicable processes have been followed When applicable processes have been followed When we have verified the code “from within” When we have verified the code “from within” When this has been checked When this has been checked and checked and checked and checked

32 32 The FAA/JAA Rules are Strict To Date: “no fatalities have been attributed to Software Failure” Have we been lucky? Have we been safe?


Download ppt "Safety Critical Systems and Certification Issues DO-178B Airborne Standard."

Similar presentations


Ads by Google