1. Experiences with Third Party Qualification of Critical Software Presenter: David Tremaine, SWI

2. Overview Evolution of Qualification (CANDU) Lessons / Challenges Future Directions © SWI 2011

3. CANDU and Software © SWI 2011 Analog Trip Meters All Emergency Trip Parameters Analog Trip Meters All Emergency Trip Parameters

4. CANDU and Software © SWI 2011 Analog Panel Meters All Plant Information Manual Control Analog Panel Meters All Plant Information Manual Control Analog Trip Meters All Emergency Trip Parameters Analog Trip Meters All Emergency Trip Parameters

5. CANDU and Software © SWI 2011 Engineering vs. Software Engineering Desk checking Unit Testing Commissioning tests Engineering vs. Software Engineering Desk checking Unit Testing Commissioning tests Analog Panel Meters All Plant Information Manual Control Analog Panel Meters All Plant Information Manual Control Analog Trip Meters All Emergency Trip Parameters Analog Trip Meters All Emergency Trip Parameters DCC Monitoring and Display Automatic Control Reactor Regulating System Boiler Pressure Control Boiler Level Control Heat Transport System Pressure … DCC Monitoring and Display Automatic Control Reactor Regulating System Boiler Pressure Control Boiler Level Control Heat Transport System Pressure …

6. Evolution of Qualification First a crisis  Darlington NGS Digital Shutdown Systems Is the software is safe? Trial and Error: A convergence of approaches © SWI 2011

7. Evolution of Qualification Next standardization Categorization: Cat I, II and III System and software engineering standards Defense-In-Depth Category I highlights Formal requirements Active Requirements Review Systematic design & code verification Hazards analysis Reliability qualification © SWI 2011

8. Evolution of Qualification Then a focus on third-party software © SWI 2011 COTS MOTS PE

9. Evolution of Qualification COG-95-179-I Guide for the Qualification of Software Products The Context: System Engineering © SWI 2011 Engineering Team Component Selection Form Fit Function Safety Category Qualifier Reliability Maintainability Reviewability Safety* Implementation System Integration V&V Commissioning System Design System Spec

10. Evolution of Qualification Qualification result  one of: Qualified for use in the application Not qualified Qualified with restrictions Qualified with project or operational obligations © SWI 2011

11. Evolution of Qualification Qualification approach Preponderance of evidence argument Derived using a combination of various methods © SWI 2011 INDIRECT EVIDENCE 1.Vendor QA Assessment 2.Operating History 3.Reference Sites 4.Maintainability Review 5.Certifications 6.Maintenance Capability 7.Anecdotal Evidence DIRECT EVIDENCE 8. Failure Modes Analysis 9. Goodness of Design 10. Safety Net Review 11. Certifications 12. Fault Tree Analysis 13. Code Review 14. Functional Testing 15. Failure Modes Testing 16. Reliability Qualification 17. Proof of Correctness

12. Evolution of Qualification CSA N290.14-07 Addresses Category I Focus on software concerns Hidden flaw 1.Recognized Program 2.Mature Product 3.Preponderance of Evidence Other common concerns 1.Security 2.Flooding 3.Modal Behaviour 4.… © SWI 2011

13. Lessons Learned 200+ components Not an appreciable improvement in SQA Even with IEC 61508 certification SQA not a factor Proven-in-use data & re-testing Configuration management / Six Sigma-Lean I&C Software is Surprisingly Good Choosing “qualifiable” software + market forces I&C software tends to be relatively simple Commercial market scaling  substantial burn-in Quality without strong SQA? © SWI 2011

14. Lessons Learned What is missing in SQA standards? People and project structure Small teams Subject matter experts Technical leadership/mentorship Engineering culture (if not software-engineering) Serious about achievement / peer pressure Professional attitude Focus on software concerns (N290.14) helps Points to important issues © SWI 2011

15. Challenges Complexity is the big foe The many obvious impacts … Understanding the system to sufficient depth Increased dormant code Vendor behaviour Obtaining adequate cooperation for a specialty market Higher consolidation + higher specialization Earlier version retirement © SWI 2011

16. Challenges The march of progress Replacement of analog meters Multi-core processors and determinism Freeware / Shareware Qualification is not a rubber stamp! Part of the engineering process All qualification “obligations” must be resolved © SWI 2011

17. Future Directions McSCert-SWI Research Tom Maibaum and Marc Bender Preparation for CSA N290.14-07 revision Other software concerns? Various avenues pursued Some joy with Peter Neumann’s List Search continues Application of assurance cases to qualification Looking at the rationale behind the qualification Re-writing sample reports using Goal Structured Notation Need an obligation box Need some rules © SWI 2011 OB: ………

18. Future Directions SDLC Utility Survey What SQA process provide the most value? Internal software survey Highlights … © SWI 2011 Active Requirements Review Scenario Based Design Presentation Desk Checking & Peer Review Unit and Stress Testing

19. Questions © SWI 2011

20. Contact Info © SWI 2011 David Tremaine CEO (416) 932-4653 mailto:dtremaine@swi.com www.swi.com