1. Software QA For Active CECs at SNS

2. 2Managed by UT-Battelle for the U.S. Department of Energy “As Scott Jerome-Parks lay dying, he clung to this wish: that his fatal radiation overdose — which left him deaf, struggling to see, unable to swallow, burned, with his teeth falling out, with ulcers in his mouth and throat, nauseated, in severe pain and finally unable to breathe — be studied and talked about publicly so that others might not have to live his nightmare.” Radiation Offers New Cures, and Ways to Do Harm NYTIMES.COM, Jan. 23, 2010 Motivation

3. 3Managed by UT-Battelle for the U.S. Department of Energy Software QA is a BHAG – Fortunately our scope is limited Observations and Questions

4. 4Managed by UT-Battelle for the U.S. Department of Energy Software QA is a BHAG – Fortunately our scope is limited Are we the “before” picture or the “after” picture? Observations and Questions

5. 5Managed by UT-Battelle for the U.S. Department of Energy Software QA is a BHAG – Fortunately our scope is limited Are we the “before” picture or the “after” picture? How much software QA is enough? – How many CEC software engineers does it take to screw in a light bulb? Seven. One to write the specification program, two to screw it in, one to check if they screwed it in, one to validate that it was screwed in correctly and two to explain why the project was late. Observations and Questions

6. 6Managed by UT-Battelle for the U.S. Department of Energy Girls just want to have fun – It takes a village If it ain’t broke don’t fix it – Change may lead to a learning experience at 3:00 am KISS, maybe – Just one more feature…. Observations and Questions

7. 7Managed by UT-Battelle for the U.S. Department of Energy Objective Review current status of QA related activities for SNS CEC software Establish a framework based on a consensus national standard(s) Come up with a comprehensive roadmap for CEC software QA at SNS

8. 8Managed by UT-Battelle for the U.S. Department of Energy Things to Keep in Mind SNS has a large PLC based CEC for the accelerator – But most of the work now is centered around new instruments These new systems are based on “safety” PLCs Limited Variability Language – Aimed at users to create their safety application functionality. Typical languages used are Ladder Diagram and Function Block Diagram

9. 9Managed by UT-Battelle for the U.S. Department of Energy SNS started out with the “standard” two redundant industrial PLC model with two programmers Transitioning from two box, two programmer model to one box, two programmer model One programmer writes the non-safety task and ½ the safety task while the other programmer writes the other ½ of the safety task AB safety PLCs have certified code modules plus diagnostics built into hardware – Use of these tools it not exactly leaping out at us The times they are a changin’

10. 10Managed by UT-Battelle for the U.S. Department of Energy Current status Things we do per procedure Things we do but not proceduralized Things we intend to start doing

11. 11Managed by UT-Battelle for the U.S. Department of Energy The following activities are required by the listed procedure: DescriptionDoc identified Software lifecycle V module reference Write SSRSSNS-ASD-IC-P031 Two programmers requiredSNS-ASD-IC-P031 Implementation testing (module testing)SNS-ASD-IC-P035 Integration testingSNS-ASD-IC-P036 Certification testing (Validation)SNS-ASD-IC-P03, SNS-OPM- 3.A-7.4 7 Modifications to software and subsequent testing requirementsSNS-ASD-IC-P03 PPS logbooksSNS-ASD-IC-P03 Use of non-network laptops for PLC program workSNS-ASD-IC-P03 Storage of PLC programs on CDsSNS-ASD-IC-P03Configuration management Details for storing PLC programs on CDsSNS-ASD-IC-P05Configuration management

12. 12Managed by UT-Battelle for the U.S. Department of Energy The following features have been partially incorporated: Expanded code comments Reference to safety function Pulse Test Complimentary inputs

13. 13Managed by UT-Battelle for the U.S. Department of Energy Standard, standard, whose got the standard Target standard is ISO 13849-1:2006(E), Safety of machinery-Safety related parts of control systems- Part 1: General principles for design Most applicable standard to current projects (Instrument PPS equipment) Addresses software QA

14. 14Managed by UT-Battelle for the U.S. Department of Energy Software lifecycle

15. 15Managed by UT-Battelle for the U.S. Department of Energy Software related software specification: The SSRS document is unique to each PPS and provides information necessary to generate the PLC program. Included are safety functions to be accomplished, guidelines such as two programmer rule, system fault definition, particular functional systems details, etc. The SSRS addresses the safety functions from the SRD. The SSRS is identified as a requirement in document SNS-ASD-IC-P03.

16. 16Managed by UT-Battelle for the U.S. Department of Energy System design: program configuration – tasks, safety tasks, routines, I/O routing, communications Module Design: Based on the system design and SSRS. a module refers to pieces of field hardware that require software to collect inputs, process data, and provide outputs. A module could be a beam shut down station, a RAD detector, a trap key sequence, etc... Should this be standardized?

17. 17Managed by UT-Battelle for the U.S. Department of Energy Coding: Is the process of writing program modules, as a standard each module may have its own routine, of be combined with other modules that contribute to similar functions. Documentation is added to identify each module and it’s components. Independent review?

18. 18Managed by UT-Battelle for the U.S. Department of Energy Module Testing - Each module is tested individually and with the system to ensure a proper and excepted outcome. How formal?

19. 19Managed by UT-Battelle for the U.S. Department of Energy Ideally will use an independent programmer in addition to two person programming team Can review code to structure test plan Has more latitude to change PPS equipment to facilitate testing Semi-formal documentation- placed in Projectwise

20. 20Managed by UT-Battelle for the U.S. Department of Energy Formal OPM procedure Structured such that “A cave man can do it” Performed per work control process annually and tracked (ASE requirement) Written by an independent person Intent is to perform “black-box” testing Focus on testing interlocks that do not have built in diagnostics Periodically testing software or just hardware faults?

21. 21Managed by UT-Battelle for the U.S. Department of Energy Improvement initiative –Standardize format, test progression, signatures, etc. –Track safety functions per SSRS –Coordinating testing with other systems (TPPS) –Compartmentalized to facilitate testing –Including warnings and cautions in lieu of a JHA –Include brief operational description for reference

22. 22Managed by UT-Battelle for the U.S. Department of Energy Use resources from vendor’s PLC safety manual Independent code review/ testing Tracking of safety functions throughout documentation Separate ACL for software items Development of standard hardware/ software layouts for instruments Future Improvements

23. 23Managed by UT-Battelle for the U.S. Department of Energy Are we there yet? PERFORMANCEMANAGEMENT 5.WORK PROCESSES 6.DESIGN 7.PROCUREMENT 8.INSPECTION & ACCEPTANCE TESTING 9.MANAGEMENT ASSESSMENT 10.INDEPENDENT ASSESSMENT 1.PROGRAM 2.PERSONNEL TRAINING & QUALIFICATION 3.QUALITY IMPROVEMENT 4.DOCUMENTS & RECORDS ASSESSMENT