1. The DMCC Perspective on the Application to Meteorological Software of DOE’s SQA Requirements Prepared by: Cliff Glantz (PNNL) Carl Mazzola (Shaw Env.) Walt Schalk (NOAA/SRL)

2. Roadmap Intro to the U.S. Department of Energy Meteorology Coordinating Council (DMCC) Why have a software quality assurance (SQA) program for safety software? DOE SQA requirements for safety software The difference between SAFETY and SAFETY-RELATED software Problems applying SQA for SAFETY software to SAFETY-RELATED software An alternate method of SQA for SAFETY- RELATED software Application to meteorological software…

3. Department of Energy Meteorology Coordinating Council (DMCC) Coordinates meteorological programs at all DOE sites. Promotes cost-effective meteorological support programs; facilitates use of common methods, procedures, and standards; and plans for future needs and missions Linked to DOE’s Emergency Management Issues Special Interest Group and Subcommittee on Consequence Assessment and Protective Actions (SCAPA) Resolves internal technical disputes with a mixture of violence and affection!

4. Background Info: SQA and Safety Software In 2000 the Defense Nuclear Facilities Safety Board (DNFSB) issued a report critical of the SQA of safety software at DOE nuclear facilities After much back & forth, in 2005 DOE issued an Order and Guide establishing SQA requirements for safety software: DOE Order 414.1c DOE Guide 414.1-4

5. Safety Software is Defined to Include: Software that’s “performs a safety function” as part of a structure, system, or component (SSC) at a nuclear facility. Software that is used to classify, design, or analyze nuclear facilities. This software helps to ensure the proper accident or hazards analysis of nuclear facilities or an SSC with a safety function Software that performs a hazard control function or a control function necessary to provide adequate protection from hazards. This software supports “eliminating, limiting, or mitigating nuclear hazards to workers, the public, or the environment…”

6. Establishing QA Requirements for Meteorological Programs ANSI/ANS-3.11 and DOE/EH-0173T: Make the point that quality-assured site meteorological data are critical for atmospheric dispersion modeling DOE O 414.1C and DOE G 414.1-4 establish SQA requirements for SAFETY software based on “ASME NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications, or other national or international consensus standards that provide an equivalent level of quality assurance.” Atmospheric dispersion models are SAFETY software if they are used for DOE hazard analyses or safety assessments at nuclear facilities

7. Establishing QA Requirements for Meteorological Programs (Cont.) It is reasonable to assume that the meteorological software used to in these hazards analyses and safety assessments should be either SAFETY or SAFETY- RELATED software. DOE/HS has not yet decided how to classify meteorological software DOE/HS could decide met software that generates data for SAFETY software must also be SAFETY software. What are the potential implications of this decision? *** HUGE ***

8. Why Not Apply Safety Software SQA Requirements? The average cost to bring widely used simple dispersion codes (e.g., EPICode) into full compliance with DOE O 414.1C is >$300K/model! For more sophisticated models, these costs would be greater. Need to find a balance that allows the effective use of resources (i.e., balance technical development needs against SQA needs). Remember – SQA helps to ensure that software does what it is designed to do, not that it is doing the right thing!

9. A Solution… Best Strategy: Answer the question for DOE before they grapple with this issue. How: Voluntarily adopt a standard for SQA, based on meteorological software being SAFETY-RELATED not SAFETY Software. Proactive: We define guidance for SAFETY-RELATED and applications before we are told what they should be.

10. DMCC Proposed Action Improve current SQA practices for met software Met software should meet a “reasonable” set of SQA requirements applicable to SAFETY-RELATED software. Apply consistent standards across DOE (and vendors) for met software development and maintenance. Encourage technical innovation and avoid “stifling” SAFETY software SQA requirements \ Be consistent with ANSI/ANS-3.11 and DOE/EH-0173T. Base SAFETY-RELATED SQA on the ten work activities required for SAFETY software, but use a more liberal approach.

11. Ten SQA Work Activities 1. Software Project Management and Quality Planning 2. Software Risk Management 3. Software Configuration Management 4. Procurement & Vendor Management 5. Software Requirements Identification and Management 6. Software Design & Implementation 7. Software Safety Design 8. Verification & Validation 9. Problem Reporting & Corrective Action 10. Training of Personnel

12. In DOE’s SQA Requirements for Safety Software… Requirements have been established for: 1. Custom developed 2. Configurable 3. Commercial off-the-shelf (COTS) software 4. Utility calculations (e.g., spreadsheets) 5. Commercial design and analysis tools. AB C For each type of software, three different levels (A, B, or C) can be assigned based on how the software is being used. For each work activity specified in the SQA requirements, a full or graded approach may be required.

13. Work ActivitiesCustom A B C COTS A B C 1. Software Project Management and Quality PlanningFULL FULL FULL GRADE GRADEGRADE 2. Software Risk ManagementFULL GRADE GRADE FULL 3. Software Configuration MgmtFULL FULL FULL GRADE GRADEFULL FULL FULL GRADE GRADE 4. Procurement & Vendor MgmtFULL FULL FULL FULL 5. Software Requirements Identification and ManagementFULL FULL FULL FULL 6. Software Design & ImplementationFULL FULL FULL NA NA NA 7. Software Safety DesignFULL FULL FULL GRADE GRADEFULL FULL FULL GRADE GRADE 8. Verification & ValidationFULL GRADE GRADE FULL 9. Problem Reporting & Corrective ActionFULL FULL FULL FULL GRADE GRADE 10. Training of PersonnelFULL FULL FULL GRADE GRADEFULL FULL FULL GRADE GRADE

14. Safety Software SQA Level of Effort

15. Safety-Related Software SQA Level of Effort

16. Applying Work Activities Develop SQA Project Management and Quality Planning documentation. Document needed SQA activities Establish SQA milestones Assign SQA responsibilities. Develop a Configuration Management Plan to: Ensure configuration control during software development Ensure secure storage of the source code, executables, software documentation, V&V test procedures & results, and SQA documentation.

17. Prepare Design and Implementation Documentation to clearly detail how the software works. This should include: sufficient information to support continuity in software development if there were an abrupt change in project personnel address and provide users with sufficient documentation to efficiently use the software and understand what it is doing. Design, conduct, and document V&V Testing to: focus on the portions of the code that were modified include baseline testing include independent testing by someone not on the software development team. Develop and implement Problem Reporting/Tracking requirements Train the software development team Prepare User’s documentation/instructions Applying Work Activities (cont)

18. Applicability of Safety-Related Software SQA Guidance SQA must be applied during the development of new software and the modification of existing software. For legacy software that is not undergoing modification, there is no requirement to apply this SQA guidance retroactively. If resources are available, priorities are to ensure that code documentation and verification testing data sets are produced, testing conducted, and testing documentation placed into configuration-managed storage.

19. In applying the guidance to COTS software, SCAPA is setting up expectations for SQA that should be met easily by the existing SQA programs of commercial software developers. The proposed SQA guidance should be compatible with most existing SQA programs. In some cases, an existing SQA program may already incorporate all of the key aspects of the proposed guidance. In other instances, there may be components in the proposed guidance that are not currently addressed. Applicability of Safety-Related Software SQA (cont)

20. Draft SQA Guidance for Safety-Related Software …

21. Conclusion DMCC will draft and propose SAFETY-RELATED SQA guidance for meteorological software. A graded approach using DOE G 414.1-4 ten work activities will be used for developing SQA guidance. Legacy software SQA will focus on code documentation and V & V activities. Upon acceptance, DMCC will distribute and promote.

22. Finalize draft DMCC SQA Plan for met software for by end of September and circulate for peer review Issue SQA Guidelines and post on DMCC website by December 2008 Path Forward

23. Questions? Cliff Glantz Chair of DOE Subcommittee on Consequence Assessment and Protective Actions (SCAPA) Pacific Northwest National Laboratory PO Box 999 Richland, WA 99352 509-375-2166 cliff.glantz@pnl.gov