1. Analysis of Current Maturity Models and Standards
Robert Sauer
Policy Analyst CDRH/OIR

2. Outline Software Quality and Maturity
Software Development Standards and Models in Safety Critical Industries
Other Approaches to Risk-Based Categorization
Discussion topics

3. Software Quality and Maturity
Software quality is an abstract concept
Existing quantitative metrics are incomplete
Initial software quality models defined quality as exhibiting desirable properties (e.g. maintainability, reliability, compatibility, testability, efficiency, ...)
Now, maturity models and standards also consider a firm’s quality management system, processes and procedures to better understand its capability to produce quality software

4. Quality System Information and Device Regulation
CDRH already uses confidence in the firm’s QMS along with other regulatory controls like adverse event reporting and recalls for risk based classification of medical devices
CDRH already uses knowledge of a certain subset of QS regulations (design controls) to tailor submission requirements for Special 510(k)’s

5. Initial Working Questions FDA considered
What best practices in software development can be applied to medical devices?
To what extent are these reflected in current standards or maturity models?
What can we learn from standards and models in other safety critical industries?
How can we use that information to realize efficiencies in our review processes for SaMD and SiMD?

6. Software Development Standards and Models in Safety Critical Industries
Medical devices
IEC 62304
Other Safety Critical industries
CMMI
DO-178C
IEC 26262
In development
IEC 82304: Health Software – General Requirements for Product Safety
IEC Series: Software Process Improvement Capability Determination - SPICE (Formerly 15504)

7. IEC/ISO 62304 Medical device software –Software life cycle processes
Description
Describes software lifecycle processes and artifacts that should be generated based on device risk class
Results in Conformance/Non-conformance Decision, different requirements based on safety class
Analysis
Aligns very well with CDRH guidance documents
Explicitly includes requirements for safety and security
Designed for and used extensively by medical device manufacturers
Has multiple reputable conformance assessment bodies (e.g. EU)
FDA Recognized
Discussion Topics
How feasibly can assessments be performed at the firm level?
How do we go beyond 62304? Feasibility of giving credit to those who do more

8. Capability Maturity Model Integration for Development V1.3 (CMMI)
Description
A collection of best practices that help organizations improve their processes. Originated in software engineering, but has been generalized for any industry.
Levels 2-5: Managed, Defined, Quantitatively Managed, Optimized
Analysis
Aligns well with CDRH guidance documents
Suggests or recommends considerations like safety and security, but does not explicitly include these aspects in the requirements
While CMMI is not designed for a specific industry and currently used by few medical device firms, it is widely used and sometimes required in other safety critical industries like defense, aviation, and aerospace
The Software Engineering Institute, which is the organization that maintains CMMI, has a rigorous certification program for becoming a CMMI appraiser, of which there are many
Discussion Topics
Return on Investment may vary depending on context
Safety is not an explicit requirement; risk is in terms of project risk, not product risk

9. Aviation
DO-178C Software Considerations in Airborne Systems and Equipment Certification
More detailed about the purpose of each part of the development life cycle and information contained in their associated artifacts
Architectural considerations
Defining and documenting development standards
Specific goals for reviews at each stage of the development process
Normal range and robustness test cases
Tool qualification document

10. Automotive IEC 26262 Road Vehicles- Functional Safety
ASIL Decomposition
Mandates or recommends methods and practices, for different ASIL levels
Methods for the verification of software unit design and implementation (Table 9)
Mechanisms for error detection (Table 4)
Structural coverage metrics at the software unit level (Table 12)
Test environments for conducting the software safety requirements verification (Table 16)
Topics to be covered in modelling and coding guidelines

11. Other Quality and Software Standards
ISO/IEC 12207: Software life cycle processes
ISO/IEC 25000: Systems and software engineering -- Systems and software Quality Requirements and Evaluation (SQuaRE)
IEEE/ISO/IEC 90003: Software Engineering -- Guidelines for the Application of ISO 9001:2008 to Computer Software
ISO Environmental management
SAE AS 9100 Quality Systems - Aerospace - Model for Quality Assurance in Design, Development, Production, Installation and Servicing
Nuclear Quality Assurance (NQA-1) Certification
ISO/IEC (2005)30: General Requirements for the Competence of Testing and Calibration Laboratories
TL 9000 Requirements Handbook (Telecommunications)

12. Other Approaches to Risk-Based Categorization
Medical device software varies in risk, functionality and complexity
Current CDRH guidance uses Level of Concern (Major, Moderate, Minor)
Many standards tailor requirements based on risk

13. Risk paradigms: DO-178C

14. Risk Paradigms: IEC 61508 Safety Integrity Level
Low demand mode: average probability of failure on demand
High demand or continuous mode: probability of dangerous failure per hour 1
≥ 10-2 to < 10-1
≥ 10-6 to < 10-5 2
≥ 10-3 to < 10-2
≥ 10-7 to < 10-6 3
≥ 10-4 to < 10-3
≥ 10-8 to < 10-7 4
≥ 10-5 to < 10-4
≥ 10-9 to < 10-8

15. Risk Paradigms: IMDRF SaMD N12
The four categories (I, II, III, IV) are based on the levels of impact on the patient or public health where accurate information provided by the SaMD to treat or diagnose, drive or inform clinical management is vital to avoid death, long-term disability or other serious deterioration of health, mitigating public health.

16. Possible Discussion Topics
What would an ideal software maturity or development process evaluation look like? How could it provide additional confidence in product quality? (Goal = provide you the most confidence in the quality of the finished device)
Are there other existing standards that could serve as a starting point?
What are the pros and cons of the standards and models identified?
What lessons can we learn from other industries?
Can a current model be adapted to the medical device industry as is, with an add on, or should we start from scratch?
Are there specific review pain points that these types of evaluation could address?
If as an option we incorporated a model like this in our review, how could we use real-world data to determine if it is working?