Analysis of Current Maturity Models and Standards Robert Sauer Policy Analyst CDRH/OIR

Outline Software Quality and Maturity Software Development Standards and Models in Safety Critical Industries Other Approaches to Risk-Based Categorization Discussion topics

Software Quality and Maturity Software quality is an abstract concept Existing quantitative metrics are incomplete Initial software quality models defined quality as exhibiting desirable properties (e.g. maintainability, reliability, compatibility, testability, efficiency, ...) Now, maturity models and standards also consider a firm’s quality management system, processes and procedures to better understand its capability to produce quality software

Quality System Information and Device Regulation CDRH already uses confidence in the firm’s QMS along with other regulatory controls like adverse event reporting and recalls for risk based classification of medical devices CDRH already uses knowledge of a certain subset of QS regulations (design controls) to tailor submission requirements for Special 510(k)’s

Initial Working Questions FDA considered What best practices in software development can be applied to medical devices? To what extent are these reflected in current standards or maturity models? What can we learn from standards and models in other safety critical industries? How can we use that information to realize efficiencies in our review processes for SaMD and SiMD?

Software Development Standards and Models in Safety Critical Industries Medical devices IEC 62304 Other Safety Critical industries CMMI DO-178C IEC 26262 In development IEC 82304: Health Software – General Requirements for Product Safety IEC 33000 Series: Software Process Improvement Capability Determination - SPICE (Formerly 15504)

IEC/ISO 62304 Medical device software –Software life cycle processes Description Describes software lifecycle processes and artifacts that should be generated based on device risk class Results in Conformance/Non-conformance Decision, different requirements based on safety class Analysis Aligns very well with CDRH guidance documents Explicitly includes requirements for safety and security Designed for and used extensively by medical device manufacturers Has multiple reputable conformance assessment bodies (e.g. EU) FDA Recognized Discussion Topics How feasibly can 62304 assessments be performed at the firm level? How do we go beyond 62304? Feasibility of giving credit to those who do more

Capability Maturity Model Integration for Development V1.3 (CMMI) Description A collection of best practices that help organizations improve their processes. Originated in software engineering, but has been generalized for any industry. Levels 2-5: Managed, Defined, Quantitatively Managed, Optimized Analysis Aligns well with CDRH guidance documents Suggests or recommends considerations like safety and security, but does not explicitly include these aspects in the requirements While CMMI is not designed for a specific industry and currently used by few medical device firms, it is widely used and sometimes required in other safety critical industries like defense, aviation, and aerospace The Software Engineering Institute, which is the organization that maintains CMMI, has a rigorous certification program for becoming a CMMI appraiser, of which there are many Discussion Topics Return on Investment may vary depending on context Safety is not an explicit requirement; risk is in terms of project risk, not product risk

Aviation DO-178C Software Considerations in Airborne Systems and Equipment Certification More detailed about the purpose of each part of the development life cycle and information contained in their associated artifacts Architectural considerations Defining and documenting development standards Specific goals for reviews at each stage of the development process Normal range and robustness test cases Tool qualification document

Automotive IEC 26262 Road Vehicles- Functional Safety ASIL Decomposition Mandates or recommends methods and practices, for different ASIL levels Methods for the verification of software unit design and implementation (Table 9) Mechanisms for error detection (Table 4) Structural coverage metrics at the software unit level (Table 12) Test environments for conducting the software safety requirements verification (Table 16) Topics to be covered in modelling and coding guidelines

Other Quality and Software Standards ISO/IEC 12207: Software life cycle processes ISO/IEC 25000: Systems and software engineering -- Systems and software Quality Requirements and Evaluation (SQuaRE) IEEE/ISO/IEC 90003: Software Engineering -- Guidelines for the Application of ISO 9001:2008 to Computer Software ISO 14000 - Environmental management SAE AS 9100 Quality Systems - Aerospace - Model for Quality Assurance in Design, Development, Production, Installation and Servicing Nuclear Quality Assurance (NQA-1) Certification ISO/IEC 17025 (2005)30: General Requirements for the Competence of Testing and Calibration Laboratories TL 9000 Requirements Handbook (Telecommunications)

Other Approaches to Risk-Based Categorization Medical device software varies in risk, functionality and complexity Current CDRH guidance uses Level of Concern (Major, Moderate, Minor) Many standards tailor requirements based on risk

Risk paradigms: DO-178C

Risk Paradigms: IEC 61508 Safety Integrity Level Low demand mode: average probability of failure on demand High demand or continuous mode: probability of dangerous failure per hour 1 ≥ 10-2 to < 10-1 ≥ 10-6 to < 10-5 2 ≥ 10-3 to < 10-2 ≥ 10-7 to < 10-6 3 ≥ 10-4 to < 10-3 ≥ 10-8 to < 10-7 4 ≥ 10-5 to < 10-4 ≥ 10-9 to < 10-8

Risk Paradigms: IMDRF SaMD N12 The four categories (I, II, III, IV) are based on the levels of impact on the patient or public health where accurate information provided by the SaMD to treat or diagnose, drive or inform clinical management is vital to avoid death, long-term disability or other serious deterioration of health, mitigating public health.

Possible Discussion Topics What would an ideal software maturity or development process evaluation look like? How could it provide additional confidence in product quality? (Goal = provide you the most confidence in the quality of the finished device) Are there other existing standards that could serve as a starting point? What are the pros and cons of the standards and models identified? What lessons can we learn from other industries? Can a current model be adapted to the medical device industry as is, with an add on, or should we start from scratch? Are there specific review pain points that these types of evaluation could address? If as an option we incorporated a model like this in our review, how could we use real-world data to determine if it is working?