1. Using Risk Assessment to Drive Software Validation Decisions Don Hopkins, Ph.D. Ursa Logic Corporation

2. May 24, 2004© 2004, DQRI. All rights reserved.2 Software Development Today Dozens of viable software methodologies currently in use. Great variation in terminologies, procedures, deliverables, and ways of organizing work. Each methodology embodies a strategy for safeguarding software quality.

3. May 24, 2004© 2004, DQRI. All rights reserved.3 From the Draft Guidance on Software Validation (2001) Validation Plans Requirement specifications Simulation tests User-site tests Structural tests Functional tests Program build tests Code inspections Code walk-throughs Technical reviews Change control Regression analysis Risk assessments Validation Reports

4. May 24, 2004© 2004, DQRI. All rights reserved.4 In what ways can part 11 discourage innovation? By including an across-the-board requirement that covered software must be validated, and publishing a guidance that defined software validation in terms of specific deliverables, the FDA established a de facto standard software development methodology for covered systems, regardless of context and regardless of risk profile.

5. May 24, 2004© 2004, DQRI. All rights reserved.5 In what ways can part 11 discourage innovation? There has been no agreed-upon language for justifying anything less or anything different than what is outlined in the guidance documents.

6. May 24, 2004© 2004, DQRI. All rights reserved.6 Risk Assessment Risk assessment can provide a means of justifying diverse approaches to software development and validation. Industry and the agency must agree on a common framework for performing risk assessment.

7. May 24, 2004© 2004, DQRI. All rights reserved.7 DQRI Risk Assessment Model Standard risk model Industry consensus on four critical questions

8. May 24, 2004© 2004, DQRI. All rights reserved.8 DQRI Risk Assessment Model Step 1: Common types of software used in clinical trials, and risks typically associated with them.

9. May 24, 2004© 2004, DQRI. All rights reserved.9 DQRI Risk Assessment Model Step 5: Outcomes that should be considered in assessing the impact of possible software failures.

10. May 24, 2004© 2004, DQRI. All rights reserved.10 DQRI Risk Assessment Model Step 6: Factors that contribute to the likelihood of system failures.

11. May 24, 2004© 2004, DQRI. All rights reserved.11 DQRI Risk Assessment Model Step 8: Dimensions on which validation procedures may vary depending on assessed risk scores.

12. May 24, 2004© 2004, DQRI. All rights reserved.12 Recommendation #1 Software validation cannot be separated from software development methodologies. Software validation has no meaning that can be usefully generalized across diverse development methodologies. Software development methodologies are still emerging; there is no consensus about which methodologies work best under what circumstances. Software validation is not the only strategy available for controlling software-related risks. The across-the-board requirement for systems to be validated should be removed from Part 11.

13. May 24, 2004© 2004, DQRI. All rights reserved.13 Recommendation #2 (a) identify software-related risks; (b) justify the controls adopted to control those risks. Instead of mandating software validation, Part 11 should require risk assessment to:

14. May 24, 2004© 2004, DQRI. All rights reserved.14 Recommendation #3 Risk assessment model Risks associated with common types of software used in clinical trials Outcomes that should be considered in assessing the impact of possible software failures Factors that contribute to the likelihood of system failures Dimensions on which validation procedures may vary as assessments of risk increase or decrease. The agency should work with industry to develop a guidance on risk assessment for software used in connection with electronic records.

15. May 24, 2004© 2004, DQRI. All rights reserved.15 Yes. A risk assessment model can be generalized to provide a framework for justifying decisions about any activity that affects the quality of electronic data. In addition to validation, audit trail, record retention, and record copying, should other areas of Part 11 (e.g., operational system and device checks) incorporate the concept of a risk-based approach? Conclusion

16. May 24, 2004© 2004, DQRI. All rights reserved.16 Thank You Don Hopkins, Ph.D. Ursa Logic Corporation hopkins@ursalogic.com For inquiries about the Data Quality Research Institute (DQRI) please contact: Kaye Fendt, MSPH kfendt@email.unc.edu