1. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The Challenges of Validating R Managing R in a Commercial Environment

2. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What does validation mean Why validate R The challenges of validating R Validation tools ValidR Questions Agenda

3. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What does validation mean? “Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product, meeting its predetermined specifications and quality attributes” U.S. Food and Drug Administration (2013). 21 CFR Part 11: Electronic Records, Electronic Signatures, http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1

4. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Why validate R? No guarantee that packages meet specifications Must comply with industry regulations to be used in regulated environments

5. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Why validate R? R Foundation response to CFR Part 11: R: Regulatory Compliance and Validation Issues A Guidance Document for the Use of R in Regulated Clinical Trial Environments 1 Document relates to only base R and recommended packages R CMD CHECK provides no guarantee that an R add-on package meets its specifications Authors do not have to write tests 1. http://www.r-project.org/doc/R-FDA.pdf

6. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com ## WARNING: ## This code is a complete hack, ## may or may not work, etc.. ## Use your own risk. You have been warned. The recommended R package: ‘ codetools’, version 0.2-8. Why validate R?

7. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What is validation? “Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product, meeting its predetermined specifications and quality attributes” U.S. Food and Drug Administration (2013). 21 CFR Part 11: Electronic Records, Electronic Signatures, http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1

8. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The challenges of validating R Define Requirements Build Software Test Software Against Requirements Create Documentation Typical Software Development Process Investigation Remediation Certification

9. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Challenge 1 : Defining requirements Packages don’t typically come with a list of requirements How do we determine the intended use? Package descriptions and help files Package vignettes Previous experience of using the packages

10. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Challenge 2 : Understanding a package Software is already built Source code is available How do we gain an understanding of the package structure? The functionMap package

11. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The functionMap package Describes the relationship of functions within an R package Parse R code Create dynamic graphics of function relationships Create a network object to show relationships Creates an interactive graphic See which functions a given function calls Determine functions that are called within each function

12. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com

13. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com

14. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Challenge 3 : Testing the requirements Package authors don’t have to write tests How do we test the requirements? Writing specific unit tests for requirements Understand the level of testing with testCoverage

15. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The testCoverage package Determines which code in a package is covered by unit tests Reads package code and adds tracepoints Runs package unit tests and marks points as hit Creates a full report of the code hit

16. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com

17. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Challenge 4 : Validation documentation Creating consistent documentation for each R package Automatic report template generation knitr Audit trail

18. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Challenge 5 : Deployment Installation Qualification (IQ) Operation Qualification (OQ) Performance Qualification (PQ)

19. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com IQ - Is everything installed as expected? Perform an audit of the installed files OQ - Does everything operate as expected? Run the tests used in validation locally PQ - Does everything perform as expected? Measure the performance of the installed software VALIDATION Qualification process

20. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What is ValidR? R (currently 3.0.2) Core and Recommended Packages 60+ additional packages Validation Packages Validation Documentation Windows Installer Linux RPMs

21. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What are the benefits? Industry Gives analysts the tools they want use Provides assurance to the IT department Reduces risk and cost to business managers Accelerated analytical cycle R community Bug fixes patches Increased user base

22. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com The ValidR roadmap September 2014 EARL Conference Release testCoverage and functionMap on CRAN Easter 2015 Release of validated R 3.1.x Summer 2015 Release of validated additional packages

23. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com What does validation mean Assurance, evidence Why validate R Provide assurance, evidence The challenges of validating R Validation tools ValidR Methodology, software, documentation Benefits Summary

24. DRAFT Richard Chandler-Mant – R Consultant rchandler-mant@mango-solutions.com Questions?