WG5 P02 Proposal 2014 Qualification of Standard Scripts

Proposal through CSS 2014 http://www. phusewiki. org/wiki/index. php Anyone should be able to submit a script, according to a check list Categorize scripts according to complexity Complexity: low, medium, high, software Output: tabulated data, analysis data, table, figure, listing Metadata for script should indicate Type of output: tabulated data, analysis data, table, figure, listing Study design: parallel, crossover, etc State of qualification

Proposal through CSS 2104 Test data Overall project should have minimum test data (SDTM & ADaM) Scripts can propose new test data, must pass (Data fit? Open CDISC?) Share program to produce test data, never binary test data 2 levels of qualification to match script complexity/output Light vs. Heavy qualification Common elements include header good programming practices clearly declared scope of script (e.g., study design(s)) test data matches scope & passes "FDA Data Fit" assessment (?) documentation inputs/outputs/dependencies/usage

Proposal through CSS 2104 Heavy qualification Beta package includes minimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproduced Draft Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) Test Peer Review: Write qualification report, incl. log/output from tests Final

Proposal through CSS 2104 Light qualification Beta package includes skip if >1 yr production use without ERROR Draft minimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproduced Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) Test Peer Review: Write qualification report, incl. log/output from tests Final

Proposal through CSS 2104 Peer Review Checklist Heavy Light Requirement specification X ? Documented or perhaps only documented in header User Guide SDTM/ADaM used in input/output Open CDISC validator or Data Fit used to check input/output GPP in source Run according to Requirement specification Tested by qualification plan, tests & results all Peer reviewed Tested by End users Robust without red errors in contributor's production environment Robust and used in FDA (other) scripts repository, ranked ******

Proposal meaningful terms in blue Qualification Certification applies to new scripts and tests Confirmation applies to updates of existing scripts States: Contributed, Development, Testing, Qualified Roles Contributor: Anyone with appropriate skills & interests Developer: CSS Working Group 5 volunteer** familiar with objectives Tester: CSS WG 05 volunteer** Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment Reviewer: Author of white papers, designers of script targets ** suggests an quick-start onboarding page in CSS Phusewiki

Proposal Qualification End-user Objectives Clear overview of purpose and resources Inspire confidence from first sight Ease of use, clear messaging from first run Consistency of scripts, learning first one makes remaining familiar Ease of converting users to contributors Contributor Objectives Standardize routine steps Modularize routine components Automate testing, issue identification Centralize & consolidate information & results

Proposal Qualification Transitions Contributed is the original State of all scripts to Development checklist includes by Developer & Reviewer D reviews components D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking R confirms contributed output matches/approximates target [ may require analysis details, specs, from contributor ] D writes Qualification instructions .docx (see template) to Testing by Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully

Proposal Qualification Transitions continued to Qualified by Tester & Environment Tester T updates posted test outputs from certification/confirmation E updates local tests and executes (posting PASS/FAIL results) Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed

Proposal Qualification Efforts Required Finalize Qualification states, roles, workflow and templates Design test structure in google code Develop scripts that will allow Environment Testing Develop general components (e.g. parameter, dependency checking) Identify Environment Testers based on Host environment SAS or R version Identify opportunities to automate qualification. E.g., Docx format for Qualification instructions is not easily machine readable Environment Testers to post results back as machine readable Script green-light/red-light qualification matrix on Phusewiki

Doing now what patients need next