1. WG5 P02 Proposal 2014 Qualification of Standard Scripts

2. Proposal through CSS 2014 http://www. phusewiki. org/wiki/index. php
Anyone should be able to submit a script, according to a check list
Categorize scripts according to complexity
Complexity: low, medium, high, software
Output: tabulated data, analysis data, table, figure, listing
Metadata for script should indicate
Type of output: tabulated data, analysis data, table, figure, listing
Study design: parallel, crossover, etc
State of qualification

3. Proposal through CSS 2104 Test data
Overall project should have minimum test data (SDTM & ADaM)
Scripts can propose new test data, must pass (Data fit? Open CDISC?)
Share program to produce test data, never binary test data
2 levels of qualification to match script complexity/output
Light vs. Heavy qualification
Common elements include
header
good programming practices
clearly declared scope of script (e.g., study design(s))
test data matches scope & passes "FDA Data Fit" assessment (?)
documentation inputs/outputs/dependencies/usage

4. Proposal through CSS 2104 Heavy qualification
Beta package includes minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Draft
Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
Test
Peer Review: Write qualification report, incl. log/output from tests
Final

5. Proposal through CSS 2104 Light qualification
Beta package includes skip if >1 yr production use without ERROR
Draft minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
Test
Peer Review: Write qualification report, incl. log/output from tests
Final

6. Proposal through CSS 2104 Peer Review Checklist Heavy Light
Requirement specification X ?
Documented or perhaps only documented in header
User Guide
SDTM/ADaM used in input/output
Open CDISC validator or Data Fit used to check input/output
GPP in source
Run according to Requirement specification
Tested by qualification plan, tests & results all Peer reviewed
Tested by End users
Robust without red errors in contributor's production environment
Robust and used in FDA (other) scripts repository, ranked ******

7. Proposal meaningful terms in blue Qualification
Certification applies to new scripts and tests
Confirmation applies to updates of existing scripts
States: Contributed, Development, Testing, Qualified
Roles
Contributor: Anyone with appropriate skills & interests
Developer: CSS Working Group 5 volunteer** familiar with objectives
Tester: CSS WG 05 volunteer**
Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment
Reviewer: Author of white papers, designers of script targets
** suggests an quick-start onboarding page in CSS Phusewiki

8. Proposal Qualification
End-user Objectives
Clear overview of purpose and resources
Inspire confidence from first sight
Ease of use, clear messaging from first run
Consistency of scripts, learning first one makes remaining familiar
Ease of converting users to contributors
Contributor Objectives
Standardize routine steps
Modularize routine components
Automate testing, issue identification
Centralize & consolidate information & results

9. Proposal Qualification
Transitions Contributed is the original State of all scripts
to Development checklist includes by Developer & Reviewer
D reviews components
D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ]
D adds standard parameter, dependency checking
R confirms contributed output matches/approximates target [ may require analysis details, specs, from contributor ]
D writes Qualification instructions .docx (see template)
to Testing by Tester
Review Qualification instructions, consider coverage of tests
Execute Qualification instructions
Work with Developer to complete execution successfully

10. Proposal Qualification
Transitions continued
to Qualified by Tester & Environment Tester
T updates posted test outputs from certification/confirmation
E updates local tests and executes (posting PASS/FAIL results)
Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed

11. Proposal Qualification
Efforts Required
Finalize Qualification states, roles, workflow and templates
Design test structure in google code
Develop scripts that will allow Environment Testing
Develop general components (e.g. parameter, dependency checking)
Identify Environment Testers based on
Host environment
SAS or R version
Identify opportunities to automate qualification. E.g.,
Docx format for Qualification instructions is not easily machine readable
Environment Testers to post results back as machine readable
Script green-light/red-light qualification matrix on Phusewiki

12. Doing now what patients need next