1. Standard Scripts - Project 2 Proposal for Qualification July 2014 Project 2 Update

2. Main Sections Summary of prior proposal, 2013 Updated proposal, July 2014

3. Main Sections  Summary of prior proposal o Concepts, definitions & meta data o Test data considerations o Heavy vs. Light qualification Updated proposal

4. Proposal from 2013 http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt Anyone should be able to submit a script, according to a check list Categorize scripts according to complexity – Complexity:low, medium, high, software – Output:tabulated data, analysis data, table, figure, listing Metadata for script should indicate – Type of output:tabulated data, analysis data, table, figure, listing – Study design:parallel, crossover, etc – State of qualification:?

5. Proposal through CSS 2104 Test data – Overall project should have minimum test data (SDTM & ADaM) – Scripts can propose new test data, must pass (Data fit? Open CDISC?)Open CDISC – Share program to produce test data, never binary test data 2 levels of qualification to match script complexity/output – Light vs. Heavy qualification – Common elements include header good programming practices clearly declared scope of script (e.g., study design(s)) test data matches scope & passes "FDA Data Fit" assessment (?) documentation inputs/outputs/dependencies/usage

6. Proposal through CSS 2104 Heavy qualification – Beta package includesminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP – Draft Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

7. Proposal through CSS 2104 Light qualification – Beta package includesskip if >1 yr production use without ERROR – Draftminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

8. Proposal through CSS 2104 Peer Review ChecklistHeavyLight Requirement specificationX? Documented or perhaps only documented in headerX User GuideXX SDTM/ADaM used in input/outputXX Open CDISC validator or Data Fit used to check input/outputXX GPPGPP in sourceXX Run according to Requirement specificationX? Tested by qualification plan, tests & results all Peer reviewedX? Tested by End usersX? Robust without red errors in contributor's production environmentXX Robust and used in FDA (other) scripts repository, ranked ******X

9. Main Sections Summary of prior proposal  Updated proposal o Motivation & objectives, as justification for elements of proposal

10. Proposal 2014 Motivation End-user Objectives – Clear overview of resources available, and the purpose & state of each – Inspire confidence from first user experience – Ease of script use, clear messaging from first run of scripts – Reproducible results in user's own environment – Consistency of scripts, learning first one makes remaining familiar – Ease of converting users to contributors Contributor & Team Objectives – Clear, standardized workflows and checklists – Modularize routine components (e.g., FUTS for dependency checking?)FUTS – Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility ) concept similar to Spotfire/R compatibility – Centralize & consolidate information & results

11. Qualification Proposal meaningful terms in blue http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx Qualification Instructions (see embedded template  ) – Certification phase of Qualification applies to new scripts and tests – Confirmation phase applies to updates of existing scripts States:Contributed, Development, Testing, Qualified Roles – Contributor: Anyone with appropriate skills & interests – Developer: CSS Working Group 5 volunteer familiar with objectives** – Tester: CSS WG 05 volunteer familiar with objectives** – Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment – Reviewer: Author of white papers, designers of script targets** ** suggests a quick-start onboarding page in CSS Phusewiki

12. Proposal Qualification Metadata for script should indicate – Whitepaper ID & output ID – Programming language & version (e.g., R v3.1.1, SAS v9.4) – Type of output:tabulated data, analysis data, table, figure, listing – Study design:parallel, crossover, etc – State of qualification:Contributed, Development, Testing, Qualified – OS is not included, since should be indicated in OS compatibility report Test Data requirements – available as a program or script (text, not binary) – based on expected standards (SDTM, ADaM) – data requirements clearly & accurately specified for each script – include expected results from these data for defined tests/checks

13. Proposal Qualification Transitions"Contributed" is the original State of all scripts – to Development, checklist includesby Developer & Reviewer R & D confer on suitability of contribution. Suitable starting point? [ may require analysis details, specs, from contributor ] D reviews components (checklist to be completed) D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking D writes Qualification instructions.docx (see template, above)Qualification instructions.docx – to Testing, checklist includesby Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully

14. Proposal Qualification Transitionscontinued – to Qualifiedby Tester & Environment Tester & Reviewer T updates reference test outputs from certification/confirmation E updates & executes local tests (posting PASS/FAIL results) R confirms script output matches intention R confirms qualification process covers important elements and considerations. R also provides user (rather than technical) feedback? Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

15. Proposal Qualification Efforts Required – Top priority Finalize Qualification states, roles, workflow, checklists, and templates – Next priorities Design test structure in google code Develop scripts that will allow Environment Testing Develop general components (e.g. parameter, dependency checking) Develop general components Identify Environment Testers based on – Host environment – SAS or R version Identify opportunities to automate qualification. E.g., – Environment Testers to post results back as machine readable – Script green-light/red-light qualification matrix on Phusewiki