1. CBIIT Quality Assurance Process Preston Wood NCI CBIIT Government Quality Representative (GQR) January 2014 RS

2. CBIIT Quality Assurance (QA) Process (1) Agenda Goals for Quality Assurance Quality Assurance Process QA Artifacts Support for Iterative Development QA Process Innovations 2

3. CBIIT Quality Assurance (QA) Process (2) Goals for Quality Assurance Requirements are verified Test Approach is appropriate Test coverage is adequate Cost and schedule is understood Testing can be completed on time Audit capability is ensured Confidence in releasing the application 3

4. CBIIT Quality Assurance (QA) Process (3) 4

5. CBIIT Quality Assurance (QA) Process (4) QA Artifacts Should be specified in early stage of the project. Test Plan Defines the test approach Outlines scope of tests Determines what time frame the project can be completed –Schedule in P6 to determine cost/time impact to project Identifies risks going into the project, and mitigation plan Requires sign-off and for any modification. 5

6. CBIIT Quality Assurance (QA) Process (5) Test Cases Step by step directions for the tester Contains verifications Part of comprehensive test approach Are living documents and updated. 6

7. CBIIT Quality Assurance (QA) Process (6) Requirements Traceability Matrix (RTM) Identifies each requirement specifically Identifies and matches test cases to requirements Shows level of planned test coverage for identified requirements. Is a living and updated document. 7

8. CBIIT Quality Assurance (QA) Process (7) QA Entrance Checklist Vital to a repeatable QA process Ensures not missing components to conduct testing: –Tier is provisioned and code is operational –List of features implemented, and known issues –Unit test report from Development –Deployment documentation –Installation guide –Test case review –RTM is complete –Tag accepted by QA (smoke test prior to formal tests) –QA Test Plan reviewed and signed-off Note: “No” is a possibility for an individual item, but has to be approved 8

9. CBIIT Quality Assurance (QA) Process (8) Test Report Conveys what was tested Contains variances on tests conducted Announces what new issues were found, fixed, or still exist Categorizes defects to show problem areas of the app Identifies risks to the project going forward, and mitigation plan Requires sign-off and is final. 9

10. CBIIT Quality Assurance (QA) Process (9) Updated Requirements Traceability Matrix (RTM) Includes additional test cases Shows completion of tests Shows final level of test coverage for identified requirements. 10

11. CBIIT Quality Assurance (QA) Process (10) QA Exit Checklist Vital to a repeatable QA process Ensures not missing components to complete the tier: Source code tag received List of defects fixed, features implemented Test cases and execution reporting Deployment documentation Installation guide Test cases list for Appscan Appscan and 508 compliance passed Note: “No” is a possibility for an individual item, but has to be approved 11

12. CBIIT Quality Assurance (QA) Process (11) Audit and Confidence The QA artifacts give us confidence in releasing the software to the next tier or Production because we have full audit capability to know: –What was tested –How it was tested –Why it was tested –Where it was tested –When it was tested –Who tested it 12

13. CBIIT Quality Assurance (QA) Process (12) And we know, Issues that are fixed, Issues that still exist, and What we are going to do about them. 13

14. CBIIT Quality Assurance (QA) Process (13) The most important aspect of this process is that it is repeatable. 14

15. CBIIT Quality Assurance (QA) Process (14) Grand Assumptions being made: All requirements are known from the start No new functionality is introduced after testing begins. What if new functionality is introduced while testing? –Not known before Test Plan written –Not a phased approach 15

16. CBIIT Quality Assurance (QA) Process (15) Challenges for QA: Test approach is now incomplete and unable to fully address the new functionality. Scope of testing may increase Project may exceed schedule or cost Project risks change and/or increase Risk mitigation plan is obsolete Project could be open to failure Audit capability compromised New project expectations may exceed QA capability 16

17. CBIIT Quality Assurance (QA) Process (16) A ripple effect will occur in the community if any challenge goes unanswered. Q: How do we address these challenges to reestablish confidence in releasing the application to Production? A: Implement support for Iterative Development 17

18. CBIIT Quality Assurance (QA) Process (17) 18 QA Support for Iterative Development Iterative Testability Assessment 1. New functional requirements are identified 2. QA conducts requirements testability assessment 3. QA determines if updates to test plan needed: Test Approach change needed? Test Scope and/or Environment change needed? Test Schedule and Milestones change needed?

19. CBIIT Quality Assurance (QA) Process (18) If all questions are “no” (typically bug fixes only), then New test cases are created RTM is updated as usual Testing resumes without changes to the Test Plan. 19

20. CBIIT Quality Assurance (QA) Process (19) If any question is “yes”, QA advises project group of the impact to the project. The project determines a cost assessment of the impact. Should the impact and cost be deemed acceptable, 1. The test plan is edited accordingly to account for the new requirements, 2. RTM is updated with new requirements and test cases 3. GQR, GPM, and GS give sign-off on the edits. 4. Then the new tag is accepted, and testing resumes for the new iteration. 20

21. CBIIT Quality Assurance (QA) Process (20) 21

22. CBIIT Quality Assurance (QA) Process (21) Benefits of the Iterative Testability Assessment Requirements are properly verified Audit capability is ensured Confidence in releasing the application is restored because… 22

23. CBIIT Quality Assurance (QA) Process (22) Benefits of the Iterative Testability Assessment New requirements were analyzed and documented. Test Approach was updated and comprehensive. New timeline and cost for QA was assessed and accounted for. Correct environment to test was confirmed. Test coverage was appropriate. So testing can be completed on time and on budget. 23

24. CBIIT Quality Assurance (QA) Process (23) Challenges to QA Support for Iterative Development Iterative development can be cost intensive –Restarting QA multiple times. –New functionality may invalidate previous verifications. –Completed tests may need to be re-run. Finding the balance point, cost analysis –More iteration/costs vs features to the community –Streamlining the sign-off process new JIRA Workflow Process –Tracking the project in P6 24

25. CBIIT Quality Assurance (QA) Process (24) QA Innovations and Focus on Individual Sub-Project Needs QA Process Modification –Sub-Projects deploying to Production tier before end of FY’13. –Critical fix deployment Funding running out Test Plan waived –assume previous test approach would be adequate Form Builder 4.0.3.1 critical fix –deployed from Dev to Production tier in two weeks 25

26. CBIIT Quality Assurance (QA) Process (25) QA Innovations and Focus on Individual Sub-Project Needs Implemented QA support for iterative development processes –QA can properly support sub-projects with multiple iterations containing new requirements –EPLC process can be maintained 26

27. CBIIT Quality Assurance (QA) Process (26) QA Innovations and Focus on Individual Sub-Project Needs QA Process Improvement –Review and grooming of QA artifact templates –One test plan –Research for a new appscan policy proposal –Improve workflow, transparency of process Adopt new JIRA workflow to track projects 27

28. CBIIT Quality Assurance (QA) Process (27) Recent QA Process Innovations: Review and Approval of Test Cases in HP Quality Center –Review, comments, approval Electronically approve QA Artifact PDF –For projects that request that level of tracking QA Document Generation by HP Quality Center –Test Case spreadsheet (alt. workflow for test case review) –RTM (needs requirement tracking in HP-QC) 28

29. CBIIT Quality Assurance (QA) Process (28) 29

30. CBIIT Quality Assurance (QA) Process (29) Reference material: CBIIT QA wiki site https://wiki.nci.nih.gov/display/CBIITQA CBIIT Engineering wiki site https://wiki.nci.nih.gov/display/CBIITeng/CBIIT+Engineering +SOPs 30

31. CBIIT Quality Assurance (QA) Process (30) Questions 31