Jama Options for Complex Testing variations

Test Run vs Test Case Status Test Plan Test Case Test Case Test Case Test Case Test Case Test Case Test Case Test Run Cycle 1 Blocked Test Run Cycle 2 Failed Test Run Cycle 3 Passed

Test Run vs Test Case Status Test Plan Test Case Test Case Test Case Test Case Test Case Test Case Test Case Test Case status shows is the most recent Test Run status Test Run Cycle 1 Blocked Test Run Cycle 2 Failed Test Run Cycle 3 Passed

Complex Testing Scenario Operating Systems Mac OSX Windows 7 Windows 8 Browser Internet Explorer (MS only) Safari (Mac only) Chrome Firefox Browser Version Internet Explorer 11, 10, 9 Safari 7, 6, 5 Chrome 33, 32, 31 Firefox 28, 27, 26 Each test needs to be run against each variation above This equals 27 different variations for each TC

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation

Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation PRO Report on the true % of passing vs failing test cases Multiple cycles and test runs per variation Tweak test case per variation See the real complexity of your testing in Coverage view Filter TCs based on variation and see roll-up stats Can organize test cases by variation AND functional area (e.g. Login tests, Email tests, etc) CON Longer setup effort: author/import test cases once - then reuse. Each time a new test cycle begins, Test Manager must create a new cycle per variation (27 in this sample) For any new test cases added must reuse into each set and then add to 27 test plans Users must constantly switch between Test Plans

Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation

Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation PRO Report on the true % of passing vs failing test cases Multiple cycles and test runs per TC variation Tweak test case per variation See the real complexity of your testing in Coverage view Filter TCs based on variation and see roll-up stats Same authoring/import effort as Opt1 but less Test Execution setup time Organize everything in a Single Test Plan Managers can add a new cycle for all variations in one process Testers have a single place to see all test variations Can still assign by variation (e.g. Becky does all Mac OS Safari 7 tests) CON Cannot organize test cases by functional area, only by variation For any new test cases added must reuse into each folder

Option 3 – Generic TC, Unique Test Plans

Option 3 – Generic TC, Unique Test Plans PRO Less setup time to author/import test cases Report on the true % of passing vs. failing test runs Organize and assign testing by variation (e.g. Becky - Mac OSX Safari 7 tests) CON Same Test Execution setup effort as Option 1 Test Manager must create 27 new cycles, one per variation Test Manager must update 27 cycles if a new test case is added Cannot see total complexity in Coverage Explorer view No TC variations per OS/Browser Different test run results may conflict the roll-up test case status. Example: Test in Chrome, Fails. Test in IE, Passes. Overall Test Case Status will be Passed

Option 4 – Generic TC and Runs

Option 4 – Generic TC and Runs PRO Quick setup time Simple management when updating or adding a Test Case (just add once) Defects are the source of what is failing on which OS/Browser ? Organize and assign by test, not by variation (e.g. Becky does single test for ALL OS and Browser versions) CON When a test fails, hard show which variations passed vs failed. Cannot show true metrics of % passed vs % failed Test Cases do not allow for variations depending on the browser, os, etc Cannot see total complexity in Coverage Explorer view Bug relationships don’t go away upon new Test Cycles so bugs from old cycles will still show in Coverage View

LOE Option 1 Option 2 Option 3 Option 4 Test Authoring Medium Easy Unique TC Unique Test Plans Option 2 Single Plan, Unique Test Groups Option 3 Generic TC Option 4 Generic Test Runs Test Authoring Add Import Update Medium Easy Test Planning Add Cycle New Release Assign Update Plans w/ new Test Cases High Test Execution Execute Log Defects Test Result Reporting % Pass/Fail Traceability