1. Jama Options for Complex Testing variations

2. Test Run vs Test Case Status
Test Plan
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Run
Cycle 1
Blocked
Test Run
Cycle 2
Failed
Test Run
Cycle 3
Passed

3. Test Run vs Test Case Status
Test Plan
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case status shows is the most recent Test Run status
Test Run
Cycle 1
Blocked
Test Run
Cycle 2
Failed
Test Run
Cycle 3
Passed

4. Complex Testing Scenario
Operating Systems
Mac OSX
Windows 7
Windows 8
Browser
Internet Explorer (MS only)
Safari (Mac only)
Chrome
Firefox
Browser Version
Internet Explorer 11, 10, 9
Safari 7, 6, 5
Chrome 33, 32, 31
Firefox 28, 27, 26
Each test needs to be run against each variation above
This equals 27 different variations for each TC

5. Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation

6. Option 1 – Unique TC per variation Reuse and Sync TCs + unique Test Plan per Variation
PRO
Report on the true % of passing vs failing test cases
Multiple cycles and test runs per variation
Tweak test case per variation
See the real complexity of your testing in Coverage view
Filter TCs based on variation and see roll-up stats
Can organize test cases by variation AND functional area (e.g. Login tests, tests, etc)
CON
Longer setup effort: author/import test cases once - then reuse.
Each time a new test cycle begins, Test Manager must create a new cycle per variation (27 in this sample)
For any new test cases added must reuse into each set and then add to 27 test plans
Users must constantly switch between Test Plans

7. Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation

8. Option 2 – Unique TC per Variation Single test plan - use Folders and Test Groups for Variation
PRO
Report on the true % of passing vs failing test cases
Multiple cycles and test runs per TC variation
Tweak test case per variation
See the real complexity of your testing in Coverage view
Filter TCs based on variation and see roll-up stats
Same authoring/import effort as Opt1 but less Test Execution setup time
Organize everything in a Single Test Plan
Managers can add a new cycle for all variations in one process
Testers have a single place to see all test variations
Can still assign by variation (e.g. Becky does all Mac OS Safari 7 tests)
CON
Cannot organize test cases by functional area, only by variation
For any new test cases added must reuse into each folder

9. Option 3 – Generic TC, Unique Test Plans

10. Option 3 – Generic TC, Unique Test Plans
PRO
Less setup time to author/import test cases
Report on the true % of passing vs. failing test runs
Organize and assign testing by variation (e.g. Becky - Mac OSX Safari 7 tests)
CON
Same Test Execution setup effort as Option 1
Test Manager must create 27 new cycles, one per variation
Test Manager must update 27 cycles if a new test case is added
Cannot see total complexity in Coverage Explorer view
No TC variations per OS/Browser
Different test run results may conflict the roll-up test case status. Example:
Test in Chrome, Fails.
Test in IE, Passes.
Overall Test Case Status will be Passed

11. Option 4 – Generic TC and Runs

12. Option 4 – Generic TC and Runs
PRO
Quick setup time
Simple management when updating or adding a Test Case (just add once)
Defects are the source of what is failing on which OS/Browser
? Organize and assign by test, not by variation (e.g. Becky does single test for ALL OS and Browser versions)
CON
When a test fails, hard show which variations passed vs failed.
Cannot show true metrics of % passed vs % failed
Test Cases do not allow for variations depending on the browser, os, etc
Cannot see total complexity in Coverage Explorer view
Bug relationships don’t go away upon new Test Cycles so bugs from old cycles will still show in Coverage View

13. LOE Option 1 Option 2 Option 3 Option 4 Test Authoring Medium Easy
Unique TC
Unique Test Plans
Option 2
Single Plan, Unique Test Groups
Option 3
Generic TC
Option 4
Generic Test Runs
Test Authoring
Add
Import
Update
Medium
Easy
Test Planning
Add Cycle
New Release
Assign
Update Plans w/ new Test Cases
High
Test Execution
Execute
Log Defects
Test Result Reporting
% Pass/Fail
Traceability