Presentation is loading. Please wait.

Presentation is loading. Please wait.

Author: Jan Fish Division: Philips Lifeline IT September, 2008 Measure Quality on the Way In – Not Just on the Way Out.

Similar presentations


Presentation on theme: "Author: Jan Fish Division: Philips Lifeline IT September, 2008 Measure Quality on the Way In – Not Just on the Way Out."— Presentation transcript:

1 Author: Jan Fish Division: Philips Lifeline IT September, 2008 Measure Quality on the Way In – Not Just on the Way Out

2 2 Philips Lifeline J. Fish For Your Consideration Traditional Measurements for Test Organizations Value Added Opportunities 4 Phase Approach –Target versus Actual Progress –Bug Patterns within Builds / Deployments –Workload Assessment for Outstanding Bugs –Bug Injection Points and Bug Removal Points Measure Quality on the Way In; Not Just on the Way Out

3 3 Philips Lifeline J. Fish Traditional Measurements for Test Organizations TEST METRICS:  Standard of measurement  Gauge effectiveness and efficiency  Gathered and interpreted throughout the test effort  Objectively measure success PHILOSOPHY:  Keep it simple  Make it meaningful  Track it  Use it

4 4 Philips Lifeline J. Fish Traditional Measurements for Test Organizations BASE METRICS may include numbers and/or percentages for: - Test cases created- Test cases in review - Test cases to be executed- Test cases re-executed - Test cases executed - Total executes - Test cases passed- Total test cases passed - Test cases failed- Total test cases failed - Test cases blocked- Defect removal cost - Bad test cases- Bad fixes - Defects corrected- Test effectiveness (QA / QA + Prod)

5 5 Philips Lifeline J. Fish Value Added Opportunities Forecast, Track and Respond:  Inputs: Number of Test Cases, Testers and Cycle Time First Run Failure Rate:  Inputs: Failed Test Cases / Executed Test Cases What is the Pattern:  Inputs: Known Patterns vs. Current Pattern Workload Distribution:  Inputs: Number and Type of Bugs in DEV, QA and Resolved Bug Injection and Removal Points:  Inputs: Error Created and Error Found Plan It, Track It and Graph It:  Inputs: Actual Progress to Targeted Goal

6 6 Philips Lifeline J. Fish Target vs. Actual – Existing Project Target Date Testcases Executed Testcases Passed Testcases Failed Failure Rate (%) PlanActualPlanActualPlanActualForecastActual Week %15.8% Week %41.8% Week %100% Running total %43.3% 43% : target

7 7 Philips Lifeline J. Fish Target Planned vs. Passed vs. Failed Test Cases

8 8 Philips Lifeline J. Fish New and Improved Target versus Actual Target Date Test Cases Executed Test Cases Passed Test Cases Failed RetestsFailure Rate (%)% Done TargetActualTargetActualTargetActualPendingDoneForecastActualTargetActual Smoke Tests 6/ %33% Running total %33%9% Functional Tests 6/ %100% Running total %67%21%18% 6/ %14% Running total %38% Exploratory Tests 6/ % Running total %33%35%44% Load Tests 6/ % Running total %28%35%53% Regression 6/ %7% Running total %18%79%97% 6/ % Running total %18%82%100%

9 9 Philips Lifeline J. Fish New and Improved Chart

10 10 Philips Lifeline J. Fish Target vs. Actual Math Target Test Cases Executed math = Target Test Cases Passed + Target Test Cases Failed Actual Test Cases Executed math = Actual Test Cases Passed + Actual Test Cases Failed Pending Retests math = Previous Pending Retests + Actual Test Cases Failed - Retests Done Running Total math = Actual Values + Previous Reported Values Forecast Failure Rate math = Target Test Case Failed / Target Test Cases Executed Actual Failure Rate math = Actual Test Case Failed / Actual Test Cases Executed Forecast % Done math = Current Target (Passed + Failed) / Final Target Test Cases Executed Actual % Done math = Current Actual (Passed + Failed) / Final Target Test Cases Executed

11 11 Philips Lifeline J. Fish The Added Values of Target vs. Actual TRACK to plan and REACT immediately  Is quality built in  Is the build / deployment truly ready for test  Is the test resource on schedule with each type of test execution  Functional  Regression  Load / Performance / Security  Bug Fix Validation

12 12 Philips Lifeline J. Fish The Added Values of Target vs. Actual PREDICT how many test cases should be run in a given time period To date, our First Run Failure Rates are:  7% - 12% maintenance of existing functionality  20% - 35% added functionality to existing Application  30% - 47% new Application developed on-site  10% - 25% new Application developed off-site

13 13 Philips Lifeline J. Fish The Added Values of Target vs. Actual ADJUST time and / or staff on an immediate basis  Substantiate “Gut Feel”  Demonstrate facts  Moderate re-work and know if it fits the schedule  Adjust the plan and determine the level of effort well before the last quadrant of the test cycle

14 14 Philips Lifeline J. Fish The Added Values of Target vs. Actual SET realistic, track-based entrance criteria (as agreed upon with upstream partners) for what is or is not acceptable quality at the start of the project USE the results to document expectations into project contracts (internal or external)  Establish what constitutes ”acceptable quality level”  Set contract conditions for scaled fees based on exceeding, meeting or failing quality objectives

15 15 Philips Lifeline J. Fish The Added Values of Target vs. Actual PUBLISH and POST printed copy in a common area  Eliminate the arguments  Eliminate the negative chatter  Document facts; don’t point fingers  Grid each application/area Failure Rates in like-sets  Review to determine likely process improvements  Continue to track implemented improvement and assess if it had a positive or negative impact

16 16 Philips Lifeline J. Fish Bug Pattern Recognition – Sample 1 Build Name DateCriticalHighMediumLowTotal Running Total b1 02/01 02/06 02/ b2 02/15 02/20 02/22 02/ b3 02/ TOTALS

17 17 Philips Lifeline J. Fish Pattern Recognition – Existing Project 2 Build Date Date CHMLTTRun TT 02/04/200802/05/ /07/ /11/200802/12/ /14/ /19/200802/20/ /22/ /25/200802/26/ /28/ TOTAL710088

18 18 Philips Lifeline J. Fish The Added Values of Pattern Recognition  INSTITUTE a process change and know the effect  INCREASE the precision of estimates  IDENTIFY trends and observations supporting or hindering test and project plan  INFORM team and management of patterns found in a crisp, clean and simple manner such that the whole team can make “next step” decisions

19 19 Philips Lifeline J. Fish Bug, Bug, who‘s got the Bug? Bugs in DevBugs in QA Pending Development Fix Pending QA Retest Resolved CHMLTTR ev FixTTCRejFutTT 11/ / /

20 20 Philips Lifeline J. Fish The Added Values of Bug Reporting by Group  WORKLOADS can be easily identified by management and team members  INFORMED DECISIONS can be made on next steps without drilling into details (who, when, what, how and where)  RE-WORK can be tracked and compared to planned work  PROGRESS toward resolving outstanding bugs

21 21 Philips Lifeline J. Fish Bug Injection/Removal Points

22 22 Philips Lifeline J. Fish Added Values of Bug Injection & Removal Points  BASIS for estimating the number of bugs to be found at each phase of the Software Development Lifecycle  IDENTIFIED work cycles that would benefit from process improvements and inspection points  VALIDATE that improvements work  DEMONSTRATE that quality must be the goal of all team members, not just the responsibility of the test organization “Quality is never an accident; it is always the result of intelligent effort.” * * John Ruskin, English writer and critic of art, architecture and society

23 23 Philips Lifeline J. Fish Measure Quality on the Way In; Not Just on the Way Out PEOPLE do not intentionally make a bad plan but they may not be able to quickly adjust the plan to current circumstances and abate risks PEOPLE do look for paths of least resistance and, if it is easy to blame others, they will MEASURING quality once product is in production tells the tale of what was not trapped and fixed but not the tale of quality before production QUALITY can not be tested into a product but you can measure quality coming into your test organization FOCUS on the level of quality at the project level


Download ppt "Author: Jan Fish Division: Philips Lifeline IT September, 2008 Measure Quality on the Way In – Not Just on the Way Out."

Similar presentations


Ads by Google