We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published bySkyla Mayer
Modified about 1 year ago
Black Box Software Testing Copyright © Cem Kaner & James Bach 1 Black Box Software Testing Spring 2005 Part 4 -- QUALITY COST ANALYSIS by Cem Kaner, J.D., Ph.D. Professor of Software Engineering Florida Institute of Technology and James Bach Principal, Satisfice Inc. Copyright (c) Cem Kaner & James Bach, This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. These notes are partially based on research that was supported by NSF Grant EIA ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Black Box Software Testing Copyright © Cem Kaner & James Bach 2 Suggested Reading: Kaner, Quality Cost Analysis: Benefits & Risks. Advocating for fixes by alerting people to quality related costs
Black Box Software Testing Copyright © Cem Kaner & James Bach 3 Money Talks: Cost of Finding & Fixing Errors This curve maps the traditionally expected increase of cost as you find and fix software errors later and later in development.
Black Box Software Testing Copyright © Cem Kaner & James Bach 4 Money Talks: Cost of Finding & Fixing Errors This is the most commonly taught cost curve in software engineering. Usually people describe it from the developers-eye view. That is, the discussion centers around how much it costs to find the bug how much it costs to fix the bug and how much it costs to distribute the bug fix. But sometimes, it pays to adopt the viewpoints of other stakeholders, who might stand to lose more money than the development and support organizations.
Black Box Software Testing Copyright © Cem Kaner & James Bach 5 Money Talks: Cost of Finding & Fixing Errors From the programmers’ viewpoints: if we find bugs in requirements, we can fix them without having to recode anything; programmers who find their own bugs can fix them without taking time to file bug reports or explain them to someone else; it is hugely expensive to deal with bugs in the field (in customers’ hands). Consider too the effects on other stakeholders. Costs escalate because more people in and out of the company are affected by bugs, and more severely affected, as the product gets closer to release. For example, think of the marketing assistant who wastes days trying to create a demo, but can’t because of bugs.
Black Box Software Testing Copyright © Cem Kaner & James Bach 6 Money Talks: Cost of Finding & Fixing Errors It is important to recognize that this cost curve is predicated on a family of development practices. When you see a curve that says, “Late changes are expensive” you can reasonably respond in either of two ways: Make fewer late changes. This is the traditional recommendation Make it cheaper to make late changes. This is a key value of the agile development movement (see Beck’s Extreme Programming Explained, or go to In this course, I push you to find ways to find bugs earlier, but my development philosophy is agile.
Black Box Software Testing Copyright © Cem Kaner & James Bach 7 Quality Cost Analysis Quality Cost Measurement is a cost control system used to identify opportunities for reducing the controllable quality- related costs The Cost of Quality is the total amount the company spends to achieve and cope with the quality of its product. This includes the company’s investments in improving quality, and its expenses arising from inadequate quality. A key goal of the quality engineer is to help the company minimize its cost of quality. »Refer to the paper, Quality Cost Analysis: Benefits & Risks.
Black Box Software Testing Copyright © Cem Kaner & James Bach 8 Quality-Related Costs PreventionAppraisal Cost of preventing customer dissatisfaction, including errors or weaknesses in software, design, documentation, and support. Cost of inspection (testing, reviews, etc.). Internal FailureExternal Failure Cost of dealing with errors discovered during development and testing. Note that the company loses money as a user (who can’t make the product work) and as a developer (who has to investigate, and possibly fix and retest it). Cost of dealing with errors that affect your customers, after the product is released.
Black Box Software Testing Copyright © Cem Kaner & James Bach 9 Examples of Quality Costs PreventionAppraisal Staff training Requirements analysis & early prototyping Fault-tolerant design Defensive programming Usability analysis Clear specification Accurate internal documentation Pre-purchase evaluation of the reliability of development tools Design review Code inspection Glass box testing Black box testing Training testers Beta testing Usability testing Pre-release out-of-box testing by customer service staff Internal FailureExternal Failure Bug fixes Regression testing Wasted in-house user time Wasted tester time Wasted writer time Wasted marketer time Wasted advertisements Direct cost of late shipment Opportunity cost of late shipment Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Supporting multiple versions in the field Refunds, recalls, warranty, legal costs Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers
Black Box Software Testing Copyright © Cem Kaner & James Bach 10 Customers’ Quality Costs Seller: external costsCustomer: failure costs (seller’s externalized costs) These illustrate costs absorbed by the seller that releases a defective product. Lost sales and lost customer goodwill Technical support calls Writing answer books (for Support) Investigating complaints Refunds, recalls, warranty, legal costs Government investigations Supporting multiple versions in the field Interim bug fix releases Shipping updated product PR to soften bad reviews Discounts to resellers These illustrate costs absorbed by the customer who buys a defective product. Wasted time Lost data Lost business Embarrassment Frustrated employees quit Failure during one-time-only tasks, e.g. demos to prospective customers Cost of replacing product Reconfiguring the system Cost of recovery software Tech support fees Injury / death
Black Box Software Testing Copyright © Cem Kaner & James Bach 11 Influencing Others Based on Costs It’s often impossible to fix every bug. Sometimes the development team will choose to not fix a bug based on their assessment of its risks for them, without considering costs to other stakeholders. –Probable tech support cost. –Risk to the customer. –Risk to the customer’s data or equipment. –Visibility in an area of interest to reviewers. –Extent to which the bug detracts from the use of the program. –How often will a customer see it? –How many customers will see it? –Does it block any testing tasks? –Degree to which it will block OEM deals or other sales. To argue against a deferral, ask yourself which stakeholder(s) will pay the cost of keeping this bug. Flag the bug to them.
Black Box Software Testing Copyright © Cem Kaner & James Bach 1 Black Box Software Testing Fall 2005 Overview—Part 2 (Mission of Testing) Cem Kaner,
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 PART USER TESTING by Cem Kaner, J.D., Ph.D.
About Quality Pre paired By: Muhammad Azhar. Scope What is Quality Quality Attributes Conclusion on software Quality Quality Concepts Quality Costs.
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Spring 2005 PART 7 -- FUNCTION TESTING by Cem Kaner, J.D.,
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing 2004 Academic Edition Part EDITING BUGS by Cem Kaner,
Session # Rational User Conference 2002 Author Note: To edit Session # go to: View/Master/Title Master ©1998, 1999, 2000, 2001, 2002 Rational Software.
1 PROJECT TASK NUMBER 5 After reading this document (http://www.kaner.com/qualcost.htm), apply these concepts to the design (and production) of the product.
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 PART PAIRED EXPLORATORY TESTING by Cem.
Copyright (c) Cem Kaner. All Rights Reserved. 1 Black Box Software Testing (Professional Seminar) Cem Kaner, J.D., Ph.D. Professor of Computer.
Copyright (c) Cem Kaner Black Box Software Testing (Academic Course - Fall 2001) Cem Kaner, J.D., Ph.D. Florida Institute of Technology Section:
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Spring 2005 PART 8 -- TEST DESIGN by Cem Kaner, J.D., Ph.D.
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 PART REGRESSION TESTING by Cem Kaner, J.D.,
IS 214 Needs Assessment and Evaluation of Information Systems Managing Usability © Copyright 2001 Kevin McBride.
Approaches to ---Testing Software Some of us “hope” that our software works as opposed to “ensuring” that our software works? Why? Just foolish Lazy Believe.
Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 Part Exercises by Cem Kaner, J.D., Ph.D.
Getting Permission When Your Use is Not Fair Slides produced by the Copyright Education & Consultation Program.
Testing E001 Access to Computing: Programming. 2 Introduction This presentation is designed to show you the importance of testing, and how it is used.
Software Testing. Overview Definition of Software Testing Problems with Testing Benefits of Testing Effective Methods for Testing.
CS3100 Software Project Management Week 26 - Quality Dr Tracy Hall.
Testing and Cost / Benefit Tor Stålhane. Why cost / benefit – 1 For most “real” software systems, the number of possible inputs is large. Thus, we can.
Black Box Software Testing Copyright © Cem Kaner & James Bach 1 Black Box Software Testing Fall 2005 Overview—Part 3 (Test oracles) Cem Kaner,
Quality-Cost Analysis Benefits and Risks Cem Kaner.
Software Defects 1. 2 What is a Software Defect? A software defect is an error, flaw, mistake, failure, or fault in software that prevents it from behaving.
© Pearson Education Copying permitted for purchasing institution only. This material is not copyright free. Functional areas Unit 1: Investigating.
Level 2 IT Users Qualification – Unit 1 Improving Productivity Name.
M ARKETING 1.02B Explain the role of customer service as a component of selling relationships.
Dtengineering 1 of 16 The Product Development Process Introduction.
Copyright (c) Cem Kaner. 1 Software Testing 1 CSE 3411 SWE 5411 Assignment #1 Replicate and Edit Bugs.
IT in the Real World A look at IT in a Fortune 500 company Ed Nelson.
Black Box Software Testing Copyright © Cem Kaner & James Bach 1 Black Box Software Testing Fall 2005 Overview for Students Cem Kaner, J.D., Ph.D.
Ethics of Software Testing Thomas LaToza CS 210 Final Presentation 12 / 2 / 2002.
Software Project Management Lecture # 10. Outline Quality Management (chapter 26) What is quality? Meaning of Quality in Various Context Some quality.
T.1-2 DESCRIPTION OF GENERAL DETAILS OF THE FIRM AND ORGANIZATION CHART T. 3 STRATEGY FOLLOWED TO GET A COMETITIVE ADVANTAGE T. 4-7 QUALITY FUNCTION DEPLOYMENT.
Chapter 8 Implementing Quality Concepts Cost Accounting Traditions and Innovations Barfield, Raiborn, Kinney.
Chapter 3 The Purchasing Function. THE PURCHASING DEPARTMENT’S FUNCTION IS TO: The three most important elements of the purchasing department’s function.
Dr Andy Brooks1 FOR0383 Software Quality Assurance Lecture 1 Introduction Forkröfur/prerequisite: FOR0283 Programming II Website:
Copyright © 2012 Pearson Education, Inc. All rights reserved. Chapter 4 The Ethics of Manufacturing and Marketing.
Agile Testing and/or Agility in Testing? Juha Itkonen SoberIT Teknillinen Korkeakoulu Test Summit, , Savonlinna.
Conducting Usability Tests ITSW 1410 Presentation Media Software Instructor: Glenda H. Easter.
Stoimen Stoimenov QA Engineer SitefinityLeads, SitefinityTeam6 Telerik QA Academy Telerik QA Academy.
Agile Methods in a Product Development Setting © 2002 Reginald Braithwaite- Lee lee.com lee.com.
1 Day-to-day planning – “Old school” How planning leads to success Parts of Phillips, Ch 4 CSSE579 Session 5 Part 1.
IB Business and Management Measuring the effectiveness of the Workforce.
Chapter 8: Maintenance and Software Evolution Ronald J. Leach Copyright Ronald J. Leach, 1997, 2009, 2014,
March 13, 2001CSci Clark University1 CSci 250 Software Design & Development Lecture #15 Tuesday, March 13, 2001.
© 2017 SlidePlayer.com Inc. All rights reserved.