By: David Hoang Martin Hoffman Measuring the Economic Impacts of an Inadequate Infrastructure for Software Testing By: David Hoang Martin Hoffman

Improved infrastructure can… Remove more bugs Detect bugs earlier Locate bugs faster 2

The longer it takes the more is costs… 3

What does this mean for the User? Fewer bugs in software operations stage Lower maintenance costs Change in avoidance activities 9

Custom vs. Commercial Software Products Custom Products: no formal release more resources needed for implementation (developer) post-purchase service (developer) 3rd party integrators To quantify the cost associated with an inadequate infrastructure for software testing we look at the costs of custom and commercial products. This is to prevent double counting. Release is different from deliverable

Estimating Software Developer Costs Data collected: Company’s total pre-release testing and post-release service costs. Surveyed developers. Additional resources required due to an Inadequate infrastructure for software testing: Labor costs Software costs Hardware costs After-sales service cost Delay costs Reputation costs Labor costs – additional employee and contract labor for pre-purchase testing and error correction, installation, and post-purchase repair - accounts for the majority of the impact Software – additional or redundant testing software purchases Hardware – equipment, computers needed in testing process After-sales – non-testing and debugging activities: servicing additional service calls and distribution of patches Delay costs – lost profits and delayed adoption du to delays and high bug counts in early versions Reputation – lost sales or market share

Estimating Software User Costs Software purchase decision Evaluation of products Lack of information on the quality of products Unable to compare across products based on price and quality uncertainty of bugs and interoperability increases Installation and Integration Higher for custom products Business operations System failures Installation and integration of custom products require more frequent integration with legacy systems. Bug or problems in either the new or legacy system can delay the system, require additional patches or drivers Potential for additional cost is very high here. Can involve both the user and developer. Most commercial SW products are typically less then custom sw. Business operations cost of loss of production, loss of data, customer information, lost sales, production delays, lost reputation, market share

Software User Cost Categories labor costs failure costs performance cost redundant systems delayed profits sales forfeited Labor – additional employees, contract labor. Testing installation, repair of software, testing sw before purchaing Failure – cost catastrophic failure of system/sw Performance – inpact on user, cost of work-arounds, loss of productivity when sw does not perform as expected Redundant – need for additional hw or sw to maintain support or back-up data in case of sw failture Common to maintain parallel system for 1+ year as a security measure. Delayed – delays in production and transactions Sales – lost transactions, discounted value.

Estimating Software Bug Costs Cost of bugs still remaining with improved sw testing infrastructure. Estimate the total cost of bugs to the user Determine linearly related costs and nonlinearly related costs User inputs: total costs associated with bugs general change in total costs in a 75% reduction in bugs Cost of product failure or installation is linear with the number of bugs Redundant system costs is not linear. Difference in cost reduction is industry dependant. Redundant systems costs – HW costs do not decrease linearly. Need for backup systems is not significantly reduced.

Estimating Software Bug Costs

Industry-specific User Costs Industries encounter different types of costs Two industries were selected: Financial services – FEDI/clearinghouse software users Transportation equipment manufacturing Quality Categories: Functionality Reliability Usability Efficiency Maintainability Portability FEDI - Financial Electronic Data Interchange

Questions?