The Problem with Good Enough Software Bj Rollison Test Training Manager Microsoft, Inc.

Overview Discuss the intent and the reality of good enough software Test engineering…where we are and where should we go

What is software quality? Three properties of software quality  Features The cool stuff that makes customers want the software  Attributes Functionality, security, usability, reliability, maintainability  Defects The bugs that make users wish they bought something else  Defects =  Quality There is no such thing as defective, high-quality software

Good enough software Utilitarian view of quality – potential positive consequences outweigh potential negative consequences “…powerful alternative to the orthodox, expensive, and boring methodologies that aim at the best possible quality.” “…secret to the success of Microsoft…”

Utilitarian view in practice Good enough: Features exceed defects 80/20 rule: Defects actually exceed features Defects Features

Development cycles in practice Hack-and-whack approach Projects that rely on testing spend a lot of time and money on testing Orthodox, boring engineered approach Projects that rely on engineering practices use test to validate quality Plan? Code 1 defects / 10 LOC Test Random, Subjective Plan Detailed plan Reviewed Modeled Code Inspected Reviewed Unit tested Test Focused

Microsoft’s success!!! Complex products/ unwanted features Bugs, Security Patches & Service Packs Un-needed/unwanted upgrades Unpredictable, reactionary

The reality of good enough software Tends to focus on features & schedule  PM’s/Dev’s are rewarded for features not quality Heavy investment in testing to validate quality  Reliance on testing to find defects  You can’t test in quality Necessitates sustained engineering  There’s never enough time to do it right the first time, but there’s always enough time to fix it.

Features vs. Quality “If Microsoft is not here in 10 years it won’t be because of a lack of features, it will be due to a lack of quality!” – Craig Zhou, Microsoft Problems in the process  Undefined features  Feature creep  Schedule driven releases

Testing Effectiveness 50% Bugs Testing Ideal Testing Actual

Sustained Engineering Costs Microsoft issued 60 patches for Windows 2000 in 2001 Security patches cost an enterprise customer about $300,000 to install It costs Microsoft about $100,000 per patch Windows XP SP 1 – risk free, but why bother Maintenance has become similar to shipping a product at Microsoft.

The changing role of test engineering

Hack – n - Whack Approach Program Mgt. Development Testing Non-detailed specs Write code Integration testing Unit testing testing System

Engineering Approach Program Mgt. Development Testing Detailed specs Write code Integration testing Unit testing System testing Model

Role of Software Test Engineer Validation  “Establishing confidence that a program…does what it is supposed to do.” – Hetzel, 1983 Falsification  “Executing a program or system with the intent of finding errors.” – Myers, 1979 Achieve testable designs  Spec reviews, modeling, writing hooks in code, etc. Provide information to developers  More than bugs, code reviews, inspections, metrics, etc. Provide information to management  “The penultimate objective of testing is to gather management information.” – Beizer, 1995

Questions