From Quality Control to Quality Assurance…and Beyond Alan Page Microsoft

Introduction My testing career started in 1993 Testing always last Testing often skipped Anyone can test “Testing in” quality My role in testing has changed significantly in the past 13 years Testing as a whole has matured Disclaimer – I know what I know and don’t claim to know more (and are not representative of microsoft)

Software testing 13 years ago Anyone can do it Technical knowledge not important Test is a stepping stone to development (often still true today) Testing occurs late in the product cycle

Software testing today Many teams using automation to produce consistent and frequent test results Often heavy emphasis on automation Testers are generally focused on: Test authoring: writing test cases and automated tests Test execution: running automated or manual test cases Functional and non-functional testing Other bug finding activities Some analysis Data analysis is often extensive, but is primarily based on defect data

Software testing tomorrow Too many bugs getting to customers Bugs need to be found sooner

Where are we going? Better bug prediction More testable software Defect prevention Inspections … How do we get there?

More testable software Testability is the degree to which components and systems are designed and implemented to make it easier for tests to achieve complete and repeatable code path coverage and simulate all usage situations in a cost efficient manner. In other words, testability determines how inexpensive it is to test. or What is the cost of testing?

Testability Testability is a design time attribute …but testers often drive testability into the design Ask questions and take action: How are we going to efficiently test this? Add testability sections to design and functional specifications Hold testability reviews

SOCK for testability Testability: SOCK Simple components are less expensive to test. Can the hardware be simulated? Can we test a multi-machine scenario on a single machine? Observable behavior and results are essential in order to determine pass or fail Can I determine if the internal structures were updated correctly? Can I tell which path the code took?

Control: can I test every nook and cranny? Can I change timeout values or other thresholds to simulate failures? Can I simulate every failure that a customer may hit? Knowledge is needed to understand expected results and compare with actual results What should happen in error cases? What circumstances cause this error? SOCK for testability

Defect analysis Known defects need to be analyzed Deep analysis can tell a lot, but it’s expensive Too lightweight of an analysis may not tell you enough Ask why Analyze to the point of action: Ask “why” until you have a sufficient root cause

Why did the program crash? The filename was longer than expected. Why did the program not expect a longer filename? The programmer was unfamiliar with file name limits Why was the programmer unfamiliar with the limits? They were new to windows programming and hadn’t been trained …or… Why was this missed during code review? Code review rules were often skipped or rushed. Ask Why?

Defect prevention Root cause analysis leads to defect prevention What could have been done to keep this defect from ever happening? Examples: Change compiler settings or use code analysis tools or scripts Build code often (continuous integration) Automatically run unit tests or verification tests at check-in

Software Inspections Not typically considered a “Test task” – but testers are well suited to drive this effort Extremely effective method of removing defects, but… Cost (training and time) tends to scare teams away

What happens when you inspect? Example of changes from inspection: Inspection Metrics Existing code Release A Release B Release C Inspection Effort0 hrs406 hrs276 hrs691 hrs Total LOC400K Inspection hrs / KLOC Total Effortn/a1062 hrs1101 hrs2169 hrs Percent in inspection0%38%25%32% Total hrs / KLOCn/a Defects / KLOC (after check-in) Measurements were performed on existing code base, and again on subsequent releases

How to make inspections less scary Team members do not know how to inspect Train the team; appoint a moderator Inspections perceived to slow down the project Schedule inspections; measure and track progress Team members fearful of inspections Non-confrontational meetings; remove threats; use as a learning experience for author

What am I trying to say here? We need to stop waiting for bugs! It’s great to find bugs before the customers do, but we need to find them earlier We need more emphasis on early detection and ultimately, prevention

How can this be achieved? Be an equal member of the engineering team Software quality is a tough problem Technical, creative people are needed to solve this problem We need clearly defined career paths for both managers and non-managers in order to keep these types of people in test Provide learning opportunities when needed

Key points Software testing is a maturing profession Need to move beyond quality control (writing test cases and running tests) Test needs to do more to drive quality improvements throughout the engineering process Technical skills are important in order to drive quality initiatives

Resources Software Inspections Tom Gilb and Dorothy Graham, Addison- Wesley The Practical Guide to Defect Prevention Harry Emil et al. Microsoft Press Random thoughts on testing

Questions?