Developer Testing and Debugging

Resources Code Complete by Steve McConnell Code Complete by Steve McConnell Safari Books Online Safari Books Online Google Image search Google Image search

Testing/Debugging Debugging - identifying the root cause of an error and correcting it Debugging - identifying the root cause of an error and correcting it Testing – detecting errors Testing – detecting errors

Role of Developer Testing in Software Quality Testing is an important part of software quality, but also is difficult. Why? Characteristics of testing: Counter-intuitive Counter-intuitive Goal - > finding errors By itself, does not improve quality By itself, does not improve quality Example: X-Ray – Does not improve your health Can’t prove absence of errors Can’t prove absence of errors

Definitions Some Types of Testing Unit Unit Component Component Integration Integration Regression Regression System System Black box Black box White box White box

Types of Testing Unit/Component Testing Unit/Component The execution of a class, package, small program, or other program element which is tested in isolation from the more complete system.

Types of Testing Integration Testing Unit/Component Combined execution of two or more classes, packages, components, or subsystems that have been created by multiple programmers or programming teams.

Types of Testing Regression Testing Unit/Component Running previously executed test cases for the purpose of finding defects in software that previously passed the same set of tests.

Types of Testing System Testing Unit/Component The execution of the software in its final configuration.

Types of Testing Black Box Testing Tester cannot see inner workings of item being tested. Tester cannot see inner workings of item being tested. ?

Types of Testing White (or Glass) Box Testing Tester is aware of the inner workings of item being tested. Tester is aware of the inner workings of item being tested.

Types of Testing Unit/Component Integration Regression System White Box Black Box Developers Special Test Personnel Other Testing: Beta Customer acceptance Performance …

Limitations of Developer Testing Tests tend to be “clean tests” Tests tend to be “clean tests” Tests tend to have an optimistic view of coverage Tests tend to have an optimistic view of coverage Tests tend to skip sophisticated test coverage Tests tend to skip sophisticated test coverage

Test First or Test Last? Benefits of writing test cases first: Benefits of writing test cases first: Doesn’t take any more time – resequences Doesn’t take any more time – resequences Forces you to think about the requirement Forces you to think about the requirement Exposed requirement problems sooner Exposed requirement problems sooner

Recommended Approach to Developer Testing Test for each relevant requirement Test for each relevant requirement Test for each relevant design concern Test for each relevant design concern Keep track of kinds of errors made Keep track of kinds of errors made Design test cases along with product Design test cases along with product

Recommended Approach to Developer Testing (cont.) Incomplete Testing – “most likely” Incomplete Testing – “most likely” Structured Basis Testing – “at least once” Structured Basis Testing – “at least once” Data Flow Testing – “follow that data” Data Flow Testing – “follow that data” Equivalence Partitioning – “streamlining” Equivalence Partitioning – “streamlining” Error Guessing – “sixth sense” Error Guessing – “sixth sense” Boundary Analysis – “to the limit” Boundary Analysis – “to the limit”

Typical Errors Summary of Boris Beizer’s study on errors: Summary of Boris Beizer’s study on errors: 25.18%Structural 25.18%Structural 22.44%Data 22.44%Data 16.19%Functionality as implemented 16.19%Functionality as implemented 9.88%Construction 9.88%Construction 8.98%Integration 8.98%Integration 8.12%Functional requirements 8.12%Functional requirements 2.76%Test definition or execution 2.76%Test definition or execution 1.74%System, software architecture 1.74%System, software architecture 4.71%Unspecified 4.71%Unspecified

Test-Support Tools Scaffolding Scaffolding Diff tools Diff tools Test data gatherers Test data gatherers Coverage monitors Coverage monitors Data recorder/logger Data recorder/logger Symbolic debuggers Symbolic debuggers System perturbers System perturbers Error databases Error databases

Improving Your Testing Repeatable Tests Repeatable Tests Automate Tests Automate Tests

Keeping Test Records Errors tend to cluster in a few error-prone classes and routines. Errors tend to cluster in a few error-prone classes and routines.

Testing Summary Testing by the developer is a key part of a full testing strategy. Testing by the developer is a key part of a full testing strategy. Writing test cases before the code takes the same amount of time and effort. Writing test cases before the code takes the same amount of time and effort. Testing is only one part of a good software-quality program. Testing is only one part of a good software-quality program. Many test cases can be generated deterministically by using basis testing, dataflow analysis, boundary analysis, classes of bad data, and classes of good data. Many test cases can be generated deterministically by using basis testing, dataflow analysis, boundary analysis, classes of bad data, and classes of good data. Errors tend to cluster in a few error-prone classes and routines. Errors tend to cluster in a few error-prone classes and routines. Automated testing is useful, essential for regression testing. Automated testing is useful, essential for regression testing. Make it regular, measure it, and use what you learn to improve it. Make it regular, measure it, and use what you learn to improve it.

Debugging Finding and Getting Rid of Bugs Finding and Getting Rid of Bugs

Debugging Overview The process of identifying the root cause of an error and correcting it. The process of identifying the root cause of an error and correcting it. Not a way to improve the quality of your software per se; it's a way to diagnose defects. Not a way to improve the quality of your software per se; it's a way to diagnose defects.

Debugging - Perspective Defects as Opportunities – You can learn: Defects as Opportunities – You can learn: About the program you’re working on About the program you’re working on About the kinds of mistakes you make About the kinds of mistakes you make About the quality of your code About the quality of your code About how you solve problems About how you solve problems About how you fix defects About how you fix defects

Finding a Defect Stabilize the error. Stabilize the error. Locate the source of the error (the "fault"). Locate the source of the error (the "fault"). Fix the defect. Fix the defect. Test the fix. Test the fix. Look for similar errors. Look for similar errors.

Fixing a Defect Understand the problem before you fix it Understand the problem before you fix it Understand the program, not just the problem Understand the program, not just the problem Confirm the defect diagnosis Confirm the defect diagnosis Relax Relax Save the original source code Save the original source code Fix the problem, not the symptom Fix the problem, not the symptom Change the code only for good reason Change the code only for good reason Make one change at a time Make one change at a time Check your fix Check your fix Add a unit test that exposes the defect Add a unit test that exposes the defect Look for similar defects Look for similar defects

Psychological Considerations in Debugging Psychological Set contributes to debugging blindness. Psychological Set contributes to debugging blindness.

Psychological Considerations in Debugging (Examples) Psychological Distance Can Help Psychological Distance Can Help First Variable Second Variable Psychological Distance stopptstcppt Almost invisible shiftrnshiftrm Almost none dcountBcountSmall claims1claims2Small productsumLarge

Debugging Tools Source-Code Comparators Source-Code Comparators Compiler Warning Messages Compiler Warning Messages Execution Profilers Execution Profilers Test Frameworks/Scaffolding Test Frameworks/Scaffolding Commercial Debuggers Commercial Debuggers

Summary Debugging is a make-or-break aspect of software development. Debugging is a make-or-break aspect of software development. A systematic approach to finding and fixing errors is critical to success. A systematic approach to finding and fixing errors is critical to success. Understand the root problem before you fix the program. Understand the root problem before you fix the program. Set your compiler warning to the pickiest level possible. Set your compiler warning to the pickiest level possible. Debugging tools are powerful aids to software development. Debugging tools are powerful aids to software development.