1. CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance
Tao Xie
©D. Marinov, T. Xie

2. Overview Why look for bugs? What are bugs? Where they come from?
How to detect them?

3. Some Costly “Bugs” NASA Mars space missions BMW airbag problems (1999)
Priority inversion (2004)
Different metric systems (1999)
BMW airbag problems (1999)
Recall of >15000 cars
Ariane 5 crash (1996)
Uncaught exception of numerical overflow
Your own (in)famous example?

4. Economic Impact
“The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002
$59.5B annual cost of inadequate software testing infrastructure
$22.2B annual potential cost reduction from feasible infrastructure improvements

5. Economic Impact – cont.
Is Knight's $440 million glitch the costliest computer bug ever?

6. Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch
Hangup
Incorrectness …

7. Dynamic vs. Static Incorrect (observed) behavior
Failure
Incorrect (unobserved) state
Error, latent error
Incorrect lines of code
Fault, defect

8. “Bugs” in IEEE 610.12-1990 Fault Error Failure
Incorrect lines of code
Error
Faults cause incorrect (unobserved) state
Failure
Errors cause incorrect (observed) behavior
Not used consistently in literature!

9. Correctness Common (partial) properties Specific properties
Segfaults, uncaught exceptions
Resource leaks
Data races, deadlocks
Statistics based
Specific properties
Requirements
Specification

10. Traditional Waterfall Model
Requirements Analysis
Design Checking
Implementation Unit Testing
Integration System Testing
Maintenance Verification

11. Phases (1) Requirements Design Specify what the software should do
Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness
Design
Specify how the software should work
Split software into modules, write specifications
Checking: check conformance to requirements

12. Phases (2) Implementation Integration Maintenance
Specify how the modules work
Unit testing: test each module in isolation
Integration
Specify how the modules interact
System testing: test module interactions
Maintenance
Evolve software as requirements change
Verification: test changes, regression testing

13. Testing Effort
Reported to be >50% of development cost [e.g., Beizer 1990]
Microsoft: 75% time spent testing
50% testers who spend all time testing
50% developers who spend half time testing

14. When to Test The later a bug is found, the higher the cost
Orders of magnitude increase in later phases
Also the smaller chance of a proper fix
Old saying: test often, test early
New methodology: test-driven development (write tests before code)

15. Software is Complex Malleable Intangible Abstract
Solves complex problems
Interacts with other software and hardware
Not continuous

16. Software Still Buggy
Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)
Consensus: total correctness impossible to achieve for (complex) software
Risk-driven finding/elimination of bugs
Focus on specific correctness properties

17. Approaches for Detecting Bugs
Software testing
Model checking
(Static) program analysis

18. Software Testing Dynamic approach
Run code for some inputs, check outputs
Checks correctness for some executions
Main questions
Test-input generation
Test-suite adequacy
Test oracles

19. Other Testing Questions
Maintenance
Selection
Minimization
Prioritization
Augmentation
Evaluation
Fault Characterization …

20. Model Checking Typically hybrid dynamic/static approach
Checks correctness for all executions
Some techniques
Explicit-state model checking
Symbolic model checking
Abstraction-based model checking

21. Static Analysis Static approach Checks correctness for all executions
Some techniques
Abstract interpretation
Dataflow analysis
Verification-condition generation

22. Comparison Level of automation Type of bugs found
Push-button vs. manual
Type of bugs found
Hard vs. easy to reproduce
High vs. low probability
Common vs. specific properties
Type of bugs (not) found

23. Soundness and Completeness
Do we detect all bugs?
Impossible for dynamic analysis
Are reported bugs real bugs?
Easy for dynamic analysis
Most practical techniques and tools are both unsound and incomplete!
False positives
False negatives

24. Analysis for Performance
Static compiler analysis, profiling
Must be sound
Correctness of transformation: equivalence
Improves execution time
Programmer time is more important
Programmer productivity
Not only finding bugs

25. Combining Dynamic and Static
Dynamic and static analyses equal in limit
Dynamic: try exhaustively all possible inputs
Static: model precisely every possible state
Synergistic opportunities
Static analysis can optimize dynamic analysis
Dynamic analysis can focus static analysis
More discussions than results

26. Current Status
Testing remains the most widely used approach for finding bugs
A lot of recent progress (within last decade) on model checking and static analysis
Model checking: from hardware to software
Static analysis: from sound to practical
Vibrant research in the area
Gap between research and practice

27. Topics Related to Finding Bugs
How to eliminate bugs?
Debugging
How to prevent bugs?
Programming language design
Software development processes
How to show absence of bugs?
Theorem proving
Model checking, program analysis