1. CS527: Advanced Topics in Software Engineering (Software Testing and Analysis)
Darko Marinov
August 26, 2008

2. Course Overview
Graduate seminar on program analysis (for bug finding), emphasis: systematic testing
Focus on a (research) project
Papers: reading in advance, writing reports, presenting, and discussing
Project: initial proposal, progress report, final presentation, paper
One or two problem sets

3. Administrative Info Meetings: TR 2-3:15pm, 1302 SC
Credit: 4 graduate hours
Auditors welcome for discussions
Can come to any lecture, mostly self-contained
Prerequisites: some software engineering, programming languages, and logic

4. Evaluation Grading Distribution Project is the most important
Presentation [20%]
Participation (reports and discussion) [20%]
Problem set(s) [20%]
Distribution
Grades will be A- centered
No guarantee for A (or even a passing grade)!
Project is the most important

5. Project Proposal (due in about a month) Progress report (midterm time)
Presentation (last weeks of classes)
Paper (last day of classes)
Nine students who took similar classes published 8 papers based on their projects
I’m happy to help, no co-authorship required

6. Fair Warnings This class will differ from most you take
Seminar style, reading papers
Centered around (research) projects
Projects are NOT easy
Require that you explore a topic in great depth
The topic can/should be fairly narrow

7. Project Overview Testing/analysis of some open-source code
We will use only Java this semester
Strongly encouraged to consider parts of IDEs (Eclipse or NetBeans), will consider refactorings
Sample topics
Automated unit testing
Test coverage and adequacy criteria
Test-input generation, test oracles …

8. Course Communication Wiki http://agora.cs.uiuc.edu/display/cs527fa08
Mailing list cs527 AT cs.uiuc.edu

9. Personnel TA: Vilas Jagganath Instructor: Darko Marinov
Office: 2107 SC, hours: by appointment
Phone number:
vbangal2 AT cs.uiuc.edu
Instructor: Darko Marinov
Office: 3116 SC, hours: by appointment
Phone number:
marinov AT cs.uiuc.edu

10. Signup Sheet Name Netid (please write it legibly) Program/Year Group
Interests
Please also sign up on Wiki (if possible, as there are some delays with netids)

11. This Lecture: Overview
Why look for bugs?
What are bugs?
Where they come from?
How to detect them?

12. Some Costly “Bugs” NASA Mars space missions BMW airbag problems (1999)
Priority inversion (2004)
Different metric systems (1999)
BMW airbag problems (1999)
Recall of >15000 cars
Ariane 5 crash (1996)
Uncaught exception of numerical overflow
Your own favorite example?

13. Some “Bugging” Bugs An example bug on my laptop
“Jumping” file after changing properties
Put a read-only file on the desktop
Change properties: rename and make not read-only
Your own favorite example?

14. Economic Impact
“The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002
$59.5B annual cost of inadequate software testing infrastructure
$22.2B annual potential cost reduction from feasible infrastructure improvements

15. Estimates Extrapolated from two studies (5% of total)
Manufacturing: transportation equipment
Services: financial institutions
Number of simplifying assumptions
“…should be considered approximations”
What is important for you?
Correctness, performance, functionality

16. Terminology
Anomaly
Bug
Crash
Defect
Error
Failure, fault G… …

17. Dynamic vs. Static Incorrect (observed) behavior
Failure, fault
Incorrect (unobserved) state
Error, latent error
Incorrect lines of code
Fault, error

18. “Bugs” in IEEE 610.12-1990 Fault Error Failure
Incorrect lines of code
Error
Faults cause incorrect (unobserved) state
Failure
Errors cause incorrect (observed) behavior
Not used consistently in literature!

19. Correctness Common (partial) properties Specific properties
Segfaults, uncaught exceptions
Resource leaks
Data races, deadlocks
Statistics based
Specific properties
Requirements
Specification

20. Traditional Waterfall Model
Requirements Analysis
Design Checking
Implementation Unit Testing
Integration System Testing
Maintenance Verification

21. Phases (1) Requirements Design Specify what the software should do
Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness
Design
Specify how the software should work
Split software into modules, write specifications
Checking: check conformance to requirements

22. Phases (2) Implementation Integration Maintenance
Specify how the modules work
Unit testing: test each module in isolation
Integration
Specify how the modules interact
System testing: test module interactions
Maintenance
Evolve software as requirements change
Verification: test changes, regression testing

23. Testing Effort
Reported to be >50% of development cost [e.g., Beizer 1990]
Microsoft: 75% time spent testing
50% testers who spend all time testing
50% developers who spend half time testing

24. When to Test The later a bug is found, the higher the cost
Orders of magnitude increase in later phases
Also the smaller chance of a proper fix
Old saying: test often, test early
New methodology: test-driven development (write tests before code)

25. Software is Complex Malleable Intangible Abstract
Solves complex problems
Interacts with other software and hardware
Not continuous

26. Software Still Buggy
Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)
Consensus: total correctness impossible to achieve for (complex) software
Risk-driven finding/elimination of bugs
Focus on specific correctness properties

27. Approaches for Detecting Bugs
Software testing
Model checking
(Static) program analysis

28. Software Testing Dynamic approach
Run code for some inputs, check outputs
Checks correctness for some executions
Main questions
Test-input generation
Test-suite adequacy
Test oracles

29. Other Testing Questions
Selection
Minimization
Prioritization
Augmentation
Evaluation
Fault Characterization …

30. Model Checking Typically hybrid dynamic/static approach
Checks correctness for all executions
Some techniques
Explicit-state model checking
Symbolic model checking
Abstraction-based model checking

31. Static Analysis Static approach Checks correctness for all executions
Some techniques
Abstract interpretation
Dataflow analysis
Verification-condition generation

32. Comparison Level of automation Type of bugs found
Push-button vs. manual
Type of bugs found
Hard vs. easy to reproduce
High vs. low probability
Common vs. specific properties
Type of bugs (not) found

33. Soundness and Completeness
Do we detect all bugs?
Impossible for dynamic analysis
Are reported bugs real bugs?
Easy for dynamic analysis
Most practical techniques and tools are both unsound and incomplete!
False positives
False negatives

34. Analysis for Performance
Static compiler analysis, profiling
Must be sound
Correctness of transformation: equivalence
Improves execution time
Programmer time is more important
Programmer productivity
Not only finding bugs

35. Combining Dynamic and Static
Dynamic and static analyses equal in limit
Dynamic: try exhaustively all possible inputs
Static: model precisely every possible state
Synergistic opportunities
Static analysis can optimize dynamic analysis
Dynamic analysis can focus static analysis
More discussions than results

36. Current Status
Testing remains the most widely used approach for finding bugs
A lot of recent progress (within last decade) on model checking and static analysis
Model checking: from hardware to software
Static analysis: from sound to practical
Vibrant research in the area
Gap between research and practice

37. Topics Related to Finding Bugs
How to eliminate bugs?
Debugging
How to prevent bugs?
Programming language design
Software development processes
How to show absence of bugs?
Theorem proving
Model checking, program analysis

38. Next Lecture Thursday, August 28, at 2pm, 1302 SC
Texts to read (listed on Wiki)
How to Read an Engineering Research Paper by William G. Griswold
Writing Good Software Engineering Research Papers by Mary Shaw (ICSE 2003)
If you have read that paper, read on another area
Assignment 0: Modify “People” on Wiki