1. cs498st Software Testing
Tao Xie

2. Teaching Staff Instructor: Tao Xie No TA Graders (likely)
(subject cs498st)
Office: SC 4237,
Office hours: by appointment
No TA
Graders (likely)

3. Administrative Info Lectures: TR 3:30pm-4:45pm, 1109 SC Credit:
3 undergraduate hours
3 or 4 graduate hours (a larger project for 4)
Prerequisites: recommended software engineering (cs427) and programming languages (cs225, cs421)
Consent of instructor

4. Course Overview Introduction to software testing
Systematic, organized approaches to testing
Based on models and coverage criteria
Testing is not (only) about finding “bugs”
Improve your testing (and development) skills
Several homework assignments
Project: testing some real software
C# code
No exams

5. Grading Points Grades Homeworks (70%) Project (30%)
A*(90%), B*(80%), C*(70%), D*(60%), F(<60%)
For more details, see the syllabus online
The instructor may lower the point limits

6. Software Tools Microsoft Visual Studio 2017 & 2015 Enterprise Edition
Available to UIUC students: ?zpid=2917
IntelliTest (Pex)
Code Contracts (available for VS 2015)
Useful resources:
Java vs. C#
C# for Java Developers

7. Project Testing some real software (in C#) Deliverables
Proposal (due in three weeks)
Progress report (around midterm)
Final report (by the grade submission deadline)
Pull requests for test code and/or bug reports
Please meet me if you cannot contribute publicly

8. Collaboration You must individually solve homework assignments
You canNOT collaborate on anything else (unless explicitly stated that you can!)
Do projects in groups, preferably two or three students but can do solo projects
Testing is a social activity
Communication matters
But we are in an education environment so cannot allow all communication

9. Course Communication
Wiki
Main class communication Piazza:
Instructor: Tao
Office: SC 4237,
Office hours: by appointment

10. Signup Sheet Name Email address (NetID@illinois.edu) Program/Year
Interests: what would you like to learn about testing?
Experience: what testing did you do?

11. Optional Textbook
“Introduction to Software Testing” by Paul Ammann and Jeff Offutt Cambridge University Press, 2nd Edition, Dec

12. This Lecture: Introduction to “Bugs”
Why look for bugs?
What are bugs?
Where they come from?
How to detect them?

13. Some Costly “Bugs” Ariane 5 crash (1996) Your own favorite examples?
Uncaught exception of numerical overflow
“Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.”
Your own favorite examples?

14. Economic Impact
“The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002
$59.5B annual cost of inadequate software testing infrastructure
$22.2B annual potential cost reduction from feasible infrastructure improvements

15. Economic Impact – cont.
Is Knight's $440 million glitch the costliest computer bug ever?
expensive-computer-bug/index.html

16. New Era…Software itself is changing...
Services

17. How people use software is changing…

18. How people use software is changing…
Individual
Isolated
Not much data/content
generation

19. How people use software is changing…
Individual
Isolated
Not much data/content
generation

20. How people use software is changing…
Social
Individual
Isolated
Collaborative
Huge amount of data/artifacts
generated anywhere anytime
Not much data/content
generation

21. How software is built & operated is changing…

22. How software is built & operated is changing…
Code centric
Data pervasive
In-lab testing
Debugging in the large
Experience & gut-feeling
Informed decision making
Centralized development
Distributed development
Long product cycle
Continuous release … …

23. Quality Attributes What is important to you?
Correctness, performance, functionality, security, …

24. Some Motivation for Testers
A slide from Debra Richardson, a professor at UC Irvine: “Analysis and Testing are Creative” (page 26/48)

26. Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch Hang
Incorrectness (from a 2009 cs498dm student)
Jitter (from a 2012 cs498dm student)
Kink (from a 2015 cs498dm student)
L…?

27. Dynamic vs. Static Incorrect (observed) behavior
Failure, fault
Incorrect (unobserved) state
Error, latent error
Incorrect lines of code
Fault, error

28. “Bugs” in IEEE 610.12-1990 Fault Error Failure
Incorrect lines of code
Error
Faults cause incorrect (unobserved) state
Failure
Errors cause incorrect (observed) behavior
Not used consistently in literature!

29. Correctness and Quality
Common (partial) properties
Segfaults, uncaught exceptions
Resource leaks
Data races, deadlocks
Statistics based
Specific properties
Requirements
Specification

30. Traditional Waterfall Model
Requirements Analysis
Design Checking
Implementation Unit Testing
Integration System Testing
Maintenance Regression Testing

31. Phases (1) Requirements Design Specify what the software should do
Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness
Design
Specify how the software should work
Split software into modules, write specifications
Checking: check conformance to requirements, using for example conformance testing

32. Phases (2) Implementation Integration Maintenance
Specify how the modules work
Unit testing: test each module in isolation
Integration
Specify how the modules interact
Integration testing: test module interactions
System testing: test the entire system
Maintenance
Evolve software as requirements change
Regression testing: test changes

33. Testing Effort
Reported to be >50% of development cost [e.g., Beizer 1990]
Microsoft: 75% time spent testing
50% testers who spend all time testing
50% developers who spend half time testing

34. When to Test The later a bug is found, the higher the cost
Orders of magnitude increase in later phases
Also the smaller chance of a proper fix
Old saying: test often, test early
New methodology: test-driven development (write tests before code)

35. Software is Complex Malleable Intangible Abstract
Solves complex problems
Interacts with other software and hardware
Not continuous

36. Software Still Buggy
Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)
Consensus: total correctness impossible to achieve for (complex) software
Risk-driven finding/elimination of bugs
Focus on specific correctness properties

37. Approaches for Detecting Bugs
Software testing
Model checking
(Static) program analysis

38. Software Testing Dynamic approach
Run code for some inputs, check outputs
Checks correctness for some executions
Main questions
Test-input generation
Test-suite adequacy
Test oracles

39. Other Testing Questions
Maintenance
Selection
Minimization
Prioritization
Augmentation
Evaluation
Fault Characterization …

40. Current Status
Testing remains the most widely used approach for finding bugs
Testing is gaining importance with test-first development and increased reliability needs
A lot of recent progress (within last decade) on model checking and static analysis
Model checking: from hardware to software
Static analysis: from sound to practical
Vibrant research in the area
Gap between research and practice

41. Topics Related to Finding Bugs
How to eliminate bugs?
Debugging
How to prevent bugs?
Programming language design
Software development processes
How to show absence of bugs?
Theorem proving
Model checking, program analysis

42. Testing Topics to Cover
Test coverage and adequacy criteria
Graph, logic, input domains, syntax-based
Test-input generation
Test oracles
Regression testing
Test automation
Testing in your domain of interest?

43. Summary of the Introduction
Eliminate bugs to save lives and money
“Bugs” may mean faults, errors, failures
Several approaches for detection: software testing, model checking, static analysis…
Software testing is the most widely used approach for validation and verification
We will cover systematic approaches to testing, based on coverage criteria for various models
Testing is not (only) about revealing faults