1. cs498dm Software Testing Darko Marinov January 17, 2012

2. Teaching Staff Instructor: Darko Marinov –Email: marinov AT illinois.edu –Office: SC 3116, 217-265-6117 –Office hours: after classes or by appointment No TA

3. Course Overview Introduction to software testing –Systematic, organized approaches to testing –Based on models and coverage criteria –Testing is not (only) about finding “bugs” –Improve your testing (and development) skills Five problem sets and a project –Centered around testing Java PathFinder (JPF)

4. JPF in One Slide Will have an entire lecture on JPF –You’ll get to learn a lot about Java/JVM/JPF in this class JPF is a tool for systematic testing of Java programs JPF is a Java Virtual Machine (JVM) that has support for fast backtracking JPF is implemented in Java itself

5. Administrative Info Lectures: TR 12:30pm-1:45pm, 1103 SC Credit: –3 undergraduate hours –3 or 4 graduate hours (a larger project for 4) Prerequisites: recommended software engineering (cs427) and programming languages (cs225, cs421) –Consent of instructor (must if not senior)

6. Grading Points –Project (25%) –Problem sets (5*15%) Grades –A*(90%), B*(80%), C*(70%), D*(60%), F(<60%) –For more details, see the syllabus –The instructor may lower the point limits

7. Project Testing a part of JPF Deliverables –Proposal (due in three weeks) –Progress report (around midterm) –Final report (by the grade submission deadline) –Bug reports (hopefully you’ll find some bugs) Extra bonus points for reporting bugs to me

8. Collaboration You must individually write solutions for the problem sets You can collaborate on everything else (unless explicitly stated not to collaborate!) –Discuss problem sets –Do projects in groups, preferably two or three students Testing is a social activity –Communication matters

9. Course Communication Wiki https://wiki.engr.illinois.edu/display/cs498dmsp12 Mailing list cs498dm AT cs.illinois.edu Instructor: Darko –Email: marinov AT illinois.edu –Office: SC 3116, 217-265-6117 –Office hours: after classes or by appointment

10. Signup Sheet Name Email address (NetID@uiuc.edu) Program/Year Interests: what would you like to learn about testing? Experience: what testing did you do?

11. Textbook “Introduction to Software Testing” by Paul Ammann and Jeff Offutt Cambridge University Press, Jan. 2008 Strongly recommended but not required –Books should be in the bookstore already

12. This Lecture: Introduction to “Bugs” Why look for bugs? What are bugs? Where they come from? How to detect them?

13. Some Costly “Bugs” NASA Mars space missions –Priority inversion (2004) –Different metric systems (1999) BMW airbag problems (1999) –Recall of 15,000+ cars Ariane 5 crash (1996) –Uncaught exception of numerical overflow –http://www.youtube.com/watch?v=kYUrqdUyEpIhttp://www.youtube.com/watch?v=kYUrqdUyEpI Your own favorite examples?

14. Some “Bugging” Bugs Smaller issues that give unexpected results Your own favorite examples?

15. Economic Impact “The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002 $59.5B annual cost of inadequate software testing infrastructure $22.2B annual potential cost reduction from feasible infrastructure improvements

16. Estimates Extrapolated from two studies (5% of total) –Manufacturing: transportation equipment –Services: financial institutions Number of simplifying assumptions “…should be considered approximations” What is important to you? –Correctness, performance, functionality

17. Some Motivation for Testers An article from SD Times, a magazine for software development managers: “Improving Software Quality” by Lindsey Vereen (page 34/68) A slide from Debra Richardson, a professor at UC Irvine: “Analysis and Testing are Creative” (page 26/48)

18. Terminology Anomaly Bug Crash Defect Error Failure, fault Glitch Hang Incorrectness J...

19. Dynamic vs. Static Incorrect (observed) behavior –Failure, fault Incorrect (unobserved) state –Error, latent error Incorrect lines of code –Fault, error

20. “Bugs” in IEEE 610.12-1990 Fault –Incorrect lines of code Error –Faults cause incorrect (unobserved) state Failure –Errors cause incorrect (observed) behavior Not used consistently in literature!

21. Correctness and Quality Common (partial) properties Segfaults, uncaught exceptions Resource leaks Data races, deadlocks Statistics based Specific properties Requirements Specification

22. Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Regression Testing We will look at general techniques, applicable in several phases of testing

23. Phases (1) Requirements Specify what the software should do Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design Specify how the software should work Split software into modules, write specifications Checking: check conformance to requirements, using for example conformance testing

24. Phases (2) Implementation Specify how the modules work –Unit testing: test each module in isolation Integration Specify how the modules interact –Integration testing: test module interactions –System testing: test the entire system Maintenance Evolve software as requirements change –Regression testing: test changes

25. Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing 50% testers who spend all time testing 50% developers who spend half time testing

26. When to Test The later a bug is found, the higher the cost Orders of magnitude increase in later phases Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests even before writing code)

27. Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

28. Software Still Buggy Folklore: 1-10 (residual) faults per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for complex software Risk-driven finding/elimination of faults Focus on specific correctness properties

29. Approaches to Detecting Bugs Software testing Model checking (Static) program analysis …

30. Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions Test-suite adequacy (coverage criteria) Test-input generation Test oracles

31. Other Testing Questions Selection Minimization Prioritization Augmentation Evaluation Fault Characterization … Testing is not (only) about finding faults!

32. Current Status Testing remains the most widely used approach to finding bugs Validation: are we building the right system? Verification: are we building the system right? Testing is gaining importance with test-first development and increased reliability needs A lot of research on testing (part of mine too) This course is not about research

33. “Schools” of Software Testing Bret Pettichord described four schools Analytic (a branch of CS/Mathematics) Factory (a managed process) Quality (a branch of quality assurance) Context-Driven (a branch of development) This course focuses on artifacts, not process Do you want a guest speaker from industry?

34. Topics Related to “Finding Bugs” How to “eliminate bugs” (localize faults)? Debugging How to “prevent bugs”? Programming language design Software development processes How to “show absence of bugs”? Theorem proving Model checking, program analysis

35. Testing Topics to Cover Test coverage and adequacy criteria Graph, logic, input domains, syntax-based Test-input generation Test oracles Model-based testing Testing software with structural inputs Test automation Testing in your domain of interest?

36. Summary of the Introduction Eliminate bugs to save lives and money “Bugs” may mean faults, errors, failures Several approaches for detection: software testing, model checking, static analysis… Software testing is the most widely used approach for validation and verification We will cover systematic approaches to testing, based on coverage criteria for various models Testing is not (only) about revealing faults