CS527: Advanced Topics in Software Engineering (Software Testing and Analysis) Darko Marinov August 26, 2008

Course Overview Graduate seminar on program analysis (for bug finding), emphasis: systematic testing Focus on a (research) project Papers: reading in advance, writing reports, presenting, and discussing Project: initial proposal, progress report, final presentation, paper One or two problem sets

Administrative Info Meetings: TR 2-3:15pm, 1302 SC Credit: 4 graduate hours Auditors welcome for discussions Can come to any lecture, mostly self-contained Prerequisites: some software engineering, programming languages, and logic

Evaluation Grading Distribution Project is the most important Presentation [20%] Participation (reports and discussion) [20%] Problem set(s) [20%] Distribution Grades will be A- centered No guarantee for A (or even a passing grade)! Project is the most important

Project Proposal (due in about a month) Progress report (midterm time) Presentation (last weeks of classes) Paper (last day of classes) Nine students who took similar classes published 8 papers based on their projects I’m happy to help, no co-authorship required

Fair Warnings This class will differ from most you take Seminar style, reading papers Centered around (research) projects Projects are NOT easy Require that you explore a topic in great depth The topic can/should be fairly narrow

Project Overview Testing/analysis of some open-source code We will use only Java this semester Strongly encouraged to consider parts of IDEs (Eclipse or NetBeans), will consider refactorings Sample topics Automated unit testing Test coverage and adequacy criteria Test-input generation, test oracles …

Course Communication Wiki http://agora.cs.uiuc.edu/display/cs527fa08 Mailing list cs527 AT cs.uiuc.edu

Personnel TA: Vilas Jagganath Instructor: Darko Marinov Office: 2107 SC, hours: by appointment Phone number: 217-244-1976 Email: vbangal2 AT cs.uiuc.edu Instructor: Darko Marinov Office: 3116 SC, hours: by appointment Phone number: 217-265-6117 Email: marinov AT cs.uiuc.edu

Signup Sheet Name Netid (please write it legibly) Program/Year Group Interests Please also sign up on Wiki (if possible, as there are some delays with netids)

This Lecture: Overview Why look for bugs? What are bugs? Where they come from? How to detect them?

Some Costly “Bugs” NASA Mars space missions BMW airbag problems (1999) Priority inversion (2004) Different metric systems (1999) BMW airbag problems (1999) Recall of >15000 cars Ariane 5 crash (1996) Uncaught exception of numerical overflow http://www.youtube.com/watch?v=kYUrqdUyEpI Your own favorite example?

Some “Bugging” Bugs An example bug on my laptop “Jumping” file after changing properties Put a read-only file on the desktop Change properties: rename and make not read-only Your own favorite example?

Economic Impact “The Economic Impact of Inadequate Infrastructure for Software Testing” NIST Report, May 2002 $59.5B annual cost of inadequate software testing infrastructure $22.2B annual potential cost reduction from feasible infrastructure improvements

Estimates Extrapolated from two studies (5% of total) Manufacturing: transportation equipment Services: financial institutions Number of simplifying assumptions “…should be considered approximations” What is important for you? Correctness, performance, functionality

Terminology Anomaly Bug Crash Defect Error Failure, fault G… …

Dynamic vs. Static Incorrect (observed) behavior Failure, fault Incorrect (unobserved) state Error, latent error Incorrect lines of code Fault, error

“Bugs” in IEEE 610.12-1990 Fault Error Failure Incorrect lines of code Error Faults cause incorrect (unobserved) state Failure Errors cause incorrect (observed) behavior Not used consistently in literature!

Correctness Common (partial) properties Specific properties Segfaults, uncaught exceptions Resource leaks Data races, deadlocks Statistics based Specific properties Requirements Specification

Traditional Waterfall Model Requirements Analysis Design Checking Implementation Unit Testing Integration System Testing Maintenance Verification

Phases (1) Requirements Design Specify what the software should do Analysis: eliminate/reduce ambiguities, inconsistencies, and incompleteness Design Specify how the software should work Split software into modules, write specifications Checking: check conformance to requirements

Phases (2) Implementation Integration Maintenance Specify how the modules work Unit testing: test each module in isolation Integration Specify how the modules interact System testing: test module interactions Maintenance Evolve software as requirements change Verification: test changes, regression testing

Testing Effort Reported to be >50% of development cost [e.g., Beizer 1990] Microsoft: 75% time spent testing 50% testers who spend all time testing 50% developers who spend half time testing

When to Test The later a bug is found, the higher the cost Orders of magnitude increase in later phases Also the smaller chance of a proper fix Old saying: test often, test early New methodology: test-driven development (write tests before code)

Software is Complex Malleable Intangible Abstract Solves complex problems Interacts with other software and hardware Not continuous

Software Still Buggy Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing) Consensus: total correctness impossible to achieve for (complex) software Risk-driven finding/elimination of bugs Focus on specific correctness properties

Approaches for Detecting Bugs Software testing Model checking (Static) program analysis

Software Testing Dynamic approach Run code for some inputs, check outputs Checks correctness for some executions Main questions Test-input generation Test-suite adequacy Test oracles

Other Testing Questions Selection Minimization Prioritization Augmentation Evaluation Fault Characterization …

Model Checking Typically hybrid dynamic/static approach Checks correctness for all executions Some techniques Explicit-state model checking Symbolic model checking Abstraction-based model checking

Static Analysis Static approach Checks correctness for all executions Some techniques Abstract interpretation Dataflow analysis Verification-condition generation

Comparison Level of automation Type of bugs found Push-button vs. manual Type of bugs found Hard vs. easy to reproduce High vs. low probability Common vs. specific properties Type of bugs (not) found

Soundness and Completeness Do we detect all bugs? Impossible for dynamic analysis Are reported bugs real bugs? Easy for dynamic analysis Most practical techniques and tools are both unsound and incomplete! False positives False negatives

Analysis for Performance Static compiler analysis, profiling Must be sound Correctness of transformation: equivalence Improves execution time Programmer time is more important Programmer productivity Not only finding bugs

Combining Dynamic and Static Dynamic and static analyses equal in limit Dynamic: try exhaustively all possible inputs Static: model precisely every possible state Synergistic opportunities Static analysis can optimize dynamic analysis Dynamic analysis can focus static analysis More discussions than results

Current Status Testing remains the most widely used approach for finding bugs A lot of recent progress (within last decade) on model checking and static analysis Model checking: from hardware to software Static analysis: from sound to practical Vibrant research in the area Gap between research and practice

Topics Related to Finding Bugs How to eliminate bugs? Debugging How to prevent bugs? Programming language design Software development processes How to show absence of bugs? Theorem proving Model checking, program analysis

Next Lecture Thursday, August 28, at 2pm, 1302 SC Texts to read (listed on Wiki) How to Read an Engineering Research Paper by William G. Griswold Writing Good Software Engineering Research Papers by Mary Shaw (ICSE 2003) If you have read that paper, read on another area Assignment 0: Modify “People” on Wiki