1. Software Testing Seminar Mooly Sagiv http://www.math.tau.ac.il/~sagiv/courses/testing.html Tel Aviv University 640-6706 Sunday 16-18 Monday 10-12 Schrieber 317

2. Bibliography Michael Young, University of Oregon UW/MSR 1999 Glenford Myers “The Art of Software Testing” 1978!

3. Outline Testing Questions Goals of Testing The Psychology of Testing Program Inspections and Reviews Test Case Design Achieving Reliability Other Techniques A Success Story

4. Standard Testing Questions Did this test execution succeed or fail? –Oracles How shall we select test cases? –Selection; generation How do we know when we’ve tested enough? –Adequacy What do we know when we’re done? –Assessment

5. Possible Goals of Testing Find faults –Glenford Myers, The Art of Software Testing Provide confidence –of reliability –of (probable) correctness –of detection (therefore absence) of particular faults

6. Testing Theory (such as it is) Plenty of negative results –Nothing guarantees correctness –Statistical confidence is prohibitively expensive –Being systematic may not improve fault detection as compared to simple random testing So what did you expect, decision procedures for undecidable problems?

7. What Information Can We Exploit? Specifications (formal or informal) –in Oracles –for Selection, Generation, Adequacy Designs –… Code –for Selection, Generation, Adequacy Usage (historical or models) Organization experience

8. The Psychology of Testing “Testing is the process of demonstrating that an errors are not present” “Testing is the process of establishing confidence that the program does what it intends to do” “Testing is the process of executing programs with the intent of finding errors” –Successful (positive) test: exposes an error

9. Black Box Testing View the program as a black box Exhaustive testing is infeasible even for tiny programs –Can never guarantee correctness –Fundamental question is economics –“Maximize investment” –Example: “Partition test”

10. White Box Testing Investigate the internal structure of the program Exhaustive paths testing is infeasible Does not even guarantee correctness –Specification is needed –Missing paths –Data dependent paths Becomes an economical question if (a-b < epsilon)

11. Testing Principles Test case must include the definition of expected results A program should not test her/his own code A programming organization should not test its own programs Thoroughly inspect the results of each test Test cases must be also written for invalid inputs Check that programs do not do unexpected things Test cases should not be thrown a way Do not plan testing assuming that there are no errors The probably of error in a piece of code is proportional to the errors found so far in this part of the code

12. Program Inspections and Reviews Conducted in groups/sessions A well established process Modern programming languages eliminate many programming errors: –Type errors –Memory violations –Uninitialized variables But many errors are not currently found –Division by zero –Overflow –Wrong precedence of logical operators

13. Test Case Design What subset of inputs has the highest probability of detecting the most errors? Combine black and white box testing

14. Higher Order Testing Show that the program does not so what the end-user expects… Different testing levels –Unit test (procedure boundaries) –Module test –Function test (discrepancies external spec.) –System test (discrepancies with original objectives) Facility test Volume test Usability test Security test Performance test...

15. Equivalence partitioning Boundary-value analysis Cause-effect graphing Error guessing State space exploration Black boxWhite box Coverage/Adequacy Data flow analysis Cleanness Correctness Mutation Slicing

16. Partition Testing (Equivalence Partitioning) Basic idea: Divide program input space into (quasi-) equivalence classes –Underlying idea of specification-based, structural, and fault-based testing

17. Specification-Based Partition Testing Divide the program input space according to identifiable cases in the specification –May emphasize boundary cases –May include combinations of features or values If all combinations are considered, the space is usually too large Systematically “cover” the categories –May be driven by scripting tools or input generators –Example: Category-Partition testing [Ostrand]

18. “Adequate” testing Ideally: adequate testing ensures some property (proof by cases) –Origins in [Goodenough & Gerhart], [Weyuker and Ostrand] –In reality: as impractical as other program proofs Practical “adequacy” criteria are really “inadequacy” criteria –If no case from class XX has been chosen, surely more testing is needed...

19. Structural Coverage Testing (In)adequacy criteria –If significant parts of program structure are not tested, testing is surely inadequate Control flow coverage criteria –Statement (node, basic block) coverage –Branch (edge) and condition coverage –Data flow (syntactic dependency) coverage –Various control-flow criteria Attempted compromise between the impossible and the inadequate

20. Basic structural criteria (ex.) a b c d e f Edge ac is required by all-edges but not by all-nodes coverage Typical loop coverage criterion would require zero iterations (cdf), one iteration (cdedf), and multiple iterations (cdededed...df)

21. Data flow coverage criteria (ex.) x := 7 y := x y := y+1 z := x+y 2 reaching definitions (one is from self) 2 reaching definitions for x, and 2 reaching definitions for y Rationale: An untested def- use association could hide an erroneous computation

22. The Infeasibility Problem Syntactically indicated behaviors (paths, data flows, etc.) are often impossible –Infeasible control flow, data flow, and data states Adequacy criteria are typically impossible to satisfy Unsatisfactory approaches: –Manual justification for omitting each impossible test case (esp. for more demanding criteria) –Adequacy “scores” based on coverage example: 95% statement coverage, 80% def-use coverage

23. Challenges in Structural Coverage Interprocedural and gross-level coverage –e.g., interprocedural data flow, call-graph coverage Regression testing Late binding (OO programming languages) –coverage of actual and apparent polymorphism Fundamental challenge: Infeasible behaviors –underlies problems in inter-procedural and polymorphic coverage, as well as obstacles to adoption of more sophisticated coverage criteria and dependence analysis

24. Structural Coverage in Practice Statement and sometimes edge or condition coverage is used in practice –Simple lower bounds on adequate testing; may even be harmful if inappropriately used for test selection Additional control flow heuristics sometimes used –Loops (never, once, many), combinations of conditions

25. Testing for Reliability Reliability is statistical, and requires a statistically valid sampling scheme Programs are complex human artifacts with few useful statistical properties In some cases the environment (usage) of the program has useful statistical properties –Usage profiles can be obtained for relatively stable, pre-existing systems (telephones), or systems with thoroughly modeled environments (avionics)

26. Arbitrary  Random A common error in naïve attempts to obtain statistical confidence measures –Arbitrary distributions may be modeled by adversary functions, not by uniform distributions Example: –If failures were distributed randomly through the execution space of a database program, it would fail at a uniform rate over time. – In reality, it may never fail until a critical table overflows, and then always fail thereafter.

27. Certifying Ultra-High reliability Problem: How can I show that system X has an expected failure rate of 10 -9 /hour? –example: probability that software will ever bring down an Airbus A320 Butler & Finelli estimate –for 10 -9 per 10 hour mission –requires: 10 10 hours testing with 1 computer –or: 10 6 hours (114 years) testing with 10,000 computers [ACM Sigsoft 91, Conf. on SW for Critical Systems]

28. Glimmers of Hope for Measuring High Reliability Random distribution of faults or failures would enable statistical reasoning and classic redundancy techniques –A whole more reliable than its parts Randomization approaches –Blum: Self-checking programs –Lipton: Redundant computations –Podgurski: Kolmogorov complexity Grail or illusion? –Difficult to generalize beyond simple functions

29. Process-Based Reliability Testing Rather than relying only on properties of the program, we may use historical characteristics of the development process Reliability growth models (Musa, Littlewood, et al) project reliability based on experience with the current system and previous similar systems

30. Fault-based testing Given a fault model –hypothesized set of deviations from correct program –typically, simple syntactic mutations; relies on coupling of simple faults with complex faults Coverage criterion: Test set should be adequate to reveal (all, or x%) faults generated by the model –similar to hardware test coverage

31. Fault Models Fault models are key to semiconductor testing –Test vectors graded by coverage of accepted model of faults (e.g., “stuck-at” faults) What are fault models for software? –What would a fault model look like? –How general would it be? Across application domains? Across organizations? Across time? Defect tracking is a start

32. The Budget Coverage Criterion A common answer to “when is testing done” –When the money is used up –When the deadline is reached This is sometimes a rational approach! –Implication 1: Test selection is more important than stopping criteria per se. –Implication 2: Practical comparison of approaches must consider the cost of test case selection

33. Test Selection: Standard Advice Specification coverage is good for selection as well as adequacy –applicable to informal as well as formal specs + Fault-based tests –usually ad hoc, sometimes from check-lists Program coverage last –to suggest uncovered cases, not just to achieve a coverage criterion

34. The Importance of Oracles Much testing research has concentrated on adequacy, and ignored oracles Much testing practice has relied on the “eyeball oracle” –Expensive, especially for regression testing makes large numbers of tests infeasible –Not dependable Automated oracles are essential to cost- effective testing

35. Sources of Oracles Specifications –sufficiently formal (e.g., Z spec) –but possibly incomplete (e.g., assertions in Anna, ADL, APP, Nana) Design models –treated as specifications, as in protocol conformance testing Prior runs (capture/replay) –especially important for regression testing and GUIs; hard problem is parameterization

36. What can be automated? Oracles –assertions; replay; from some specifications Selection (Generation) –scripting; specification-driven; replay variations –selective regression test Coverage –statement, branch, dependence Management

37. Design for Test: 3 Principles Observability –Providing the right interfaces to observe the behavior of an individual unit or subsystem Controllability –Providing interfaces to force behaviors of interest Partitioning –Separating control and observation of one component from details of others Adapted from circuit and chip design

38. Problems & Opportunities Compositionality –for components; for regression Specifications –low entry barrier, incremental payoff Synergy with Analysis –conformance test w/ verified models –“backstop” for unsafe assumptions … (your idea here)

39. A recent success story The Prefix program analysis tool Analyzes C/C++ sources Scans for cleanness bugs, e.g., dereferences to NULL pointers Symbolically executes the program on some paths May miss some errors and generate false alarms Tried on Windows 2000 Located 65,000 potential bugs 28,000 out of which are real bugs