Presentation is loading. Please wait.

Presentation is loading. Please wait.

Software testing Main issues:

Similar presentations


Presentation on theme: "Software testing Main issues:"— Presentation transcript:

1 Software testing Main issues:
There are a great many testing techniques Often, only the final code is tested © SE, Testing, Hans van Vliet

2 Nasty question Suppose you are being asked to lead the team to test the software that controls a new X-ray machine. Would you take that job? Would you take it if you could name your own price? What if the contract says you’ll be charged with murder in case a patient dies because of a mal-functioning of the software? As an exercise, refer the students to the Therac-case, discussed in chapter 1. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

3 Overview Preliminaries All sorts of test techniques
Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

4 State-of-the-Art 30-85 errors are made per 1000 lines of source code
extensively tested software contains errors per 1000 lines of source code testing is postponed, as a consequence: the later an error is discovered, the more it costs to fix it. error distribution: 60% design, 40% implementation. 66% of the design errors are not discovered until the software has become operational. It thus really pays off to start testing early. See also following picture SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

5 Relative cost of error correction
100 50 20 10 5 Old picture from Boehm’s book. It shows that errors discovered during operation might cost 100 times as much as errors discovered during requirements engineering. 2 1 RE design code test operation SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

6 Lessons Many errors are made in the early phases
These errors are discovered late Repairing those errors is costly  It pays off to start testing real early SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

7 How then to proceed? Exhaustive testing most often is not feasible
Random statistical testing does not work either if you want to find errors Therefore, we look for systematic ways to proceed during testing As a simple illustration of why exhaustive testing does not work: take a simple loop with an if statement in it. Exhaustive testing if the loop is executed 100 times takes 2100 test cases Random testing does work if you want to achieve reliability (see later sections/slides). SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

8 Classification of testing techniques
Classification based on the criterion to measure the adequacy of a set of test cases: coverage-based testing fault-based testing error-based testing Classification based on the source of information to derive test cases: black-box testing (functional, specification-based) white-box testing (structural, program-based) Coverage-based: e.g. how many statements or requirements have been tested so far Fault-based: e.g., how many seeded faults are found Error-based: focus on error-prone points, e.g. off-by-one points Black-box: you do not look inside, but only base yourself on the specification/functional description White-box: you do look inside, to the structure, the actual program/specification. This classification is mostly used at the module level. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

9 Some preliminary questions
What exactly is an error? How does the testing process look like? When is test technique A superior to test technique B? What do we want to achieve during testing? When to stop testing? SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

10 Error, fault, failure an error is a human activity resulting in software containing a fault a fault is the manifestation of an error a fault may result in a failure For example, I may accidentally assume a procedure is only called with a positive argument (the error). So I forget to test for negative values (the fault). Now if the procedure is actually called with a negative argument, something may go wrong (wrong answer, abortion): the failure Note that the relation between errors, faults and failures need not be 1-1. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

11 When exactly is a failure a failure?
Failure is a relative notion: e.g. a failure w.r.t. the specification document Verification: evaluate a product to see whether it satisfies the conditions specified at the start: Have we built the system right? Validation: evaluate a product to see whether it does what we think it should do: Have we built the right system? But even with this definition, things may be subtle. Suppose a program contains a fault which never shows up, say because a certain piece of the code never gets executed. Is this “latent” fault actually a fault? If not, does it become a fault if we reuse this part of the program in another context? See also next slide. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

12 Point to ponder: maiden flight of Ariane 5
The Ariane 5 took off and exploded within 40 seconds. Ultimate cause: overflow in conversion of some variable; this case was not tested. In the Ariane 4, this did not cause any problem. This variable related to the horizontal speed of the rocket. The piece of software in question only served to speed up the restart of the launching process in case something went wrong and one had to stop the launch prematurily. The software ran for about a minute after launching. The Ariane 4 is much slower than the Ariane 5, so within this one minute, the rocket was still going up, and the variable in question had a small value. In the Ariane 5, by this time horizontal speed was much higher. So, failure to specify boundary conditions for this software? Reuse failure? SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

13 Testing process subset of input expected output oracle input test
strategy test results compare P P subset of input real output If exhausting testing does not work, we have to select a good subset. But how do we determine the quality of such a test set? This is a very crucial step, and the various test techniques all address this issue in one way or another. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

14 Test adequacy criteria
Specifies requirements for testing Can be used as stopping rule: stop testing if 100% of the statements have been tested Can be used as measurement: a test set that covers 80% of the test cases is better than one which covers 70% Can be used as test case generator: look for a test which exercises some statements not covered by the tests so far A given test adequacy criterion and the associated test technique are opposite sides of the same coin Note that the stopping rule view is a special case of the measurement view. We use these adequacy criteria to decide whether one testing technique is better than another. A number of such relations between test techniques is given later on. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

15 What is our goal during testing?
Objective 1: find as many faults as possible Objective 2: make you feel confident that the software works OK Objective 1 is the kind of objective used in all kinds of functional and structural test techniques. These try to systematically exercise the software so as to make sure we test “everything”. The idea behind objective 2 is that we might not be interested in faults that never show up, but we really want to find those that hae a large probability of manifesting themselves. So we pursue a high reliability. Random testing then works, provided the test cases profile matches the operational profile, I.e. the distribution of test cases mimics actual use of the system. An example development method where this objective is applied is Cleanroom. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

16 Example constructive approach
Task: test module that sorts an array A[1..n]. A contains integers; n  1000 Solution: take n = 0, 1, 37, 999, For n = 37, 999, take A as follows: A contains random integers A contains increasing integers A contains decreasing integers These are equivalence classes: we assume that one element from such a class suffices This works if the partition is perfect This as example approach where we want to find as many faults as possible. The partition is perfect iff the paths in the program follow the equivalence classes chosen. For instance, we assume that the sorting module treat all arrays of length 1 < n < 999 the same. Probably, those of length 1 an 999 are also treated in the same way, but just to make sure we test these boundary cases separately. Now if the sorting program treats, say, arrays with negative numbers differently from those with positive numbers, this equivalence class partitioning is not perfect, and a fault in the program may go unnoticed because we may happen to use a test case that, say, only has positive numbers, and none that has negative numbers. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

17 Testing models Demonstration: make sure the software satisfies the specs Destruction: try to make the software fail Evaluation: detect faults in early phases Prevention: prevent faults in early phases The first two models are phase models; testing is a phase following coding. The demonstration mode is often used when testing one’s own software. This model also applies when the test set is not carefully/systematically constructed. All kinds of structural and functional techniques follow the destructive mode of operation. The last two models acknowledge that testing is something that has to be done in every development phase. For instance, requirements can be reviewed too. And by making sure that there is a test for every requirement, including every non-functional requirement, you can even prevent errors from being made in the first place. Over the years, a gradual shift can be observed, from demonstration to prevention. time SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

18 Testing and the life cycle
requirements engineering criteria: completeness, consistency, feasibility, and testability. typical errors: missing, wrong, and extra information determine testing strategy generate functional test cases test specification, through reviews and the like design functional and structural tests can be devised on the basis of the decomposition the design itself can be tested (against the requirements) formal verification techniques the architecture can be evaluated SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

19 Testing and the life cycle (cnt’d)
implementation check consistency implementation and previous documents code-inspection and code-walkthrough all kinds of functional and structural test techniques extensive tool support formal verification techniques maintenance regression testing: either retest all, or a more selective retest SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

20 Test-Driven Development (TDD)
First write the tests, then do the design/implementation Part of agile approaches like XP Supported by tools, eg. JUnit Is more than a mere test technique; it subsumes part of the design work SE, Testing, Hans van Vliet, ©2008

21 Steps of TDD Add a test Run all tests, and see that the system fails
Make a small change to make the test work Run all tests again, and see they all run properly Refactor the system to improve its design and remove redundancies SE, Testing, Hans van Vliet, ©2008

22 Test Stages module-unit testing and integration testing system testing
bottom-up versus top-down testing system testing acceptance testing installation testing SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

23 Test documentation (IEEE 928)
Test plan Test design specification Test case specification Test procedure specification Test item transmittal report Test log Test incident report Test summary report SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

24 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

25 Manual Test Techniques
static versus dynamic analysis compiler does a lot of static testing static test techniques reading, informal versus peer review walkthrough and inspections correctness proofs, e.g., pre-and post-conditions: {P} S {Q} stepwise abstraction Correctness proofs: complex, not one very often Stepwise abstraction: opposite of stepwise refinement, so you develop pre and post-conditions of a module by working backwards from the individual statements SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

26 (Fagan) inspection Going through the code, statement by statement
Team with ~4 members, with specific roles: moderator: organization, chairperson code author: silent observer (two) inspectors, readers: paraphrase the code Uses checklist of well-known faults Result: list of problems encountered Much of the real value of this type of technique is in the learning process that the peole get involved in. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

27 Example checklist Wrong use of data: variable not initialized, dangling pointer, array index out of bounds, … Faults in declarations: undeclared variable, variable declared twice, … Faults in computation: division by zero, mixed-type expressions, wrong operator priorities, … Faults in relational expressions: incorrect Boolean operator, wrong operator priorities, . Faults in control flow: infinite loops, loops that execute n-1 or n+1 times instead of n, ... SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

28 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

29 Coverage-based testing
Goodness is determined by the coverage of the product by the test set so far: e.g., % of statements or requirements tested Often based on control-flow graph of the program Three techniques: control-flow coverage data-flow coverage coverage-based testing of requirements SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

30 Example of control-flow coverage
procedure bubble (var a: array [1..n] of integer; n: integer); var i, j: temp: integer; begin for i:= 2 to n do if a[i] >= a[i-1] then goto next endif; j:= i; loop: if j <= 1 then goto next endif; if a[j] >= a[j-1] then goto next endif; temp:= a[j]; a[j]:= a[j-1]; a[j-1]:= temp; j:= j-1; goto loop; next: skip; enddo end bubble; input: n=2, a[1] = 5, a[2] = 3 SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

31 Example of control-flow coverage (cnt’d)
procedure bubble (var a: array [1..n] of integer; n: integer); var i, j: temp: integer; begin for i:= 2 to n do if a[i] >= a[i-1] then goto next endif; j:= i; loop: if j <= 1 then goto next endif; if a[j] >= a[j-1] then goto next endif; temp:= a[j]; a[j]:= a[j-1]; a[j-1]:= temp; j:= j-1; goto loop; next: skip; enddo end bubble; a[i]=a[i-1] input: n=2, a[1] = 5, a[2] = 3 SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

32 Control-flow coverage
This example is about All-Nodes coverage, statement coverage A stronger criterion: All-Edges coverage, branch coverage Variations exercise all combinations of elementary predicates in a branch condition Strongest: All-Paths coverage ( exhaustive testing) Special case: all linear independent paths, the cyclomatic number criterion In branch coverage, both branches of an if-statement are tested, even if one is empty In normal branch coverage, a combined condition like a = 1 and b = 2 requires two tests. We may also test al four combinations of the two simple predicates. The cyclomatic number criterion is related to the cyclomatic complexity metric of McCabe SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

33 Data-flow coverage Looks how variables are treated along paths through the control graph. Variables are defined when they get a new value. A definition in statement X is alive in statement Y if there is a path from X to Y in which this variable is not defined anew. Such a path is called definition-clear. We may now test all definition-clear paths between each definition and each use of that definition and each successor of that node: All-Uses coverage. We have to include each successor to enforce that all branches following a P-use are taken. Further variations differentiate between uses in a predicate (C-use) and uses elsewhere (computations, C-use). This leads to criteria like All-C-uses/Some-P-uses and the like. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

34 Coverage-based testing of requirements
Requirements may be represented as graphs, where the nodes represent elementary requirements, and the edges represent relations (like yes/no) between requirements. And next we may apply the earlier coverage criteria to this graph SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

35 Example translation of requirements to a graph
A user may order new books. He is shown a screen with fields to fill in. Certain fields are mandatory. One field is used to check whether the department’s budget is large enough. If so, the book is ordered and the budget reduced accordingly. Enter fields Notify user All mandatory fields there? Check budget Order book Notify user SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

36 Similarity with Use Case success scenario
User fills form Book info checked Dept budget checked Order placed User is informed Enter fields Notify user All mandatory fields there? Check budget Order book Notify user SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

37 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

38 Fault-based testing In coverage-based testing, we take the structure of the artifact to be tested into account In fault-based testing, we do not directly consider this artifact We just look for a test set with a high ability to detect faults Two techniques: Fault seeding Mutation testing SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

39 Fault seeding © SE, Testing, Hans van Vliet
Kinds of variations in program testing: seed faults founded by one group into the program tested by another group. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

40 Mutation testing procedure insert(a, b, n, x);
begin bool found:= false; for i:= 1 to n do if a[i] = x then found:= true; goto leave endif enddo; leave: if found then b[i]:= b[i] + 1 else n:= n+1; a[n]:= x; b[n]:= 1 endif end insert; n-1 2 In each variation, mutant, one simple change is made. - SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

41 Mutation testing (cnt’d)
procedure insert(a, b, n, x); begin bool found:= false; for i:= 1 to n do if a[i] = x then found:= true; goto leave endif enddo; leave: if found then b[i]:= b[i] + 1 else n:= n+1; a[n]:= x; b[n]:= 1 endif end insert; n-1 Note that if we happen to insert an element of a that occurs before the final element, we won’t notice a difference SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

42 How tests are treated by mutants
Let P be the original, and P’ the mutant Suppose we have two tests: T1 is a test, which inserts an element that equals a[k] with k<n T2 is another test, which inserts an element that does not equal an element a[k] with k<n Now P and P’ will behave the same on T1, while they differ for T2 In some sense, T2 is a “better” test, since it in a way tests this upper bound of the for-loop, which T1 does not SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

43 How to use mutants in testing
If a test produces different results for one of the mutants, that mutant is said to be dead If a test set leaves us with many live mutants, that test set is of low quality If we have M mutants, and a test set results in D dead mutants, then the mutation adequacy score is D/M A larger mutation adequacy score means a better test set SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

44 Strong vs weak mutation testing
Suppose we have a program P with a component T We have a mutant T’ of T Since T is part of P, we then also have a mutant P’ of P In weak mutation testing, we require that T and T’ produce different results, but P and P’ may still produce the same results In strong mutation testing, we require that P and P’ produce different results SE, Testing, Hans van Vliet, ©2008

45 Assumptions underlying mutation testing
Competent Programmer Hypothesis: competent programmers write programs that are approximately correct Coupling Effect Hypothesis: tests that reveal simple fault can also reveal complex faults SE, Testing, Hans van Vliet, ©2008

46 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008

47 Error-based testing Decomposes input (such as requirements) in a number of subdomains Tests inputs from each of these subdomains, and especially points near and just on the boundaries of these subdomains -- those being the spots where we tend to make errors In fact, this is a systematic way of doing what experienced programmers do: test for 0, 1, nil, etc SE, Testing, Hans van Vliet, ©2008

48 Error-based testing, example
Example requirement: Library maintains a list of “hot” books. Each new book is added to this list. After six months, it is removed again. Also, if book is more than four months on the list, and has not been borrowed more than four times a month, or it is more than two months old and has been borrowed at most twice, it is removed from the list. SE, Testing, Hans van Vliet, ©2008

49 Example (cnt’d) av # of loans A B 5 2 2 4 6 age
This is a graphical view of this same requirement. It shows the two-dimensional (age, average number of loans per month) domain. The subdomains are bordered by lnes such as age=6, or (age=4, 0<= av<= 5). For each border, it is indicated which of the adjacent subdomains is closed by putting a hachure at that side; a subdomain is closed at some border iff that border belongs to the subdomain; otherwise it is open. 2 2 4 6 age SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

50 Strategies for error-based testing
An ON point is a point on the border of a subdomain If a subdomain is open w.r.t. some border, then an OFF point of that border is a point just inside that border If a subdomain is closed w.r.t. some border, then an OFF point of that border is a point just outside that border So the circle on the line age=6 is an ON point of both A and B The other circle is an OFF point of both A and B SE, Testing, Hans van Vliet, ©2008

51 Strategies for error-based testing (cnt’d)
Suppose we have subdomains Di, i=1,..n Create test set with N test cases for ON points of each border B of each subdomain Di, and at least one test case for an OFF point of each border This set is called N1 domain adequate SE, Testing, Hans van Vliet, ©2008

52 Application to programs
if x < 6 then … elsif x > 4 and y < 5 then … elsif x > 2 and y <= 2 then … else ... This yields the same picture, with the same borders, and can be used with the same test set. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

53 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008

54 Comparison of test adequacy criteria
Criterion A is stronger than criterion B if, for all programs P and all test sets T, X-adequacy implies Y-adequacy In that sense, e.g., All-Edges is stronger that All-Nodes coverage (All-Edges “subsumes” All-Nodes) One problem: such criteria can only deal with paths that can be executed (are feasible). So, if you have dead code, you can never obtain 100% statement coverage. Sometimes, the subsumes relation only holds for the feasible version. Usually, stronger metrics induce more costs SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

55 Desirable properties of adequacy criteria
applicability property non-exhaustive applicability property monotonicity property inadequate empty set property antiextensionality property general multiplicity change property antidecomposition property anticomposition property renaming property complexity property statement coverage property These properties relate to program-based criteria. The first four are rather general and should apply to any test adequacy criterion. E.G. the applicability criterion says:for every program, there is an adequate test set. This is not true for the All-Nodes and Al-Edges criteria, e.g., since they may have dead code, so that you cannot achieve 100% coverage. Anticomposition: if components have been tested adequately, this does not mean their composition is also tested adequately (re Ariane 5 disaster). This does not hold for the All-Nodes and All-Edges. SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet

56 Experimental results There is no uniform best test technique
The use of multiple techniques results in the discovery of more faults (Fagan) inspections have been found to be very cost effective Early attention to testing does pay off SE, Testing, Hans van Vliet, ©2008

57 Overview Preliminaries All sorts of test techniques
manual techniques coverage-based techniques fault-based techniques error-based techniques Comparison of test techniques Software reliability SE, Testing, Hans van Vliet, ©2008

58 Software reliability Interested in expected number of failures (not faults) … in a certain period of time … of a certain product … running in a certain environment SE, Testing, Hans van Vliet, ©2008

59 Software reliability: definition
Probability that the system will not fail during a certain period of time in a certain environment SE, Testing, Hans van Vliet, ©2008

60 Failure behavior Subsequent failures are modeled by a stochastic process Failure behavior changes over time (e.g. because errors are corrected)  stochastic process is non-homogeneous () = average number of failures until time  () = average number of failures at time  (failure intensity) () is the derivative of () SE, Testing, Hans van Vliet, ©2008

61 Failure intensity () and mean failures ()
SE, Testing, Hans van Vliet, ©2008

62 Operational profile Input results in the execution of a certain sequence of instructions Different input  (probably) different sequence Input domain can thus be split in a series of equivalence classes Set of possible input classes together with their probabilities SE, Testing, Hans van Vliet, ©2008

63 Two simple models Basic execution time model (BM)
Decrease in failure intensity is constant over time Assumes uniform operational profile Effectiveness of fault correction is constant over time Logarithmic Poisson execution time model (LPM) First failures contribute more to decrease in failure intensity than later failures Assumes non-uniform operational profile Effectiveness of fault correction decreases over time SE, Testing, Hans van Vliet, ©2008

64 Estimating model parameters (for BM)
SE, Testing, Hans van Vliet, ©2008

65 Summary Do test as early as possible Testing is a continuous process
Design with testability in mind Test activities must be carefully planned, controlled and documented. No single reliability model performs best consistently SE, Testing, Hans van Vliet, ©2008 © SE, Testing, Hans van Vliet


Download ppt "Software testing Main issues:"

Similar presentations


Ads by Google