Presentation on theme: "Test Case Design Methodologies (Black-box methods)"— Presentation transcript:
1Test Case Design Methodologies (Black-box methods) P. Pavankumar, CSTEModerator,
2Agenda Black-box testing Test Case – principles Test Case Design Equivalence PartitioningBoundary Value AnalysisCause – Effect GraphError Guessing
3Black-box testingIn a Black-box testing the tester is completely unconcerned about the internal behaviour and structure of the program (tester views the program as a black box).The tester is only interested in finding circumstances in which the program does not behave according to its specifications.Test data are derived solely from the specifications.Black-box testing strategy is also called as data-driven or input/output-driven testing.
4Black-box testingTo find all the errors in the program the criterion is exhaustive input testing.Exhaustive input testing is the use of every possible input condition as a test case.Exhaustive testing is next to impossible and therefore a test of any program must be necessarily incomplete (i.e. the testing cannot guarantee the absence of all errors).The obvious strategy then is to try to reduce this incompleteness as much as possible.
5Test Case – principles* A good test case is one that has a high probability of detecting an as-yet undiscovered error.A successful test case is one that detects an as-yet undiscovered error.A necessary part of the test case is a definition of the expected output or resultTest cases must be written for invalid and unexpected, as well as valid and expected input conditions* Courtesy: The Art of Software Testing - Myers
6Test Case DesignThe most important consideration in software testing is the design or invention of effective test cases.The reason for the importance of test-case design stems from the fact that “complete” testing is impossible.Given constraints on time, cost etc., the key issue of testing becomes What subset of all possible test cases has the highest probability of detecting the most errors?The study of test-case-design methodologies supplies reasonable answers to the above question.
7Equivalence Partitioning Equivalence Partitioning is a black-box design method where test cases are developed by partition the input domain of a program into a finite number of equivalence classes such that one can reasonably assume (but, of course, not be absolutely sure) that test of a representative value of each class is equivalent to a test of any other value.That is, if one test case in an equivalence class detects an error, all other test cases in the equivalence class would be expected to find the same error.Conversely, if the test case did not detect an error, we would expect that no other test case in the equivalence class would find an error (unless a subset of the equivalence class falls within any other equivalence class, since equivalence classes may overlap one another).
8Equivalence Partitioning Input domain1234Input domainpartitioned into foursub-domains.Four test inputs, oneselected from each sub-domain.Too many test inputs.
9How to partition? Inputs to a program provide clues to partitioning. Example 1:Suppose that program P takes an integer X as input.For X< 0 the program is required to perform task T1 and for X>=0 task T2.
10How to partition? (contd.) The input domain is prohibitively large because X can assume a large number of values.However, we expect P to behave the same way for all X<0.Similarly, we expect P to perform the same way for all values of X>=0.We therefore partition the input domain of P into two sub-domains.
11Two sub-domains X<0 X>=0 One test case: X=-3 Another test case: Equivalence classAnother test case:X=15All test inputs in the X<0 sub-domain are considered equivalent(there behaviour is same i.e. they are less than X).The assumption is that if one test input in this sub-domain revealsan error in the program, so will the others.This is true of the test inputs in the X>=0 sub-domain also.
12Non-overlapping Partitions In the previous example, the two equivalence classes are non-overlapping. In other words the two sub-domains are disjoint.When the sub-domains are disjoint, it is sufficient to pick one test input from each equivalence class to test the program.
13Non-overlapping Partitions An equivalence class is considered covered when at least one test has been selected from it.In partition testing our goal is to cover all equivalence classes.
14Overlapping Partitions Example 2:Suppose that program P takes three integers X, Y and Z. It is known that:X < YZ > Y
16Overlapping Partitions – Test Selection In this example, we could select 4 test cases as:X=4, Y=7, Z=1 satisfies X<YX=4, Y=2, Z=1 satisfies X>=YX=1, Y=7, Z=9 satisfies Z>YX=1, Y=7, Z=2 satisfies Z<=YThus, we have one test case from each equivalence class.
17Overlapping Partitions – Test Selection However, we may also select only 2 test inputs and satisfy all four equivalence classes:X=4, Y=7, Z=1 satisfies X<Y and Z<=YX=4, Y=2, Z=3 satisfies X>=Y and Z>YThus, we have reduced the number of test cases from 4 to 2 while covering each equivalence class.
18Partitioning using non-numeric data In the previous two examples the inputs were integers. One can derive equivalence classes for other types of data also.Example 3:Suppose that program P takes one character X and one string Y as inputs. P performs task T1 for all lower case characters and T2 for upper case characters. Also, it performs task T3 for the null string and T4 for all other strings.
19Partitioning using non-numeric data X: lc, Y: not nullX: lcX:UCY: nullY: not nullX: UC, Y: not nullX: lc, Y: nullX: UC, Y: nulllc: Lower case characterUC: Upper case characternull: null string.
20Non-numeric Data Once again we have overlapping partitions. We can select only 2 test inputs to cover all four equivalence classes.These are:X: lower case, Y: null stringX: upper case, Y: not a null string
21Guidelines to Equivalence Partitioning Input condition specifies a range: create one for the valid case and two for the invalid cases.e.g. for a<=X<=b the classes area<=X<=b (valid case)X<a and X>b (the invalid cases)
22Guidelines (contd..)Input condition specifies a value: create one for the valid value and two for incorrect values (below and above the valid value).e.g. (“select minimum of one and maximum of 6 employees from the list of employees”)-- identify one valid equivalence class and-- two invalid equivalence classes (no employees and more than 6 employees)This may not be possible for certain data types, e.g. for Boolean (yes / no).
23Guidelines (contd..)Input condition specifies a member of a set: create one for the valid value and one for the invalid (not in the set) value.e.g. type of the programming languages must be C++, Java, Perl,-- Identify a valid equivalence class for each and-- One invalid equivalence class (e.g. C#)
24Equivalence Partitioning - Facts Equivalence Partitioning is vastly superior to a random selection of test cases.It uses the fewest test cases possible to cover the most input requirements.Although some guidelines are given to assist in the identification of equivalence classes, doing so is mostly an intuitive process.It overlooks certain types of high yield test cases that boundary-value analysis can determine.
25Boundary-value Analysis Boundary Analysis is black-box technique that consists of developing test cases and data that focus on the input and output boundaries of a given function.Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not.Boundary conditions are those situations directly on, above and beneath the edges of input equivalence classes and output equivalence classes.
26Boundary-value Analysis Boundary-value analysis differs from equivalence portioning in two respectsRather than selecting any element in an equivalence class as being representative, boundary-value analysis requires that one or more elements be selected such that each edge of the equivalence class is the subject to a test.Rather than just focusing attention on the input conditions (input space), test cases are also derived by considering the result space (i.e. output equivalence classes).
27Guidelines - Boundary-value Analysis If an input condition specifies a range of values, write test cases for the ends of the range, and invalid input test cases for situation just beyond the ends.For instance if the valid domain of an input value is -1.0 and +1.0 write test cases fro the situations -1.0, 1.0, and 1.001
28Guidelines - Boundary-value Analysis If and input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values.For instance if an input file can contain records, write test cases for 0,1,255, and 256 records.
29Guidelines - Boundary-value Analysis Use guideline 1 for each output condition.For instance, if a program computes the monthly tax deduction and if the minimum is rs.0.00 and the maximum is rs.1000 write test cases that cause rs.0.00 and rs.1000 to be deducted.Also, see if it is possible to invent test cases that might cause a negative deduction or a deduction more rs.1000
30Guidelines - Boundary-value Analysis Use guideline 2 for each output condition.For instance, if an information retrieval system displays the most-relevant abstracts based on the input request, but never more than four abstracts,write test cases such that the program displays zero, one and four abstractswrite a test case that might cause the program to erroneously display five abstracts.
31Boundary-value Analysis - example For example, a program which edits credit limits within a given range (10000 – 15,000) would havethree equivalence classes:Less than 10,000 (invalid)Between 10,000 and 15,000 (valid)Greater than 15,000 (invalid)In same credit limit example,boundary analysis would test:Low boundary plus or minus one (9,999 and 10,001)On the boundary (10,000 and 15,000)Upper boundary plus or minus one (14,999 and 15,001)
32Boundary-value Analysis - Facts Boundary-value compliments Equivalence PartitioningThe technique on the surface sounds simple, but boundary conditions may be very subtle and hence identification of them requires a lot of thought.One weakness of Boundary-value analysis and Equivalence Partitioning is that they do not explore combinations of input circumstances.
33Cause-Effect Graphing Cause Effect Graphing is a systematic method of generating test cases representing combinations of conditions.The graph is actually a digital logic circuit (a combination logic network), but rather than using standard electronics notation, a somewhat simpler notation is used.No knowledge of electronics is necessary other than an understanding of Boolean logic (i.e. understating the logic operators and, or, and not)
34Cause-Effect Graphing A CAUSE may be thought of as a distinct input condition, or an ‘‘equivalence class’’ of input conditions.An EFFECT may be thought of as a distinct output condition, or a meaningful change in program state.Causes and Effects are represented as Boolean variables and the logical relationships among them CAN (but need not) be represented as one or more Boolean graphs.
36Cause-Effect Graphing Process to derive test casesThe specification is divided into “workable” pieces.The CAUSES and EFFECTS in the specification are identified.Causes and Effects are identified by reading the specification word by word and underlining words or phases that describe causes and effects.Each cause and effect is assigned a unique number.
37Cause-Effect Graphing Process to derive test casesThe semantic content of the specification is analyzed and transformed into a Boolean graph linking the causes and effects. This is the cause-effect graph.The graph is annotated with constraints describing combinations of cause and/or effects that are impossible because of syntactic or environmental constraints.By methodically tracing state conditions in the graph the graph is converted into a limited-entry decision table. Each column in the table represents a test case.The columns in the decision table are converted into test cases.
38Cause-Effect Graphing - Example The character in column 1 must be an “A” or a “B”. The character in column 2 must be digit. In this situation the file update is made. If the first character is incorrect, message X12 is issued. If the second character is not a digit, message X13 is issued.The causes are:1 – character in column 1 is “A”2 - character in column 1 is “B”3 - character in column 2 is a digitAnd the effects are:70 – update made71 – message X12 is issued72 – message X13 is issued
39Cause-Effect Graphing Cause-Effect graph of the exampleNotice the intermediate node 11 that was created.The graph represents the specification by setting all possible states of the causes and seeing that the effects are set to the correct values. Do achieve this we create dummy nodes like 11.70111271723
40Cause-Effect Graphing Cause-Effect graph of the exampleNotice the intermediate node 11 that was created.The graph represents the specification by setting all possible states of the causes and seeing that the effects are set to the correct values. Do achieve this we create dummy nodes like 11.70111271723
41Cause-Effect Graphing Cause-Effect graph - apply the constraintsIn most programs, certain combinations of causes are impossible because of syntactic or environmental considerations (e.g. a character cannot be an “A” and a “B” simultaneously). To account such scenarios we use constraintsExclusive Constraints (at most one of ‘a’ and ‘b’ can be 1)Inclusive Constraint (at least one of ‘a’, ‘b’, and ‘c’ must always be 1)One and Only One (one and only one of ‘a’ and ‘b’ must be 1)Requires (for ‘a’ to be 1, ‘b’ must be 1)Masks (constraint among effects – if effect ‘a’ is 1, effect ‘b’ is forced to 0)
42Cause-Effect Graphing Cause-Effect graph - apply the constraintsConstraints SymbolsaExclusiveaInclusiveEbbIC
43Cause-Effect Graphing Cause-Effect graph after applying Exclusive constraints701E11271723
44Cause-Effect Graphing Cause-Effect graph after applying Exclusive constraints701E11271723
45Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect Graph12345CAUSESCharacter ACharacter BDigitEFFECTSUpdate madeMessage X12Message X13
46Cause-Effect Graphing Designing a Decision Table from the Cause-Effect Graph70≠ A1A≠ BE11B271Di≠ Di723
47Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphPossible Values12345CAUSESCharacter A≠Character BDigitEFFECTSUpdate madeMessage X12Message X13
48Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphApplying Relationships / Combinations12345CAUSESCharacter A or BCharacter BDigitEFFECTSUpdate madeXMessage X12Message X13
49Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphApplying Relationships / Combinations12345CAUSESCharacter A or B≠Character BDigitEFFECTSUpdate madeXMessage X12Message X13
50Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphApplying Relationships / Combinations12345CAUSESCharacter A or B≠Character BDigitEFFECTSUpdate madeXMessage X12Message X13
51Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphEach Column is one Test Case.123CAUSESCharacter A or B≠Character BDigitEFFECTSUpdate madeXMessage X12Message X13
52Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphEach Column is one Test Case.123CAUSESCharacter A or B≠Character BDigitEFFECTSUpdate madeXMessage X12Message X13
53Cause-Effect Graphing - Example Designing a Decision Table from the Cause-Effect GraphEach Column is one Test Case.123CAUSESCharacter A or B≠Character BDigitEFFECTSUpdate madeXMessage X12Message X13
54Cause-Effect Graphing - Facts Since cause-effect graphing requires the translation of a specification into a Boolean logic network, it gives one a different perspective on, and additional insight into, the specification. It’s a good way to uncover ambiguities and incompleteness in specifications.Cause-Effect graph does not adequately explore boundary conditions.The raise in the Causes and Effects increases the complexity of drawing the graphThe most difficult aspect of the technique is the conversion of the graph into the decision table. Improper conversion can lead to inaccurate test cases.
55Error GuessingError Guessing is bases on the theory that test cases can be developed based upon the intuition and experience of the Test Engineer.For example if an input is the date, a test engineer may try February 29, 2000 or 9/9/0099
56A Reasonable StrategyThe test-case design methodologies discussed can be combined into an overall strategy. The reason for combining them should be obvious by now; each contributes a particular set of useful test cases, but none of them by itself contributes a thorough set of test cases.Again, the use of this strategy will not guarantee that all errors will be found, but it help in eliminating maximum errors.
57Thank you P. Pavankumar, CSTE Moderator, firstname.lastname@example.org