1. Introduction to Software Testing Chapter 3.1 Logic Coverage Paul Ammann & Jeff Offutt

2. 2 Covering Logic Expressions  Logic expressions show up in many situations  Covering logic expressions is required by the US Federal Aviation Administration for safety critical software  Logical expressions can come from many sources – Decisions in programs – FSMs and statecharts – Requirements  Tests are intended to choose some subset of the total number of truth assignments to the expressions

3. 3 Logic Predicates and Clauses  A predicate is an expression that evaluates to a boolean value  Predicates can contain – boolean variables – non-boolean variables that contain >, =, <=, != – boolean function calls  Internal structure is created by logical operators – ¬ – the negation operator –  – the and operator –  – the or operator –  – the implication operator –  – the exclusive or operator –  – the equivalence operator  A clause is a predicate with no logical operators

4. 4 Example (a = n*o) Includes four clauses: (a < b) – relational expression f (z) – boolean-valued function D – boolean variable (m >= n*o) – relational expression  Most predicates have few clauses – It would be nice to quantify that claim!  Sources of predicates – Decisions in programs – Guards in finite state machines – Decisions in UML activity graphs – Requirements, both formal and informal – SQL queries

5. Problem: List all the clauses for the predicate below: ((f 0))  (M  (e < d + c)) Solution: There are four: f 0, M, and e < d + c.

6. Problem: 2 Write the predicate to represent the requirement: List all the wireless mice that either retail for more than $100 or for which the store has more than 20 items. Also list non-wireless mice that retail for more than $50. The predicate describing whether to list a given mouse is: ((mouseType = wireless)  ((retail > 100)  (stock > 20)))  (  (mouseType = wireless)  (retail > 50))

7. Clause  A clause is a predicate that does not contain any of the logical operators.  For example, the predicate (a = b) ∨ C ∧ p(x) contains three clauses: a relational expression (a = b), a boolean variable C the function call p(x).  Because they may contain a structure of their own, relational expressions require special treatment.

8. The Predicate A predicate may be written in a variety of logically equivalent ways. For example, the predicate ((a = b) ∨ C) ∧ ((a = b) ∨ p(x)) is logically equivalent to the predicate given in the previous slide  but ((a = b) ∧ p(x)) ∨ (C ∧ p(x)) is not.  The usual rules of boolean algebra may be used to convert boolean expressions into equivalent forms.

9. Testing and Covering Predicates  We use predicates in testing as follows : – Developing a model of the software as one or more predicates – Requiring tests to satisfy some combination of clauses  Abbreviations: – P is the set of predicates – p is a single predicate in P – C is the set of clauses in P – C p is the set of clauses in predicate p – c is a single clause in C Introduction to Software Testing (Ch 3) © Ammann & Offutt9

10. 10 Predicate and Clause Coverage  Each predicate and each clause be evaluated to both true and false Predicate Coverage (PC) : For each p in P, TR contains two requirements: p evaluates to true, and p evaluates to false. Clause Coverage (CC) : For each c in C, TR contains two requirements: c evaluates to true, and c evaluates to false.

11. Predicate Coverage ((a > b) ∨ C) ∧ p(x) two tests that satisfy predicate coverage are (a = 5, b = 4, C = true, p(x) = true) and (a = 5, b = 6, C = false, p(x) = false). Predicate coverage for the above clause could also be satisfied with the two tests  (a = 5, b = 4, C = true, p(x) = true) and  ( a = 5, b = 4, C = true, p(x) = false) the first two clauses, a = 5, b = 4, never have the value false!

12. Clause Coverage  Predicate ((a > b) ∨ C) ∧ p(x) requires different values to satisfy Clause Coverage Clause coverage requires that (a > b) = true and false, C = true and false, p(x) = true and false. These requirements can be satisfied with two tests: ((a =5, b = 4), (C = true), p(x) = true) and ((a = 5, b = 6), (C = false), p(x) = false).

13. Active Clause Coverage p= a  b ends up with total four requirements in TR. Two for clause a and two for clause b.  For clause a, a determines p if and only if b is false. So we have two test requirements: (a= true, b= false), (a= false, b= false)  For clause b, b determines p if and only if a is false. So we have two requirements (a = false, b= true), (a= false, b= false)) This is summarized in the partial truth table (bold face)

14.  Clause coverage does not subsume predicate coverage,  Predicate coverage does not subsume clause coverage,  Example: The predicate p = a ∨ b. The clauses C are {a, b}. The four test inputs that enumerate the combinations of logical values for the clauses:  Test set T 23 = {2, 3} satisfies clause coverage, but not predicate coverage, because p is never false.  Test set T 24 = {2, 4} satisfies predicate coverage, but not clause coverage, because b is never true. FINALLY: These two test sets demonstrate that neither predicate coverage nor clause coverage subsumes the other.

15. 15 Predicate Coverage Another Example ((a = n*o) predicate coverage Predicate = true a = 5, b = 10, D = true, m = 1, n = 1, o = 1 = (5 = 1*1) = true  true  TRUE = true Predicate = false a = 10, b = 5, D = false, m = 1, n = 1, o = 1 = (10 = 1*1) = false  false  TRUE = false

16. Introduction to Software Testing (Ch 3) © Ammann & Offutt 16 Two tests (a < b) = true a = 5, b = 10 (a < b) = false a = 10, b = 5 D = true D = false m >= n*o = true m = 1, n = 1, o = 1 m >= n*o = false m = 1, n = 2, o = 2 true cases 1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1 false cases 2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2 ((a = n*o) Clause coverage

17. Problems with Predicate and Clause Coverage Introduction to Software Testing (Ch 3) © Ammann & Offutt 17  Predicate Coverage does not fully exercise all the clauses, especially in the presence of short circuit evaluation  Clause Coverage does not always ensure Predicate Coverage –That is, we can satisfy Clause Coverage without causing the predicate to be both true and false –This is definitely not what we want !  The simplest solution is to test all combinations

18. 18 Combinatorial Coverage Combinatorial Coverage requires every possible combination Sometimes called Multiple Condition Coverage Combinatorial Coverage (CoC) : For each p in P, TR has test requirements for the clauses in Cp to evaluate to each possible combination of truth values. a < bDm >= n*o ((a = n*o) 1TTTT 2TTFF 3TFTT 4TFFF 5FTTT 6FTFF 7FFTF 8FFFF

19. Combinatorial Coverage A predicate p with n independent clauses has 2 n possible assignments of truth values.  The general idea is simple and comprehensive  Combinatorial coverage is impractical for predicates with more than a few clauses.  A powerful collection of test criteria that are based on the notion of making individual clauses “active”  We check to see that if we vary a clause in a situation where the clause should affect the predicate. Ifact, the clause does affect the predicate.  Finally: Getting the details right is hard What exactly does “independently” mean ? This idea is simplified as making clauses active … 19

20. Active Clauses  Clause coverage has a weakness : The values do not always make a difference  Consider the first test for clause coverage, which caused each clause to be true: (5 = 1*1)  Only the first clause counts !  To really test the results of a clause, the clause should be the determining factor in the value of the predicate

21. Determination  A clause c i in predicate p, called the major clause, determines p if and only if the values of the remaining minor clauses c j are such that changing c i changes the value of p  This is considered to make the clause active

22. Determining Predicates 22  Goal : Find tests for each clause when the clause determines the value of the predicate P = A  B if B = true, p is always true. so if B = false, A determines p. if A = false, B determines p. P = A  B if B = false, p is always false. so if B = true, A determines p. if A = true, B determines p.

23. 23 p = a  b 1)a = true, b = false 2)a = false, b = false 3)a = false, b = true 4)a = false, b = false Active Clause Coverage  Ambiguity : Do the minor clauses have to have the same values when the major clause is true and false? Active Clause Coverage (ACC) : For each p in P and each major clause c i in Cp, choose minor clauses c j, j != i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Duplicate a is major clauseb is major clause

24. 24 Resolving the Ambiguity  This question caused confusion among testers for years  Considering this carefully leads to three separate criteria : –Minor clauses do not need to be the same –Minor clauses do need to be the same –Minor clauses force the predicate to become both true and false p = a  (b  c) Major clause : a a = true, b = false, c = true a = false, b = false, c = false c = false Is this allowed ?

25. 25 Example : p = (a ∨ b) ∧ c p = (a ∨ b) ∧ c Finding values for predicate coverage is easy and was already shown in Section 3.2. Two test requirements are TR PC = {p = true, p = false} Finding values for predicate coverage is easy. Test Requiremet PC = {p = true, p = false} and they can be satisfied with the following values for the clauses: Suppose that clauses a, b, and c were defined in terms of Java program variables as follows The complete expanded predicate is : p = (x < y ∨ done) ∧ list.contains(str)

26. 26 Test Requirements for Predicate Coverage.  The values for the program variables need not be the same in a particular test case if the goal is to set a clause to a particular value.  For example clause a is true in both tests, even though program variables x and y have different values.

27. 27. Satisfying The Same Example as Clause Coverage TR CC = {a = true, a = false, b = true, b = false, c = true, c = false} They can be satisfied with the following values for the clauses (blank cells represent are « don’t-care values»):

28. 28 Require all combinations of values for the clauses. We have eight test requirements, which can be satisfied with the following values: Combinatorial Coverage

29. 29 General Active Clause Coverage  General active clause coverage requires that each major clause be true and false and the minor clauses be such that the major clause determines the value of the predicate.  Similarly to clause coverage, three pairs of test requirements can be defined: TR GACC = {(a = true ∧ p a, a = false ∧ p a ), (b = true ∧ p b, b = false ∧ p b ), (c = true ∧ p c, c = false ∧ p c )}

30. 30  The test requirements can be satisfied with the following table for the clauses.  These can be the same as with clause coverage The exception is that the blank cells from clause coverage are replaced with the values  Values for major clauses are indicated with upper case letters in boldface. The duplication; the first and fifth rows are identical, and the second and fourth are identical. Four tests General Active Clause Cov. Four tests are needed to satisfy General Active Clause Cov.

31. Exercise p =a  (b  c) i)Clauses are a, b, c ii)Compute (and simplify) the conditions under which each of the clauses determines predicate p p a =  b  c p b =  a  c p c =  a  b iii) Write the complete truth table for all clauses. Introduction to Software Testing (Ch 3) © Ammann & Offutt 31 p = a _ (b ^ c)