Presentation is loading. Please wait.

Presentation is loading. Please wait.

Some Testing Techniques (enhanced 6 + 4) Course Software Testing & Verification 2013/14 Wishnu Prasetya.

Similar presentations

Presentation on theme: "Some Testing Techniques (enhanced 6 + 4) Course Software Testing & Verification 2013/14 Wishnu Prasetya."— Presentation transcript:

1 Some Testing Techniques (enhanced 6 + 4) Course Software Testing & Verification 2013/14 Wishnu Prasetya

2 Plan Unit testing  tool Mocking (Sec )  enhanced Black box testing  Input Partitioning (Ch 4). Regression (6.1)  enhanced 2

3 Unit Testing Making sure that the units are correct. Insufficient unit testing can be costly: debugging an error at the system-test level is much more costly than at the unit level (some reports suggest more than 10x). Don’t confuse the concept of “unit testing” and “unit testing tool”; the later can often also be used to facilitate integration and system testing. 3

4 But what is a “unit” ? In principle, you decide. “Units” are just something you can compose to build something bigger. Possibilities: function, method, or class. But be aware that different types of units may have types of interactions and complexity, thus requiring different testing approach – a function’s behavior depends only on its parameters; does not do any side effect. – procedure depends-on/affects params and global variables – method: static vars, params, instance vars – class: is a collection of interacting methods 4

5 Unit testing in C# You need at least Visual Studio Professional; and for code coverage feedback you need at least Premium. Check these tutorials/docs (Visual Studio 2010): – Walkthrough: Creating and Running Unit Tests – Walkthrough: Run Tests and View Code Coverage – Walkthrough: Using the Command-line Test Utility – API Reference for Testing Tools for Visual Studio, in particular Microsoft.VisualStudio.TestTools.UnitTesting, containg classes like Assert, CollectionAssert,... In this lecture we will just go through the concepts 5

6 The structure of a “test project” 6 class Thermometer private double val private double scale private double offset public Thermometer(double s, double o) public double value() public double warmUp(double v) public double coolDown(double v) A test project is a just a project in your solution that contains your test-classes.

7 The structure of a “test project” A solution may contain multiple projects; it may thus contain multiple test projects. A test project is used to group related test classes. You decide what “related” means; e.g. you may want to put all test-cases for the class Thermometer in its own test project. A test class is used to group related test method. A test method does the actual testing work. It may encode a single test-case, or multiple test-cases. You decide. 7

8 Test Class and Test Method 8 [TestClass()] public class ThermometerTest { private TestContext testContextInstance; //[ClassInitialize()] //public static void MyClassInitialize(...)... //[ClassCleanup()] //public static void MyClassCleanup()... //[TestInitialize()] //public void MyTestInitialize()... //[TestCleanup()] //public void MyTestCleanup()... [TestMethod()] public void valueTest1()... [TestMethod()] public void valueTest2().... } public void valueTest1() { target = new Thermometer(1,0); double expected = ; double actual = target.value(); Assert.AreEqual(expected, actual); } Be careful when comparing floating numbers, you may have to take imprecision into account, e.g. use this instead: AreEqual(expected,actual,delta,”...”)

9 Positive and Negative Test Positive test: test the program on its normal parameters’ range. But can we afford to assume that the program is always called in its normal range? Else do Negative test: test that the program beyond its normal range. E.g. when unit testing a method, to test if it throws the right kind of exceptions. 9

10 Test Oracle some patterns  will return later Full: Assert.IsTrue( T == ) Partial: Assert.IsTrue( T >= ) Property-based : Assert.isTrue(safe(T)) 10 public void valueTest1() { target = new Thermometer(1,273.15); double expected = ; double actual = target.value(); Assert.AreEqual(expected, actual); } An oracle specifies your expectation on the program’s responses. Usually partial, but allow re-use the predicate “safe” in other test-cases. More costly to maintain, e.g. if you change the intended behavior of the program.

11 Discussion: propose test cases in particular the oracles reverse(a) { N = a.length if (N  1) return for (int i=0; i< N/2 ; i++) swap(a,i, N-1-i) } incomeTax(i) { if (i  18218) return * i t = 419 if (i  32738) return t * (i – 18218) t += 1568 if (i  54367) return t * (i – 32738) } Property-based testing fits nicely for reverse, but not for incomeTax; for the latter we’ll have to fall back to conrete-value oracles, which unfortunately tend to be more costly to maintain.

12 Discussion: the Oracle Problem (6.5) Every test-case needs an oracle; how to construct it!?  always a big problem! Using concrete values as oracles is often powerful, but potentially expensive to maintain. Using “properties” on the other hand has the problem that often it is hard to write a complete yet simple property capturing correctness. E.g. how to express an oracle for a sorting function? Alternatively we can fall back to redundancy-based testing. 12

13 Inspecting Test Result 13

14 Inspecting Coverage 14

15 Finding the source of an error: use a debugger! 15 Add break points; execution is stopped at every BP. You can proceed to the next BP, or execute one step at a time: step-into, step-over, step-out. VisualStudio uses IntelliTrace  logging  you can even inspect previous BPs.

16 The Debug class Debug.Print(“therm. created”) Debug.Assert(t.scale() > 0) to check for properties that should hold in your program. Will be removed if you build with debug off (for release). Check the doc of System.Diagnostics.Debug 16

17 (Unit) Testing a class Many classes have methods that interact with each other (e.g. as in Stack). How to test these interactions? How to specify and test these interactions? Options: – class invariant – Abstract Data Type (ADT) – Finite State Machine (FSM) 17

18 Specifying with class invariant Regardless the interaction with other methods, each method of a class C has to keep the state of its target object consistent. Express this with a class invariant, e.g. – this.T >= – this.saldo >= 0 – forall x in persons, typeOf(x) is Employee Class invariant cannot express a constraint over the interaction itself. 18

19 Formulate and test the class invariant 19 Stack private Object[] content private int top public Stack() push(T x) T pop() bool classinv() { return 0  top && top

20 Specifying a class as an ADT An Abstract Data Type (ADT) is a model of a (stateful) data structure. The data structure is modeled abstractly by only describing a set of operations (without exposing the actual state). The semantic is described in terms of “logical properties” (also called the ADT’s axioms) over those operations. 20

21 Example : stack 21 Stack bool isEmpty() push(T x) T pop() Stack axioms : For all x,s : s.push(x) ; y = s.pop ; assert (y==x ) For all x and s : s.push(x) ; assert (  s.isEmpty()) For all x and s.isEmpty() : s.push(x) ; s.pop assert (s.isEmpty()) Depending of the offered operations, it may be hard/not possible to get a complete axiomatization.

22 Test each ADT’s axiom 22 For all x,s : s.push(x) ; y = s.pop ; assert (y==x ) For example, three test cases : 1. empty s 2. non-empty s 3. s already contains x

23 Specifying a class with a finite state machine (FSM) (2.5.1, 2.5.2) 23 open() close() write() File: good when the sequence with which the methods of the class is called matters For example, test that after every valid sequence the class invariant hold. relevant concepts to apply: node coverage, edge coverage, edge- pair coverage. FSM can also come from your UML models.

24 In a larger project.... You want to test your class Heater ; it uses Thermometer which is not ready yet! We can opt to use a mock Thermometer. A mock of a program P: – has the same interface as P – only implement a very small subset of P’s behavior – fully under your control Analogously we have the concept of mock object. Make mocks yourself e.g. exploiting inheritance, or use a mocking tool. 24

25 Mocking with Moq 25 test1() { Heater heater = new Heater() var mock = new Mock () mock.Setup(t => t.value()).Returns( ) heater.thermometer = mock.object heater.limit = heater.check() Assert.IsFalse( } interface IThermometer double value() double warmUp(double v) class Heater double limit bool active public check() { if (thermometer.value() >= limit) active = false } thermometer

26 Mocking with Moq (google it for more info!) 26 var mock = new Mock () mock.Setup(t => t.value()).Returns( ) mock.Setup(t => t.warmUp(0)).Returns(0) mock.Setup(t => t.warmUp(It.IsInRange (-10, 10, Range.Inclusive)).Returns(0) mock.Setup(t => t.warmUp (It.IsAny ())).Returns((double s) => s ) Many more mock-functionalities in Moq; but in general mocking can be tedious. E.g. what to do when your Heater wants to call warmUp in an iteration?

27 Beyond unit testing, what if we don’t have access to the source code ? Or you deliberately want to abstract away Or, you do have access, but you don’t have a tool to instrument the code (Def 1.26) White box testing : common at the unit-testing level (Def 1.25) Black box testing: common at the system-testing level. Approaches : – Model-based testing – Partition-based testing 27

28 Model-based testing We already seen that  use an FSM as a model of the system you test Such an FSM specifies : – sequences which should be valid and invalid – these tell you which sequences to test – furthermore, coverage can be defined over the FSM 28 new(fname) save() type(c) TextEditor:

29 Model-based testing Predicates can be defined on the FSM’s states to specify properties that should hold; but keep in mind that in black box testing you cannot just inspect objects state at will. Useful concepts : mutators and inspectors. Mutators : methods with side effect  arrows in the FSM. Inspectors : methods with no side effect  used to express your state predicates, 29 size() > 0 new(...) save() type(c) TextEditor: type(c) size() = 0

30 Partitioning the inputs Based on “your best understanding” of save’s semantic. Terminology: characteristic, block. The domain of a characteristic is divided into disjoint blocks; the union of these blocks must cover the entire domain of the characteristic. Assumption : values of the same block are “equivalent” 30 save(String fname, Object o) fname : (A) existing file (B) non-existing file o : (P) null (Q) non-null serializable (R) non-null non-serializable

31 So, what input values to choose? (C4.23, ALL) All combinations must be tested. |T| = (  i: 0  i

32 t-wise coverage (C4.25, pair-wise coverage). Each pair of blocks (from different characteristics) must be tested. (C4.26, t-wise coverage). Generalization of pair-wise. Obviously stronger than EACH CHOICE, and still scalable. Problem: we just blindly combine; no semantical awareness. 32 ABAB PQPQ XYXY T : (A,P,X), (A,Q,Y),... more?

33 Adding a bit of semantic (C4.27, Base Choice Coverage, BCC) Decide a single base test t 0. Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics). |T| = 1 + (  i : 0  i

34 Or, more bits of semantics (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks. |T| at most M + (  i : 0  i

35 Example-2, MBCC (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remaining non-base blocks. 35 ABCABC PQRPQR XYZXYZ Bold : base blocks Chosen base tests = (A,P,X), (A,Q,Y) These produce these additional test requirements: (B,P,X) (C,P,X) (B,Q,Y) (C,Q,Y) (A,R,X) (A,R,Y) (A,P,Z) (A,Q,Z) base-blocks are not cross-combined except as in the base tests. non-base blocks are not cross- combined with each other.

36 Constraints, to exclude non-sensical cases Example: – combo (A,P,Y) is not allowed. – if P is selected, then X must also be selected. Solvable: pair-wise coverage + (A,P,Y) is not allowed. Can be unsolvable, e.g. pair-wise coverage + (A,P) is not allowed. General problem: given a coverage criterion C and a set of constraints, find a test set T satisfying both. In general the problem is not trivial to solve. 36

37 Overview of partition-based coverage 37 EACH CHOICE ALL t-Wise Multiple Base Choice Coverage Pair-Wise Base Choice Coverage

38 Regression Test To test that a new modification in your program does not break old functionalities. To be efficient, people typically reuse existing test sets. Usually applied for system-testing, where the problem is considered as more urgent. Challenge: very time consuming (hours/days!). There are also research to apply it continuously at the unit level, as you edit your program; see it as extended type-checking. Result was positive! 38 (Saff & Enrst, An experimental evaluation of continuous testing during development. ISSTA 2004)

39 Some concepts first... Test Selection Problem: suppose P has been modified to P’. Let T be the test set used on P. Choose a subset T’  T to test P’. Obviously, exclude obsolete test cases: those that can’t execute or whose oracles no longer reflect P’ semantic. Let’s assume: we can identify them. You want the selection to be safe : T’ includes all test-cases in T that will execute differently on P’. Only attractive if the cost of calculating T’ + executing T’ is less than simply re-executing the whole T. 39

40 Idea: select those that pass through modified code If m is the only method in P that changes, the obvious strategy is to select only test-cases that pass through m. Better: only select test-cases that pass m’s “modified” branch. 40 m(x) { if (d) y = x+1 else y=0 } m(x) { if (d) y = x-1 else y=0 } (orginal) (modified)

41 Corner case The first if is modified by removing an else-branch. Using the previous strategy means that we have to select all test-cases that pass m. Yet we see that the paths [  d, e, stmt] and [  d,  e] present in both old and new m; so there is actually no need to select them. 41 m(x) { if (d) y = x+1 ; if (e) stmt } m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } else if (e) stmt }

42 Looking at it abstractly with CFG 42 m(x) { if (d) y = x+1 ; if (e) stmt } m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } else if (e) stmt } d y=x+1 e stmt u=0 end e stmt d y=x+1 e stmt end Notice that [  d, e, stmt, end] and [  d,  e, end] appear in both, and equivalent.

43 Some concepts We assume: P is deterministic.  each test-case always generate the same test path. Let p and p’ be the test-paths of a test-case t when executed on P and P’; t is modification traversing if not(p  p’).  let’s select modification traversing test-cases. p  p’ if they have the same length, and for each i, p i  p’ i  the latter means they contain the same sequence of instructions. So far this is not helpful, because such a selection strategy requires us to first execute t on P’. Then it is not attractive anymore! 43

44 “Intersection” Graph First, extend CFG so that branches are labelled by the corresponding decision value (e.g. T/F for if-branch). Label non-branch edge with some constant value. Each node of G’’ is a pair (u,u’). Then G’’ is defined like this : – The pair of initial nodes (s0,s0’)  G’’. – If (u,u’)  G’’, and u  u’, and u  v is an edge in G, and u’  v’ and edge in G’ both with the same label, then (u,u’)  (v,v’) should be an edge in G’’. 44 a c end b G : a c end b G’ : c d G’’ = G  G’: a,a c,c end,end b,b c,c end,d

45 “Intersection” Graph Each path p in G’’ describes how a path in G would be executed on G’ if the same decisions are taken along the way. Note that this is calculated without re-executing any test-case on P’. Any path in G’’ ends either in a proper exit node (green), or in a pair (u,u’) where not u  v (red). This would be the first time a test-case would hit a modified code when re-executed on P’. The old test-cases are assumed to have been instrumented, so that we know which nodes/edges in G it traversed. 45 a c end b G : a c end b G’ : c d G’’ = G  G’: a,a c,c end,end b,b c,c end,d

46 Selection Algorithm Select test-cases that pass [a,b,c,end] in G  expensive to check! (Safe but not minimalistic) Select test-cases that pass a node u in G that is part of a red-node in G’’.  same problem as before, it will select also select [a,c,end] which is not modification traversal. (Rothermel-Harold, 1997) Select test-cases that pass an edge e in G that in G’’ leads to a red-node in G’’.  actually the same problem. 46 a c end b G : a c end b G’ : c d G’’ = G  G’: a,a c,c end,end b,b c,c end,d

47 Selection Algorithm (Ball algorithm,1998) Partition G’’ nodes to those can can reach a green- node (G partition), and those that cannot (NG partition). Look at edges in G’’ that cross these partitions (so, from G to NG). A test path p is modification traversing if and only if it passes through a crossing edge (as meant above).  use this as the selection criterion. 47 a c end b G : a c end b G’ : c d G’’ = G  G’: a,a c,c end,end b,b c,c end,d NG-partition G-partition

Download ppt "Some Testing Techniques (enhanced 6 + 4) Course Software Testing & Verification 2013/14 Wishnu Prasetya."

Similar presentations

Ads by Google