1. Workshop on Integrating Software Testing into Programming Courses (WISTPC14:2) Friday July 18, 2014 Introduction to Software Testing

2. What is software testing? Software testing is the process of operating software under specified conditions, observing or recording the results and making an evaluation of some aspect of the software. (IEEE/ANSI std 610.12-1990) 4

3. 3 Overview of Testing - Terminology Software testing is the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite execution domain, against the expected behavior. (Guide to the Software Engineering Body of Knowledge 2004 Version)

4. How much testing is enough? Testing can show the presence of errors but not their absence. – Edsger Dijkstra 4

5. Errors Discovered over Time Number of errors 5 Time

6. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 6 Important Terms Validation & Verification Validation: – The process of evaluating software at the end of software development to ensure compliance with intended usage Verification: – The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase IV&V stands for “independent verification and validation ”

7. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 7 Static and Dynamic Testing Static Testing: – Testing without executing the program. This include software inspections and some forms of analyses. Dynamic Testing: – Testing by executing the program with real inputs

8. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 8 Software Faults, Errors & Failures Software Fault: – A static defect in the software Software Error: – An incorrect internal state that is the manifestation of some fault Software Failure: – External, incorrect behavior with respect to the requirements or other description of the expected behavior

9. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 9 Testing & Debugging Testing: – Finding inputs that cause the software to fail Debugging: – The process of finding a fault given a failure

10. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 10 Fault & Failure Model Three conditions necessary for a failure to be observed 1.Reachability : The location or locations in the program that contain the fault must be reached 2.Infection : The state of the program must be incorrect 3.Propagation : The infected state must propagate to cause some output of the program to be incorrect

11. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 11 Test Case Test Case Values: – The values that directly satisfy one test requirement Expected Results: – The result that will be produced when executing the test if the program satisfies it intended behavior

12. Testing Concepts Test case components: 1.Name – identifies the test case, it is a good idea to derive the name from the requirement being tested. 2.Purpose – states the purpose of the test and relates it to the requirement (or scenario). 3.Test set up – describe the h/w and s/w and environment required for a successful test. 4.Input – description of the input data or commands. 5.Expected output (or Oracle) – expected test results against which the output of the test is compared.

13. What is a successful test case? One that produces expected results? OR One that produces a failure? 13

14. Errors Discovered over Time Number of errors 14 Time

15. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 15 Observability and Controllability Software Observability: – How easy it is to observe the behavior of a program in terms of its outputs, effects on the environment and other hardware and software components Software that affects hardware devices, databases, or remote files have low observability

16. Software Controllability: – How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors Easy to control software with inputs from keyboards Inputs from hardware sensors or distributed software is harder Data abstraction reduces controllability and observability 16

17. Levels of Testing Unit Testing – Refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors. (wikipedia, 2010) Integration Testing – Is any type of software testing that seeks to verify the interfaces between components against a software design. Components may be integrated in an iterative way or all together ("big bang"). (wikipedia, 2010)

18. Levels of Testing System Testing – Testing a completely integrated system to verify that it meets its requirements. (wikipedia, 2010) See http://en.wikipedia.org/wiki/Software_testinghttp://en.wikipedia.org/wiki/Software_testing 18

19. Unit Testing Focuses on the building blocks of the software system i.e., objects and subsystems. Many unit testing techniques have been devised including: equivalence testing, state-based testing, boundary testing, domain testing, control flow- based testing (statement, branch). 19

20. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 20 White-box and Black-box Testing Black-box testing: – Deriving tests from external descriptions of the software, including specifications, requirements, and design White-box testing: – Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements

21. Criteria Based on Structures 1.Graphs 2.Logical expressions 3.Input domain characteristics 4.Syntactic structures – Ammann & Offutt, Introduction to Software Testing 21

22. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 22 1.Graphs 2.Logical Expressions 3.Input Domain Characterization 4.Syntactic Structures (not X or not Y) and A and B if (x > y) z = x - y; else z = 2 * x; A: {0, 1, >1} B: {600, 700, 800} C: {swe, cs, isa, infs} Structures : Four ways to model software

23. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt23 1. Graph Coverage – Structural 6 5 3 2 17 4 Node (Statement) Cover every node 12567 1343567 This graph may represent statements & branches methods & calls components & signals states and transitions Edge (Branch) Cover every edge 12567 1343567 1357 Path Cover every path 12567 1257 13567 1357 1343567 134357 …

24. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt24 Defs & Uses Pairs (x, 1, (1,2)), (x, 1, (1,3)) (y, 1, 4), (y, 1, 6) (a, 2, (5,6)), (a, 2, (5,7)), (a, 3, (5,6)), (a, 3, (5,7)), (m, 4, 7), (m, 6, 7) 1. Graph Coverage – Data Flow 6 5 3 2 17 4 This graph contains: defs: nodes & edges where variables get values uses: nodes & edges where values are accessed def = {x, y} def = {a, m} def = {a} def = {m} use = {x} use = {a} use = {y} use = {m} use = {y} All Defs Every def used once 1, 2, 5, 6, 7 1, 3, 4, 3, 5, 7 All Uses Every def “reaches” every use 1, 2, 5, 6, 7 1, 2, 5, 7 1, 3, 5, 6, 7 1, 3, 5, 7 1, 3, 4, 3, 5,7

25. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt25 1. Graph - FSM Example Memory Seats in a Lexus ES 300 Driver 1 Configuration Driver 2 Configuration [Ignition = off] | Button2 [Ignition = off] | Button1 Modified Configuration sideMirrors ()[Ignition = on] | lumbar ()[Ignition = on] | seatBottom ()[Ignition = on] | seatBack () [Ignition = on] | New Configuration Driver 1 New Configuration Driver 2 [Ignition = on] | Reset AND Button1 [Ignition = on] | Reset AND Button2 Ignition = off (to Modified) Guard (safety constraint) Trigger (input)

26. 26 Control Flow Adequacy Criteria cont Example: Source code public int Fun(int x){ k = 0; while (x <= 10 && k < 3){ if (x%2 != 0) k = k + 1; x = x + 1; } if (x < 0){ x = 10; k = 0; } return k; } K=0 x<=10 && k<3 x%2 != 0 x=x+1 k=k+1 return k B C D E F I Entry Flow graph A X<0 X = 10 K = 0 T F T T F F G H

27. 27 Unit Testing – Statement Coverage 1.Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p. Whitebox testing technique. Generate test data to execute every stmt in the program at least once. Exercise: Indentify value(s) of x to execute every stmt in Fun(int x) at least once.

28. 28 Unit Testing – Branch Coverage 2.Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e. Whitebox testing technique. Generate test data to exercise the true and false outcomes of every decision. Exercise: Indentify value(s) of x to execute every branch in Fun(int x) at least once.

29. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 29 2. Logical Expressions ( (a > b) or G ) and (x < y) Transitions Software Specifications Program Decision Statements Logical Expressions

30. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 30 2. Logical Expressions Predicate Coverage : Each predicate must be true and false –( (a>b) or G ) and (x < y) = True, False Clause Coverage : Each clause must be true and false – (a > b) = True, False – G = True, False – (x < y) = True, False Combinatorial Coverage : Various combinations of clauses – Active Clause Coverage: Each clause must determine the predicate’s result ( (a > b) or G ) and (x < y)

31. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 31 2. Logic – Active Clause Coverage ( (a > b) or G ) and (x < y) 1 T F T 2 F F T duplicate 3 F T T 4 F F T 5 T T T 6 T T F With these values for G and (x b) determines the value of the predicate

32. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 32 3. Input Domain Characterization Describe the input domain of the software – Identify inputs, parameters, or other categorization – Partition each input into finite sets of representative values – Choose combinations of values System level – Number of students { 0, 1, >1 } – Level of course { 600, 700, 800 } – Major { swe, cs, isa, infs } Unit level – Parameters F (int X, int Y) – Possible values X: { 2 }, Y : { 10, 20, 30 } – Tests F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)

33. Unit Testing – Boundary Analysis Test cases are generated using the extremes of the input domain, e.g. maximum, minimum, just inside/outside boundaries, typical values, and error values. It is similar to Equivalence Partitioning but focuses on "corner cases“. Exercise: write test case input using boundary analysis for the getNumberDaysInMonth() method. 33

34. Unit Testing – Equivalence Partitioning Equivalence partitioning is a blackbox testing technique that minimizes the number of test cases. Possible inputs are partitioned into equivalence testing classes, and a test case is selected from each class. Assumption - system behaves in a similar way for all members of an equiv. class. Criteria used to determine equivalence classes: coverage, disjointedness, representation. 34

35. Unit Testing – Equivalence Partitioning Equivalence classValue for month inputValue for year input Months with 31 days, non-leap yrs.7 (July)1901 Months with 31 days, leap yrs.7 (July)1904 Months with 30 days, non-leap yrs.6 (June)1901 Months with 30 days, leap yrs.6 (June)1904 Months with 28 or 29 days, non-leap yrs.2 (February)1901 Months with 28 or 29 days, leap yrs.2 (February)1904 35 Example: Valid inputs to test the getNumberDaysInMonth() method

36. Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com © Ammann & Offutt 36 4. Syntactic Structures Based on a grammar, or other syntactic definition Primary example is mutation testing 1.Induce small changes to the program: mutants 2. Find tests that cause the mutant programs to fail: killing mutants 3.Failure is defined as different output from the original program 4. Check the output of useful tests on the original program Example program and mutants if (x > y) z = x - y; else z = 2 * x; if (x > y)  if (x >= y) z = x - y;  z = x + y;  z = x – m; else z = 2 * x;

37. Miami University, Oxford OH 37