1. Introduction to Software Testing
Workshop on Integrating Software Testing into Programming Courses
Friday June 12, 2015

2. What is software testing?
Software testing is the process of operating software under specified conditions, observing or recording the results and making an evaluation of some aspect of the software.
(IEEE/ANSI std )

3. Overview of Testing - Terminology
Software testing is the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite execution domain, against the expected behavior.
(Guide to the Software Engineering Body of Knowledge 2004 Version)

4. Does testing always involve running a program?
Static Testing:
Testing without executing the program.
This include software inspections and some forms of analyses.
Dynamic Testing:
Testing by executing the program with real inputs

5. What is a successful test case?
One that produces expected results? OR
One that produces a failure?

6. What is a test case?
E.g., What is one test case to test a program that classifies triangles?
Test Case Values:
The values that directly satisfy one test requirement
Expected Results:
The result that will be produced when executing the test if the program satisfies it intended behavior

7. Testing Concepts Test case components:
Name – identifies the test case, it is a good idea to derive the name from the requirement being tested.
Purpose – states the purpose of the test and relates it to the requirement (or scenario).
Test set up – describe the h/w and s/w and environment required for a successful test.
Input – description of the input data or commands.
Expected output (or Oracle) – expected test results against which the output of the test is compared.

8. How do we know the right answer? The Oracle Problem
From previous version of the system
Checking by hand
Simulations
Checks for reasonableness

9. How much testing is enough?
Testing can show the presence of errors but not their absence.
Edsger Dijkstra

10. Errors Discovered over Time
Number of errors
Time

11. What is the difference between a fault and a failure?
Software Fault:
A static defect in the software
Software Failure:
External, incorrect behavior with respect to the requirements or other description of the expected behavior

12. What is the difference between testing and debugging?
Finding inputs that cause the software to fail
Debugging:
The process of finding a fault given a failure

13. Important Terms Validation & Verification
The process of evaluating software at the end of software development to ensure compliance with intended usage
Verification:
The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase
IV&V stands for “independent verification and validation”
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

14. How is unit testing different from integration testing?
How is integration testing different from system testing?

15. Levels of Testing Unit Testing Integration Testing
Refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors. (wikipedia, 2010)
Integration Testing
Is any type of software testing that seeks to verify the interfaces between components against a software design. Components may be integrated in an iterative way or all together ("big bang"). (wikipedia, )

16. Levels of Testing System Testing
Testing a completely integrated system to verify that it meets its requirements. (wikipedia, 2010)
See

17. Unit Testing
Focuses on the building blocks of the software system i.e., objects and subsystems.
Many unit testing techniques have been devised including: equivalence testing, state-based testing, boundary testing, domain testing, control flow- based testing (statement, branch).

18. What kinds of errors do you tend to find in unit testing versus integration testing?
Between integration testing and systems testing?

19. Who does acceptance testing?

20. What is the difference between white (or glass) box and black box testing?
Deriving tests from external descriptions of the software, including specifications, requirements, and design
White-box testing:
Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

21. In what way is testing like sampling?
Why is sampling needed?
Is exhaustive testing possible?

22. What makes program more difficult to test?
Complexity
Why might models of programs be helpful to guide testing?
What kinds of models?

23. Criteria Based on Structures and Models
Graphs
Logical expressions
Input domain characteristics
Syntactic structures
Ammann & Offutt, Introduction to Software Testing

24. How can a control flow graph be used to find test cases?
What are criteria that we could use with control flow graphs?
Coverage: statement (node), branch, …

25. Control Flow Adequacy Criteria
Flow graph
Example: Source code
public int Fun(int x){
k = 0;
while (x <= 10 && k < 3){
if (x%2 != 0)
k = k + 1;
x = x + 1; }
if (x < 0){
x = 10;
return k;
Entry A
K=0 B F
x<=10 && k<3 C G T F
X<0 F
x%2 != 0 D T T
X = 10
K = 0
k=k+1 E H
x=x+1 F
return k I

26. 1. Graph Coverage – Structural6 5 3 2 1 7 4
Path
Cover every path
12567
1257
13567
1357 …
Edge (Branch)
Cover every edge
12567
1357
Node (Statement)
Cover every node
12567
This graph may represent
statements & branches
methods & calls
components & signals
states and transitions
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

27. How can a data flow graph be used to find test cases?

28. 1. Graph Coverage – Data Flow
This graph contains:
defs: nodes & edges where variables get values
uses: nodes & edges where values are accessed
def = {x, y}
def = {a , m}
def = {a}
def = {m} 6 5 3 2 1 7 4
use = {x}
use = {a}
use = {y}
use = {m}
All Defs
Every def used once
1, 2, 5, 6, 7
1, 3, 4, 3, 5, 7
Defs & Uses Pairs
(x, 1, (1,2)), (x, 1, (1,3))
(y, 1, 4), (y, 1, 6)
(a, 2, (5,6)), (a, 2, (5,7)), (a, 3, (5,6)), (a, 3, (5,7)),
(m, 4, 7), (m, 6, 7)
All Uses
Every def “reaches” every use
1, 2, 5, 6, 7
1, 2, 5, 7
1, 3, 5, 6, 7
1, 3, 5, 7
1, 3, 4, 3, 5,7
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

29. 1. Graph - FSM Example Memory Seats in a Lexus ES 300
Guard (safety constraint)
Trigger (input)
Driver 1
Configuration
Driver 2
[Ignition = off] | Button2
[Ignition = off] | Button1
sideMirrors ()
[Ignition = on] |
(to Modified)
lumbar ()
[Ignition = on] |
Modified
Configuration
seatBottom ()
[Ignition = on] |
seatBack ()
[Ignition = on] |
Ignition = off
New
Configuration
Driver 1
Driver 2
[Ignition = on] | Reset AND Button1
[Ignition = on] | Reset AND Button2
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

30. Unit Testing – Statement Coverage
Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p.
Whitebox testing technique.
Generate test data to execute every stmt in the program at least once.
Exercise: Indentify value(s) of x to execute every stmt in Fun(int x) at least once.

31. Unit Testing – Branch Coverage
Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e. Whitebox testing technique.
Generate test data to exercise the true and false outcomes of every decision.
Exercise: Indentify value(s) of x to execute every branch in Fun(int x) at least once.

32. How do predicates add complexity to programs and thus to testing?

33. ( (a > b) or G ) and (x < y)
2. Logical Expressions
( (a > b) or G ) and (x < y)
Transitions
Logical
Expressions
Program Decision Statements
Software Specifications
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

34. ( (a > b) or G ) and (x < y)
2. Logical Expressions
( (a > b) or G ) and (x < y)
Predicate Coverage : Each predicate must be true and false
( (a>b) or G ) and (x < y) = True, False
Clause Coverage : Each clause must be true and false
(a > b) = True, False
G = True, False
(x < y) = True, False
Combinatorial Coverage : Various combinations of clauses
Active Clause Coverage: Each clause must determine the predicate’s result
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

35. Logic – Active Clause Coverage
( (a > b) or G ) and (x < y)
1 T F T
2 F F T
With these values for G and (x<y), (a>b) determines the value of the predicate
duplicate
3 F T T
4 F F T
5 T T T
6 T T F
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

36. How can knowledge of the input space help in sampling for testing?

37. Input Domain Characterization
Describe the input domain of the software
Identify inputs, parameters, or other categorization
Partition each input into finite sets of representative values
Choose combinations of values
System level
Number of students { 0, 1, >1 }
Level of course { 600, 700, 800 }
Major { swe, cs, isa, infs }
Unit level
Parameters F (int X, int Y)
Possible values X: { <0, 0, 1, 2, >2 }, Y : { 10, 20, 30 }
Tests
F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

38. Unit Testing – Boundary Analysis
Test cases are generated using the extremes of the input domain, e.g. maximum, minimum, just inside/outside boundaries, typical values, and error values.
It is similar to Equivalence Partitioning but focuses on "corner cases“.
Exercise: write test case input using boundary analysis for the getNumberDaysInMonth() method.

39. Unit Testing – Equivalence Partitioning
Equivalence partitioning is a blackbox testing technique that minimizes the number of test cases.
Possible inputs are partitioned into equivalence testing classes, and a test case is selected from each class.
Assumption - system behaves in a similar way for all members of an equiv. class.
Criteria used to determine equivalence classes: coverage, disjointedness, representation.

40. Unit Testing – Equivalence Partitioning
Example: Valid inputs to test the getNumberDaysInMonth() method
Equivalence class
Value for month input
Value for year input
Months with 31 days, non-leap yrs.
7 (July)
1901
Months with 31 days, leap yrs.
1904
Months with 30 days, non-leap yrs.
6 (June)
Months with 30 days, leap yrs.
Months with 28 or 29 days, non-leap yrs.
2 (February)
Months with 28 or 29 days, leap yrs.

41. 4. Syntactic Structures
Based on a grammar, or other syntactic definition
Primary example is mutation testing
Induce small changes to the program: mutants
Find tests that cause the mutant programs to fail: killing mutants
Failure is defined as different output from the original program
Check the output of useful tests on the original program
Example program and mutants
if (x > y)
if (x >= y)
z = x - y;
 z = x + y;
 z = x – m;
else
z = 2 * x;
if (x > y)
z = x - y;
else
z = 2 * x;
Introduction to Software Testing (Ch 1),
© Ammann & Offutt

42. Miami University, Oxford OH