1. Introduction to Software Engineering
Chapter 6
Software Testing
Software Engineering Consortium Taiwan
(SECT)
March 2005
© by SEC

2. Some General Concepts about Software Testing
The objective is: Ensure Quality
Detect bug {error, failure, inconsistent with specs}
Hint the cause of bugs
Test Plan  Test Cases  Testing Report  Bug tracking
Automation Tools

3. A Typical Test Case 案例 (操作/輸入) 應產生行為(輸出) 實際產生結果 通過測試與否
2.3於使用者登入密碼欄內輸入正確密碼後按Enter。
程式必須告知使用者登入成功。
程式告知使用者輸入登入成功。
　成功 
2.4於使用者登入密碼欄內輸入錯誤密碼後按Enter。
程式必須告知使用者輸入密碼錯誤。
程式告知使用者輸入密碼錯誤。
2.5 於使用者登入密碼欄內不輸入密碼後按Enter。
程式必須告知使用者未輸入密碼。 失敗
未顯示錯誤訊息，直接跳回輸入畫面

4. Software Testing 6.1 Software Testing Fundamentals
Chapter 6
Software Testing
6.1 Software Testing Fundamentals
6.2 Testing Principles
6.3 Testing Techniques
6.3.1 White-Box Testing Techniques
6.3.2 Black-Box Testing Techniques
6.4 Software Testing Strategies
6.4.1 Verification and Validation
6.4.2 The V Model
6.4.3 Unit Testing
6.4.4 Integration Testing
6.4.5 Regression Testing
6.4.6 Validation Testing
6.4.7 Alpha Testing
6.4.8 Beta Testing
6.4.9 System Testing
315
© by SEC

5. 6.1 Software Testing Fundamentals
What is software testing?
– Myers: The process of executing a program with the intent
of finding errors.
– Beizer: The act of executing tests. Tests are designed and
then executed to demonstrate the correspondence between
an element and its specification.
– IEEE: The process of exercising or evaluating a system or
system component by manual or automated means to verify
that it satisfies specified requirements or to identify
differences between expected and actual results.
316
© by SEC

6. 6.2 Testing Principles Davis in his book, 201 Principles of Software
Development, suggests a set of testing principles:
– All tests should be traceable to customer requirements.
– Tests should be planned long before testing begins.
– The Pareto principle applies to software testing.
80% of all errors uncovered during testing will likely be traceable
to 20% of all program modules.
318
© by SEC

7. 6.2 Testing Principles (cont’d)
– Testing should begin “in the small” and progress toward
testing “in the large”.
– Exhaustive testing is not possible.
– To be most effective, testing should be conducted by an
independent third party.
319
© by SEC

8. Attributes of A Good Test
Kaner, Falk, and Nguyen in their book, Testing
Computer Software, suggest the following attributes
of a “good” test
– A good test has a high probability of finding an error
– A good test is not redundant
– A good test should be “best of breed”
– A good test should be neither too simple nor too complex
320
© by SEC

9. Wason作業
要檢驗「假如一張卡片一面是母音字母，那它的另一面就是偶數」，需要翻哪些卡片才能確定？(Wason & Johnson, 1972）
要檢驗「假如一個人在喝酒，那他一定滿18歲」，需要翻哪幾張卡片才能確定？ E F 4 7 喝酒
喝可樂
22歲
16歲 9

10. Who Tests the Software? Development Engineers
– Understand the system, but test “gently”
– Driven by “delivery”
– Only perform unit tests and integration tests
Software Test Engineers and Testers
– Need to learn the system, but attempt to break it
– Driven by “quality”
– Define test cases, write test specifications, run tests
321
© by SEC

11. Another example
A function takes two numbers and return the average of the two numbers
How many tests would be needed to make sure that the function has no bug?
(10, 20)15, (12,21)16.5, …

12. Exhaustive Testing
Is it possible to develop test cases to exercise all
logical paths of a program?
– Even small programs, the number of possible logical paths
can be very large
– Example: Consider a program contains 2 nested loops that
each executes 1 to 20 times depending on the input data.
The interior loop has 4 if-then-else constructs
There are 1014 possible paths! If we exercise one test per
millisecond, it would take 3170 years to complete test
Exhaustive testing is impractical for large software
system
– Selective testing is required
322
© by SEC

13. When Testing Can Stop? When we run out of budget or time…
When the test adequate criteria is met
Testing is never done, the burden simply shifts from
you to the customer
Only when the product has been irrevocably retired
323
© by SEC

14. Question
A program has two versions. The engineering team detected and fixed 100 bugs in n version A, while 25 bugs were detected and fixed in version B.
After the fix, which program would you take as the final version?

15. 6.3 Testing Techniques Two general software testing approaches:
– White-box testing:
Design tests to exercise internal structures of the software to make
sure they operates according to specifications and designs
– Black-box testing:
Design tests to exercise each function of the software and check its
errors.
– White-box and black-box testing approaches can uncover
different class of errors and are complement each other
324
© by SEC

16. Exercise
A program reads three integer values as the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene(不等邊), isosceles(等腰), or equilateral(等邊).
Write a set of test cases that you feel would adequately test this program.
Will it make any difference whether you know the algorithm of the program or not?

17. 6.3.1 White-Box Testing White-box testing
– Also known as glass-box testing or structural testing
– Has the knowledge of the program’s structures
– A test case design method that uses the control structure of
the procedural design to derive test cases
– Focus on the control structures, logical paths, logical
conditions, data flows, internal data structures, and loops.
– W. Hetzel describes white-box testing as “testing in the
small”
325
© by SEC

18. Basis Path Testing Basic path testing (a white-box testing technique):
– First proposed by Tom McCabe.
– Can be used to derive a logical complexity measure for a
procedure design.
– Used as a guide for defining a basis set of execution path.
– Guarantee to execute every statement in the program at
least one time.
327
© by SEC

19. How to test this program?
Read Weight (w)
Read Height (h)
IF w > 400 THEN
Print “invalid weight”
ENDIF
If h > 3 THEN
Print “invalid height”
BMI=w/(h*h)
Print ( “BMI = “ BMI)
IF BMI < 20 THEN
Print “underweight”
ELSE IF BMI >= 20 AND BMI <=25 THEN
Print “ideal weight”
ELSE IF BMI > 25 THEN
Print “overweight”
ENDIF

20. Test Coverage

21. The basic structured-constructs in a flow graph :
Basis Path Testing (cont’d)
The basic structured-constructs in a flow graph :
Sequence
If Else
While . . .
Until
Case
328
© by SEC

22. Basis Path Testing (cont’d)
Flow graph notation (control flow graph)
– Node represents one or more procedural statements.
A sequence of process boxes and a decision diamond can map into
a single node
A predicate node is a node with two or more edges emanating
from it
– Edge (or link) represents flow of control
– Region: areas bounded by edges and nodes
When counting regions, include the area outside the graph as a
region
329
© by SEC

23. Occurs when one or more Boolean operators (OR, AND, NAND,
Basis Path Testing (cont’d)
– Compound condition
Occurs when one or more Boolean operators (OR, AND, NAND,
NOR) is present in a conditional statement
A separate node is created for each of the conditions C1 and C2 in
the statement IF C1 AND C2
predicate nodes
if (c1 AND c2) then
print T;
else
print F;
end if; c1 F c2 F
regions T
330
© by SEC

24. binarySearch() Example
public int binarySearch(int sortedArray[ ], int searchValue) {
int bottom = 0;
int top = sortedArray.length() - 1;
int middle, locationOfsearchValue;
boolean found = FALSE;
locationOfsearchValue = -1; /* the location of searchValue in the sortedArray */
/* location = -1 means that searchValue is not found */
while ( bottom <= top && !found)
middle = (top + bottom)/2;
if (searchValue == sortedArray[ middle ])
found = TRUE;
locationOfsearchValue = middle; }
6 else if (searchValue < sortedArray[ middle ])
top = middle - 1;
else
bottom = middle + 1;
} // end while 9
10 return locationOfsearchValue; 1 { 3 5
331
© by SEC

25. The Control Flow Graph (CFG) of Function binarySearch() 12 3 4
Please count the number of:
Nodes
Arcs
Predictable Nodes
Regions 6 5 8 7 9 10
332
© by SEC

26. Cyclomatic Complexity
Cyclomatic complexity is a software metric
– provides a quantitative measure of the global complexity of
a program.
– When this metric is used in the context of the basis path
testing
the value of cyclomatic complexity defines the number of
independent paths in the basis set of a program
the value of cyclomatic complexity defines an upper bound of
number of tests (i.e., paths) that must be designed and exercised to
guarantee coverage of all program statements
333
© by SEC

27. – An independent path is any path of the program that
Cyclomatic Complexity (cont’d)
Independent path
– An independent path is any path of the program that
introduce at least one new set of procedural statements or a
new condition
– In a flow graph, an independent path must move along at
least one edge that has not been traversed before the path is
defined
Examples: consider the CFG of binarySearch()
– Path 1:
– Path 2:
independent paths
– Path 3:
– Path 4: (not an independent path)
© by SEC
334

28. Cyclomatic Complexity (cont’d)
Three ways to compute cyclomatic complexity:
– The number of regions (R) of the flow graph correspond to the
cyclomatic complexity. V(G)= R
– Cyclomatic complexity, V(G), for a flow graph G is
defined as V(G) = E - N + 2
where E is the number of flow graph edges and N is the
number of flow graph nodes.
– Cyclomatic complexity, V(G) = P + 1
where P is the number of predicate nodes contained in the
flow graph G.
335
© by SEC

29. Cyclomatic Complexity of Function binarySearch() predicate nodes2 3 1 R3 4 R2 R1
regions 6
R4 7 5 R5 8 9 10
336
© by SEC

30. Deriving Basis Test Cases
The following steps can be applied to derive the
basis set:
1. Using the design or code as a foundation, draw the
corresponding flow graph.
2. Determine the cyclomatic complexity of the flow graph.
V(G) = 5 regions
V(G) = 13 edges – 10 nodes + 2 = 5
V(G) = 4 predicate nodes + 1 = 5
337
© by SEC

31. Deriving Basis Test Cases (cont’d)
3. Determine a basis set of linearly independent paths.
Path 1:
Path 2:
Path 3: …
Path 4: …
Path 5: …
4. Prepare test cases that force the execution of each path in
the basis set
Path 1 test case:
– Inputs: sortedArray = { }, searchValue = 2
– Expected results: locationOfSearchValue = -1
338
© by SEC

32. Deriving Basis Test Cases (cont’d)
Path 2 test case: cannot be tested stand-alone!
– Inputs: sortedArray = {2, 4, 6}, searchValue = 8
– Expected results: locationOfSearchValue = -1
Path 3 test case:
– Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 6
– Expected results: locationOfSearchValue = 2
Path 4 test case:
– Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 4
– Expected results: locationOfSearchValue = 1
Path 5 test case:
– Inputs: sortedArray = {2, 4, 6, 8, 10}, searchValue = 10
– Expected results: locationOfSearchValue = 4
339
© by SEC

33. Deriving Basis Test Cases (cont’d)
Each test cases is executed and compared to its
expected results.
Once all test cases have been exercised, we can be
sure that all statements are executed at least once
Note: some independent paths cannot be individually
tested because the input data are determined after
traversing the paths.
– In binarySearch(), the initial value of variable found is
FALSE, hence path 2 can only be tested as part of path 3, 4,
and 5 tests
340
© by SEC

34. 練習 請畫出本程式的flow graph 計算V(G) 列出所有的basis path 針對每個basis path設計測試資料
public double calculate(int amount) {
double rushCharge = 0;
if (nextday.equals("yes") ) {
rushCharge = 14.50; }
double tax = amount * .0725;
if (amount >= 1000) {
shipCharge = amount * rushCharge;
} else if (amount >= 200) {
shipCharge = amount * rushCharge;
} else if (amount >= 100) {
shipCharge = rushCharge;
} else if (amount >= 50) {
shipCharge = rushCharge;
} else if (amount >= 25) {
shipCharge = rushCharge;
} else {
shipCharge = rushCharge;
total = amount + tax + shipcharge;
return total;
} //end calculate
請畫出本程式的flow graph
計算V(G)
列出所有的basis path
針對每個basis path設計測試資料

35. path nextday amount expected result
V(G) = E - N + 2         =         =  7
Path 1:  Path 2:  Path 3:  Path 4:  Path 5:  Path 6:  Path 7: 
path     nextday   amount     expected result
1            yes           10                       no           1500        ????.?? 3            no            300                      no            150                      no            75                        no            30                        no            10           

36. Graph Matrices A graph matrix
– A tabular representation of a flow graph
– A square matrix with a size equal to the number of nodes
on the flow graph
– Matrix entries correspond to the edges between nodes
– Adding link weight to each edge to represent
The connection between nodes
The probability of the edge to be executed
The resource (e.g., processing time or memory) required for
traversing the edge
341
© by SEC

37. Graph Matrices (cont’d)
A connection matrix
– A graph matrix with the link weight is 1 (representing a
connection exists) or 0 (representing a connection does not
exist)
– Each row of the matrix with two or more entries represents
a predicate node
– Provide another method for computing the cyclomatic
complexity of a flow graph
342
© by SEC

38. Graph Matrices (cont’d)
Connected to node 1 a
Node 4 5 1 2 b d a b c 3 4 5 c
f g
Graph Matrix 1 4 3 2 e 5 2 3 f g e d 5
Connections 4 1 1
1-1=0 2 1 1
2-1=1 3 4 5 1 1
Cyclomatic
complexity
2-1=1
Flow Graph 1
Connection Matrix
3+1=4
343
© by SEC

39. Summery of Basis Path Testing
Derive the Control Flow Graph & calculate V(G): number of basis paths
Identify the basis paths
Design test cases for each basis path
Conduct the tests according to the test cases

40. Condition Testing Condition Testing Logical conditions
– Test case design method that exercise the logical conditions
in a program module
Logical conditions
– Simple condition:
(a rel-op b) where rel-op={<, ≤, =, ≠, ≥, >} (may be negated with
NOT), e.g., a≤b; NOT(a≤b)
– Compound condition:
two or more simple conditions connected with AND, OR, e.g.,
(a>b) AND (c<d)
344
© by SEC

41. Condition Testing (cont’d)
– Relational expression:
(E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g.,
((a*b+c)>(a+b+c))
– Boolean expression
A condition without relational expressions
If a condition is incorrect, then at least one
component of the condition is incorrect.
345
© by SEC

42. Condition Testing (cont’d)
The type of errors in a condition include :
– Boolean operator (incorrect/missing/extra Boolean operators)
– Boolean variable error
– Boolean parenthesis error
– Relational operator error
– Arithmetic expression error
The purpose of condition testing is to detect not only errors in
the conditions of a program but also other errors in the
program.
– Detect faults in the condition, statements executed before the condition,
and the statements executed after the condition
346
© by SEC

43. – The simplest condition testing strategy
Condition Testing (cont’d)
Branch testing
– The simplest condition testing strategy
– For a compound condition C, test (1) true and false
branches of C and (2) every simple condition of C at least
once
– Example: for the statement: C = (a>b) AND (c<d) we test for
TRUE and FALSE C
a>b
c<d
TRUE and FALSE
347
© by SEC

44. Condition Testing (cont’d)
Domain testing
– For a relational expression, require 3 or 4 tests to be derived
– For an expression E1 rel-op E2, test for E1 greater than,
equal to, or less than E2
– Guarantees detection of rel-op error if E1 and E2 are correct
– To detect errors in E1 and E2, the difference between E1
and E2 for the tests E1 greater or less than E2 should be as
small as possible
348
© by SEC

45. examples if (A or B) then print C If (A > B) then print C
Test case: input A:true, B: true, output:?
Test case: input A:true, B: false, output:?
Test case: input A:false, B: true, output:?
Test case: input A:false, B: false, output:?
If (A > B) then print C
Test case: input A:10, B: 9, output:?
Test case: input A:11, B: 12, output:?
Test case: input A: 13, B: 13, output:?

46. Condition Tests Function getTaxRate (amount, prdType) {
If ( amount < 1000) { Return 5% }
if (amount > 1000 AND (prdType == FOOD OR prdType == DRINKS) ) {
Return 10%
} else {
Return 15% }

47. Data Flow Testing Data flow testing
– A testing technique that selects test paths of a program
according to the locations of definitions and uses of
variables in the program.
– DEF(S) = {X| statement S contains a definition of X}
– USE(S) = {X| statement S contains a use of X}
– The definition of variable X at statement S is said to be live
at statement S’ if there exists a path from statement S to
statement S’ that contains no other definition of X
352
© by SEC

48. Data Flow Testing (cont’d)
– A definition-use (DU) chain of variable X is of the form
[X, S, S’], where S and S’ are statement numbers, X is
DEF(S) and USE(S’), and the definition of X in statement
S is live at statement S’.
DU testing strategy
– A strategy for selecting the derived DU chains as test cases
All use coverage: every DU chain to be covered at least once
353
© by SEC

49. Example i = get_random(); j = get_random(); if (a<30) {
(1) Which lines are definition occurrences of variable i?
(2) Which lines are use occurrences of variable i?
i = get_random();
j = get_random();
if (a<30) {
i= get_random();
} else {
j=5; }
if (a>b) {
i=8;
j=6;
} else {
printf("%d\n",j); }
if (b<30) {
printf("%d\n",i);
j=6;
(1) Lines 1, 4, and 8.
(2) Lines 14 and 17.
(3) There are six (3*2) def-use pairs, 5 of which are feasible.
1-17 (a=35, b=32)
1-14(a=35, b=29)
4-14 (a=25, b=24)
4-17 (not feasible)
8-14 (a=29, b=25)
8-187 (a=35, b=34)

50. Identify the def-use pairs for variable j
Design test cases for these def-use pairs

51. Loop Testing Simple Loops (n iterations) – skip the loop entirely
– only one pass through the loop
– two passes through the loop
– m passes through the loop where m < n
– n-1, n, n+1 passes through the loop
354
© by SEC

52. Loop Testing (cont’d) Nested Loops – conduct simple loop tests for the
innermost loop while holding the outer
loops at their minimum iteration
– work outward, conducting tests for the
next innermost loop
– continue until all the loops have been
tested
– tests grow geometrically as the level of
nesting increases
355
© by SEC

53. Loop Testing (cont’d) Concatenated Loops
– independent loops using simple loop
testing
– dependent loops using nested loop
testing
356
© by SEC

54. Loop Testing (cont’d) Unstructured Loops
– Whenever possible, redesign the
code to reflect the use of structured
programming constructs.
– Then apply the appropriate loop
testing strategy.
357
© by SEC

55. Loop Testing Example Inputs:
ordered integer array A[1], ..., A[n] such that A[ i ] < A[ i+1] for i between 1 and n-1
Key: an integer to search, n: number of the array
Outputs: integer: the location of ‘key’ in the array (0 for not found)

56. Binary Search found := false low := 1 high := n
while ((low <= high) && not found) do
mid := floor( (low + high) /2 )
if A[mid] = key then
index := mid
found := true
Else
if A[mid]<key then
low := mid + 1
else
high := mid - 1
end if
end while
if not found then
index := 0

57. Summery: White Box Testing
Basis Path Testing
Identify every basis path ( V(G) )
Test every basis path
Condition Testing
Branch Testing: test every branch and condition component (True/False Combinations)
Domain Testing: test close values around a relational condition (>, ==, <)
Data Flow Testing
Test every Definition-Use path
Loop Testing
Test every loop from inside out

58. In Class Exercise
procedure bubbleSort( A : list of sortable items ) defined as:
do swapped := false
for each i in 0 to length(A) - 2 do:
if A[i] > A[i+1] then
swap( A[i], A[i+1] )
swapped := true
end if
end for
while swapped
end procedure

59. Test Adequacy Criteria
– Stopping rule determinates whether sufficient testing has
been done
How do you know when you have tested enough?
How do you pick test cases?
Control flow coverage criteria
– Statement coverage
– Edge coverage
– Condition coverage
– Path coverage
358
© by SEC

60. 6.3.2 Black-Box Testing Black-box testing
– Also known as functional testing, behavioral testing, or
specification-based testing
– Does not have the knowledge of the program’s structures
– Discover program errors based on program requirements
and product specifications
– Derive sets of inputs to fully exercise all functional
requirements for a program
– Complementary to white-box testing
359
© by SEC

61. 6.3.2 Black-Box Testing (cont’d)
– Focuses on the functional requirements of the software
including functions, operations, external interfaces,
external data and information
– Attempts to find errors in the following:
Incorrect of missing functions,
Interface errors,
Errors in data structures or external database access,
Performance errors, and
Initialization and termination errors.
360
© by SEC

62. Equivalence Partitioning
– Divide the input domain of a program into classes of data
– Derive test cases based on these partitions
– Test case design for equivalence partitioning is based on an
evaluation of equivalence classes for an input domain
361
© by SEC

63. Equivalence Partitioning (cont’d)
An equivalence class represents a set of valid or
invalid states for input condition.
An input condition is:
– a specific numeric value, a range of values
– a set of related values, or a Boolean condition
362
© by SEC

64. Equivalence Partitioning (cont’d)
Equivalence classes can be defined using the
following guidelines:
– If an input condition specifies a range, one valid and two
invalid equivalence class are defined.
– If an input condition requires a specific value, one valid
and two invalid equivalence classes are defined.
– If an input condition specifies a member of a set, one valid
and one invalid equivalence classes are defined.
– If an input condition is Boolean, one valid and one invalid
classes are defined.
363
© by SEC

65. Equivalence Partitioning: Automated
Banking Example
Consider an automated banking application. The user can dial the
bank from a personal computer, provide a six-digit password, and
follow with a series of keyword commands that activate the
banking function. The software for the application accepts data in
the following form:
Area Code
Prefix
Suffix
Password
commands
Blank or three-digit number
Three-digit number, not
beginning with 0 or 1
Four-digit number
Six-character alphanumeric
"Check", "deposit", "pay", etc.
364
© by SEC

66. Equivalence Partitioning: Automated Banking Example (cont’d)
Data Item
Area Code
Prefix
Suffix
Password
Commands
Input Condition
Boolean
range
value
Remarks
The area code may or may not be present
Values between 200 and 999 with specific
exception
Specified value > 200
Four-digit length
Password may or not be present
Six-character string
"Check", "deposit", "pay", etc.
set
Test cases for each input domain data item can be developed and
executed.
Test cases are selected so that the largest number of attributes of an
equivalence class are exercised at once
365
© by SEC

67. In Class Exercise Grocery Store Example
Consider a software module that is intended to accept:
the name of a grocery item
a list of the different sizes the item comes in, specified in ounces.
The specifications state that the item name is to be alphabetic characters 2 to 15 characters in length.
Each size may be a value in the range of 1 to 48, whole numbers only.
The sizes are to be entered in ascending order (smaller sizes first).
A maximum of five sizes may be entered for each item.
The item name is to be entered first, followed by a comma, then followed by a list of sizes.
A comma will be used to separate each size.
Spaces (blanks) are to be ignored anywhere in the input.
Please derive the equivalence partitions of the inputs.
Source:

68. Derived Equivalence Classes
Item name is alphabetic (valid)
Item name is not alphabetic (invalid)
Item name is less than 2 characters in length (invalid)
Item name is 2 to 15 characters in length (valid)
Item name is greater than 15 characters in length (invalid)
Size value is less than 1 (invalid)
Size value is in the range 1 to 48 (valid)
Size value is greater than 48 (invalid)
Size value is a whole number (valid)
Size value is a decimal (invalid)
Size value is numeric (valid)
Size value includes nonnumeric characters (invalid)
Size values entered in ascending order (valid)
Size values entered in non-ascending order (invalid)
No size values entered (invalid)
One to five size values entered (valid)
More than five sizes entered (invalid)
Item name is first (valid)
Item name is not first (invalid)
A single comma separates each entry in list (valid)
A comma does not separate two or more entries in the list (invalid)
The entry contains no blanks (???)
The entry contains blanks (????)

69. Derived Black Box Test Cases#
Test Data
Expected Outcome
Classes Covered 1
xy,1 T
1,4,7,9,11,13,16,18,20,22 2
AbcDefghijklmno,1,2,3  ,4,48
1,4,7,9,11,13,16,18,20,23 3
a2x,1 F 4
A,1 5
abcdefghijklmnop 6
Xy,0 7
XY,49 8
Xy,2.5 10 9
xy,2,1,3,4,5 14 Xy 15 11
XY,1,2,3,4,5,6 17 12
1,Xy,2,3,4,5 19 13
XY2,3,4,5,6 21
AB,2#7

70. Selecting Test Values: Boundary Values
More errors occur at the boundaries of an input/output domain
than in the “center”.
As well as selecting test data “inside” an equivalence class,
we also select data at the “edges” of the equivalence class.
user output FK
queries mouse formats input
picks
input domain
data
prompts
output
domain
366
© by SEC

71. Boundary Value Analysis
Boundary value analysis (BVA)
– A test case design technique complements to equivalence
partition
– Rather selecting any element from an equivalence class,
BVA leads to the selection of test cases that exercise
bounding values (“edge” of the class)
– Unlike equivalence partition that derives test cases only
from input conditions, BVA derives test cases from both
input conditions and output domain
367
© by SEC

72. Boundary Value Analysis (cont’d)
Guidelines:
1. If an input condition specifies a range [a, b], test cases
should be designed with value a and b, just above and
below a and b
Example: Integer D with input condition [-3, 10], BVA test values
are -3, 10, 11, -2, 0
2. If an input condition specifies a number values, test cases
should be developed to exercise the minimum, number,
maximum number, and values just above and below
minimum and maximum
Example: Enumerate data E with input condition: {3, 5, 100,
102}, BVA test values are 3, 102, 2, 4, 101, 103
368
© by SEC

73. Boundary Value Analysis (cont’d)
3. Guidelines 1 and 2 are applied to output condition
4. If internal program data structures have prescribed
boundaries, be certain to design a test case to exercise the
data structure at its boundary
Array input condition:
– empty, single element, full element, out-of-boundary
Search output condition:
– element is inside array or the element is not inside array
369
© by SEC

74. Intuition and Experiences
Also known as error guessing
Identify potential errors and design test cases based
on intuition and experiences
Test cases can be derived by making a list of possible
errors or error-prone situations
– Empty or null lists/strings
– Zero instances/occurrences
– Blanks or null characters in strings
– Negative numbers
– Keep a defect history
373
© by SEC

75. Summery: Black Box Testing
Equivalence Partitions
Boundary Value Analysis
Intuition and Experience

76. 請為下列情況設計test case 教務系統從文字檔匯入某一課程成績資料。輸入課號後，上傳檔案, 格式為學號,成績\n學號,成績\n….
若已有舊資料則覆蓋掉
成績須為 0-100之整數
學號若不存在，或成績不合法，則捨棄該行資料，並列印錯誤訊息後，繼續處理下一行
處理完成後，列印尚未輸入成績學生的學號清單

77. 請為下列情況設計test case 計算業務員薪水 底薪3萬 達到業績目標 獎金 5000 超過目標的業績，可抽成3%，抽成最高可到5萬
業績比上月成長50%(含)以上者獎金 1000

78. 6.4 Software Testing Strategies
Testing begins at the module level and works outward toward
the integration of the entire computer-based system.
Different testing techniques are appropriate at different points
in time.
Testing is conducted by the developer of the software and (for
large projects) an independent test group.
Testing and debugging are different activities, but debugging
must be accommodated in any testing strategy
374
© by SEC

79. Phases of Software Testing
Unit Testing
Module
Integration Testing
Subsystem and System
Software System
System Testing
Software System
Software + Hardware System
Emulation / Parallel Testing
Physical Environment, Installation and data download 9
© 2006 by SEC

80. Classification of Software Testing Software Testing
Static Testing
Black Box Testing
Dynamic Testing
White Box Testing
Data Flow Testing
Control Flow Testing 10
© 2006 by SEC

81. Classification of Software Testing
Black Box
Partition Testing
Boundary Testing
Transaction Flow
Testing
Syntax Testing
Logic-based Testing
Random Testing
White Box
Control Flow Testing
Statement Coverage
Branches Coverage
Conditions Coverage
Loops Coverage
Data Flow Testing
All-Def-Path
All-Use-Path
All-DU-Path 11
© 2006 by SEC

82. 6.4.1 Verification and Validation
Software testing is one element of a broader topic
that is often referred to as verification and validation
(V&V).
Verification – are we building the product correctly?
– Refers to the set of activities that ensure that software
correctly implements a specific function.
Validation – are we building the correct product?
– Refers to the set of activities that ensure that the software
that has been built is traceable to customer requirements
375
© by SEC

83. 6.4.1 Verification and Validation (cont’d)
The definition of V& V encompasses many of the
activities that we have referred to as software quality
assurance (SQA) including
– Formal technical review, quality and configuration audits,
performance monitoring, simulation, feasibility study,
documentation review, database review, algorithm analysis,
development testing, qualification testing, and installation
testing
376
© by SEC

84. 6.4.2 The V Model
The V model
– Emerged in reaction to some waterfall models that showed
testing as a single phase following the traditional
development phases of requirements analysis, high-level
design, detailed design and coding.
– The V model portrays several distinct testing levels and
illustrates how each level addresses a different stage of the
software lifecycle.
– The V shows the typical sequence of development
activities on the left-hand (downhill) side and the
corresponding sequence of test execution activities on the
right-hand (uphill) side.
377
© by SEC

85. 6.4.2 The V Model (cont’d) Requirements Analysis Architecture Design
Detailed
Code
Acceptance
Testing
System
Integration
Unit Testing
378
© by SEC

86. 6.4.2 The V Model (cont’d)
The V model is valuable because it highlights the
existence of several levels of testing and delineates
how each relates to a different development phase:
– Unit testing: concentrates on each unit (i.e., component) of
the software (white box)
– Integration testing: focuses on design and the construction
of the software architecture (black box, limited amount of
white box)
379
© by SEC

87. 6.4.2 The V Model (cont’d)
– System testing: verifies that all elements mesh properly and
that overall system function/performance is achieved.
– Acceptance testing: are ordinarily performed by the
business/users to confirm that the product meets the
business requirements.
380
© by SEC

88. 6.4.3 Unit Testing Unit testing
– Is normally considered as an adjunct to the coding step
– Focuses verification effort on the smallest unit of software design – the
software component or module
– Using the component-level design description as a guide
– Provide a release criterion for a programming task
– Encourage early defect discovery
– Discourage big-bang integration with its late defect discovery
– Find "side effect" defects, especially in highly coupling designs.
– Is white-box oriented and the step can be conducted in parallel for
multiple components
381
© by SEC

89. 6.4.3 Unit Testing (cont’d) interface local data structures
boundary conditions
independent paths
error-handling paths
Test
Cases
Module
382
© by SEC

90. Considerations of Unit Test
Interface
– Ensures that information properly flows in and out of the component
Local data structures
– Ensures that data stored temporarily maintains its integrity during
execution
Boundary conditions
– Ensures that the component operates properly at boundaries established
to limit or restrict processing
Independent paths (basis paths)
– Ensures that all paths in a component have been executed at least once
Error-handling paths
– Ensures that errors are correctly handled
383
© by SEC

91. Unit Test Cases Unit test cases should be designed to uncover errors
due to erroneous computation, incorrect
comparisons, or improper control flow
Common errors in computation
– Misunderstood or incorrect arithmetic precedence
– Mixed mode operations
– Incorrect initialization
– Precision inaccuracy
– Incorrect symbolic representation of an expression
384
© by SEC

92. Unit Test Cases (cont’d)
Common errors in comparison and control flow
– Comparison of different data types
– Incorrect logical operators or precedence
– Expectation of equality when precision error makes
equality unlikely
– Incorrect comparison of variables
– Improper or nonexistent loop termination
– Failure to exit when divergent iteration is encountered
– Improperly modified loop variables
385
© by SEC

93. Unit Test Cases (cont’d)
What should be tested when error handling is evaluated?
– Error description is unintelligible
– Error noted does not correspond to error encountered
– Error condition causes system intervention prior to error handling
– Exception-condition processing is incorrect
– Error description does not provide enough information to assist in the
location of the cause of the error
Boundary testing
– An important task of the unit test step
– Test cases that exercise data structure, control flow, and data values
just below, at, and just above maxima and minima are vary likely to
uncover errors
386
© by SEC

94. Unit Test Procedure
Review of design information (provided as guidance) to
establish test cases that are likely to uncover errors previous
noted
Each test case should be couple with a set of expected results
Develop driver and/or stub software for each unit test
– Driver: nothing more than a “main program” that accept test case data,
passes such data to the component to be tested, and prints relevant
results
– Stub: serves to replace modules that are subordinate called by the
component to be tested.
– Drivers and stubs represent overhead
Execute and evaluate the unit test
387
© by SEC

95. Unit Test Environment interface local data structures
Driver
interface
local data structures
boundary conditions
independent paths
error-handling paths
Test
Module
to be
tested
Stub
Stub
results
Cases
388
© by SEC

96. Component Testing
When testing objects, complete test coverage should include
The testing in isolation of all operations associated with the object.
The setting and interrogation of all attributes associated with the object.
The exercise of the object in all possible states.
All events that cause a state change in the object should be simulated.

97. Component Testing (cont’d)
Test the attribute
Check if it has been set up when the weather station is installed.
Test the operation
Testing reportWeather, calibrate, test, startup, shutdown in isolation or in a specific sequence.

98. Component Testing (cont’d)
Testing states
A state model is needed.
Identify sequences of state transitions that have to be tested and define event sequences to force these transitions.
Examples of state sequences that should be tested.
Shutdown ® Waiting ® Shutdown
Waiting ® Calibrating ® Testing ® Transmitting ® Waiting
Waiting ® Collecting ® Waiting ® Summarising ® Transmitting ® Waiting

99. Component Testing (cont’d)
When inheritance is used, it is more difficult to test
All subclasses should be tested with all inherited operations.
Reason: the inherited operation may make assumptions about other operations and attributes and these may have been changed when inherited.

100. State Model of the class “WeatherStation”
Operation
calibrate()
Calibrating
calibration OK
Shutdown
Startup()
test()
Waiting
Testing
transmission done
test complete
shutdown()
Transmitting
collection
done
clock
reportWeather()
weather summary
complete
Collecting
Summarising

101. 6.4.4 Integration Testing Big bang (non-incremental integration)
– All components are combined in advance. The entire
program is tested as a whole
– When a set of errors is encountered, correction is difficult
because isolation of causes is complicated by the vast
expanse of entire program
– Once these errors are corrected, new ones appear and the
process continues in a seemingly endless loop
389
© by SEC

102. 6.4.4 Integration Testing (cont’d)
Incremental integration
– The program is constructed and tested in small increments,
where errors are easier to isolate and correct
– Interfaces are more likely to be tested completely
– A systematic test approach may be applied
Top-down integration
Bottom-up integration
Sandwich testing (combination of above two approaches)
390
© by SEC

103. Top Down Integration Modules are integrated by moving download through
the control hierarchy
– Beginning with the main control module (main program)
– Modules subordinate (and ultimately subordinate) to main
control module are incorporated into the structure in either
a depth-first or breadth-first manner
391
© by SEC

104. Top Down Integration (cont’d)
The integration process:
– The main control module is used as a test driver and stubs
are substituted for all components directly subordinate to
the main control module
– Depending the depth-first or breadth-first, subordinate
stubs are replaced one at a time with actual components
– Test are conducted as each component is integrated
– On completion of each set of tests, another stub is replaced
with the real component
– Regression testing may be conducted to ensure that new
errors have not been introduced
– Repeat from step 2 until the entire program structure is
built
392
© by SEC

105. Top Down Integration (cont’d)
top module is
tested with stubs M1 M2 M3 M4
stubs are replaced
one at a time M5 M6 M7
as new modules are integrated, M8
some subset of tests is re-run
393
© by SEC

106. Top Down Integration (cont’d)
Advantages
– Is better at discovering errors in the system architecture (verifies major
control or decision points early)
– Can demonstrate a complete function of the system early (if depth-first
integration is selected)
Disadvantages
– Logistical problems can raise 　
– Need to write and test stubs (costly)
To solve the logistical problem:
– Delay many tests until stubs are replaced with actual modules
– Develop stubs that perform limited functions that simulate the actual
module (requires significant overhead)
– Integrate the software from the bottom of the hierarchy upward
394
© by SEC

107. Bottom Up Integration
Begins construction and testing with atomic modules (i.e.,
components at the lowest levels in the program structure).
The integration process:
– Low-level components are combined into clusters (sometimes called
builds) that perform a specific software subfunction
– A test driver is written to coordinate test case input and output
– The cluster is tested
– The test drivers are removed and clusters are combined moving upward
in the program structure.
As the integration moves upward, the need for separate test
drivers lessens.
395
© by SEC

108. Bottom Up Integration (cont’d) McMa Mb D1 D2 D3
Cluster 3
Cluster 1
Cluster 2
396
© by SEC

109. Bottom Up Integration (cont’d)
Advantages
– Easier test case design and no need for stubs
– Can starts at an early stage in the development process (not
require to wait until the architectural design of the system
to be completed)
– Potentially reusable modules are adequately tested
Disadvantages
– User interface components are tested last
– The program as an entity does not exist until the last
module is added
– Major design faults show up late
397
© by SEC

110. Sandwich Integration Testing
The advantages of one integration strategy tend to result in
disadvantages for the other strategy.
In practice, most integration involves a combination of both
the top-down and bottom-up integration strategies, i.e.,
sandwich integration and testing
The sandwich integration and testing
– The decision modules (logic modules) are integrated and tested top-
down so that major design faults can be revealed early
– The worker modules (operational modules) are integrated and tested
bottom-up so that the potentially reusable modules are adequately
tested while reducing the constructions of test stubs
– The sandwich integration and testing has the strengths of both top-
down and bottom-up integration strategies while avoiding their
weaknesses
398
© by SEC

111. 6.4.5 Regression Testing Each time a new module is added as part of
integration testing, or a bug is fixed (as the results of
uncovering errors through testing) in the software,
the software changes.
These changes may cause problems with functions
that previously worked.
Regression testing is the re-execution of some subset
of tests that have already been conducted to ensure
that changes have not generated unintended side
effects.
399
© by SEC

112. 6.4.5 Regression Testing (cont’d)
Regression testing may be conducted
– Manually by re-executing a subset of all test cases
– Using automated capture/playback tools – enable software engineer to
capture test cases and results for subsequent playback and comparison.
The regression test suite contains three different classes of test
cases:
– A representative sample of tests that will exercise all software
functions
– Additional tests that focus on software functions that are likely to be
affected by the change
– Tests that focus on the software components that have been changed
400
© by SEC

113. 6.4.5 Regression Testing (cont’d)
As integration testing proceeds, the number of
regression tests can grown quite large
– The regression test suite should be designed to include only
those tests that address one or more classes of errors in
each of the major program functions.
– It is impractical and inefficient to re-execute every test for
every program function once a change has occurred
401
© by SEC

114. 6.4.6 Validation Testing Validation testing: Validation test criteria
– validation succeeds when software functions in a manner that can be
reasonably expected by the customer
– reasonable expectations are defined in the Software Requirements
Specification: a section called Validation Criteria
Validation test criteria
– All functional requirements are satisfied
– All behavioral characteristics are achieved
– All performance requirements are attained
– Documentation is correct
– Human engineering and other requirements are met (transportability,
compatibility, error recovery, maintainability)
402
© by SEC

115. 6.4.6 Validation Testing (cont’d)
Configuration review (sometimes called an audit)
– An important element of the validation process
– ensures that all elements of the software delivered to the
customer (the software configuration) have been properly
developed and documented
Alpha and beta testing
– When software is developed as a product to be used by
many customers, a process called alpha and beta testing is
used to uncover errors that only end-user seems able to
find.
403
© by SEC

116. 6.4.7 Alpha Testing Alpha Tests – Conducted at developer’s site
– Conducted by customer
– Developer looks over the shoulder
– Developer records the errors and problems
– Conducted in a controlled environment
404
© by SEC

117. 6.4.8 Beta Testing Beta Tests
– Conducted at one or more customer sites
– Conducted by end-user
– Developer is not present
– Uncontrolled environment
– Errors may be real or imagined
– Customer reports the errors
– Accordingly modifications are made
– Final product is released
For commercial products α - β testing is done
405
© by SEC

118. 6.4.9 System Testing System test: The focus areas are:
– The system software is tested as a whole. It verifies all elements mesh
properly to make sure that all system functions and performance are
achieved in the target environment.
The focus areas are: –
System functions and performance –
System reliability and recoverability (recovery test)
System installation (installation test) – –
System behavior in the special conditions (stress and load test)
System user operations (acceptance test/alpha test)
Hardware and software integration and collaboration
Integration of external software and the system – – –
System testers: test engineers in ITG or SQA people.
406
© by SEC

119. 6.4.9 System Testing (cont’d)
Recovery tests:
– verify that the system can recover when forced to fail in various ways
database recovery is particularly important
– e.g., measure time to recover (MTTR)
Stress tests:
– verify that the system can continue functioning when confronted with
many simultaneous requests (abnormal situations)
– execute the system by demanding resource in abnormal quantity,
frequency, or volume (subject to extreme data & event traffic )
– excessive interrupt, high input data rate, maximum memory, …
– how high can we go? Do we fail-soft or collapse?
– sensitivity testing (a variation of stress testing)
– attempts to uncover data combinations within valid input classes that
may cause instability or improper processing (performance
degradation)
407
© by SEC

120. 6.4.9 System Testing (cont’d)
Security tests:
– Verify that access protection mechanisms work make penetration cost
more than value of entry
– Subject to compromise attempts
– E.G., Measure average time to break in
Performance:
– Is designed to test the run-time performance of software (real-time and
embedded systems) within the context of an integrated system
– Measure speed, resource utilization under various circumstances.
– Is often coupled with stress testing and usually requires both hardware
and software instrumentation
– Occurs throughout all steps in the testing process
408
© by SEC

121. Testing on Web-Based Softwares More Difficult than Traditional Testing
The Heterogeneous Nature of their components
The Heterogeneous Nature of the running
environments
The extremely wide number of users distributed
all around the world and accessing the application
services in a concurrent and global way
The possibility to generate software components
at run time 23
© 2006 by SEC

122. Special Requirements of Web-based Software Testing
Browser Independent
No Buffering, Caching
Fronts and Preferences
Object Mode
Tables and Forms
Frames 26
© 2006 by SEC

123. Web Testing Tools Web testing tools Web log Analyzers
Weblint;
W3C Validator;
Net Mechanic;
Doctor HTML;
TCAT for Java;
SilkTest;
Visual Test;
Web log Analyzers
FastStats;
Analog; 84
© 2006 by SEC

124. A Poor Test Case Example
Test ID
Description
Expected Result
Actual Result 1
Player 1 rolls dice and moves
Player 1 moves on board 2
Player 2 rolls dice and moves
Player 2 moves on board

125. A Better Test Case Example
Test ID
Description
Expected Result
Actual Result 1
Game begins and initiated with two players
Player 1 has $1200 and in location 0
Player 2 has $1200 and in location 0 2
Precondition:
Test 1 success
Player 1 dice roll: 3
Player 1 moves to location 3 3
Test 1 & 2 has completed
Player 2 dice roll: 4
Player 1 is on location 3
Player 2 moves to location 4

126. A Monopoly Example Requirement:
When a player lands on the “Go to Jail” cell, the player goes directly to jail, does not pass go, does not collect $200
On the next turn, the player must pay $50 to get out of jail and does not roll the dice or advance. If the player does not have enough money, he or she is out of the game

127. Monopoly: derived tests
Does the player get sent to jail after landing on “Go to Jail”?
Does the player receive $200 if “Go” is between the current space and jail?
Is $50 correctly deducted if the player has more than $50?
Is the player out of the game if he or she has less than $50? …

128. Derived Test Case 1 Precondition: Only 1 player Player has $1200
Test ID
Description
Expected Result
Actual Result 1
Precondition:
Only 1 player
Player has $1200
Player is 3 steps to “Go to Jail” Cell
“Go” is between the Jail and “Go to Jail” Cell
Player dice roll: 3
Player clicks “End Turn” button
Player is sent to jail
Player’s money is $1200
On next turn, the “Get Out of Jail” button is enabled and “Roll Dice” Button is disabled
1.1
Player clicks “Get Out of Jail” button.
Player’s money is $1150
The “Get Out of Jail” button is disabled

129. Derived Test Case 2 Precondition: Only 1 player Player has $20
Test ID
Description
Expected Result
Actual Result 2
Precondition:
Only 1 player
Player has $20
Player is 3 steps to “Go to Jail” Cell
“Go” is between the Jail and “Go to Jail” Cell
Player dice roll: 3
Player clicks “End Turn” button
Player is sent to jail
Player’s money is $20
On next turn, the “Get Out of Jail” button is enabled and “Roll Dice” Button is disabled
2.1
Player clicks “Get Out of Jail” button.
Screen displays ”You are out!”
The game ends

130. Requirements:
If a Player (A) lands on property owned by another player (B), A must pay rent to B.
If A does not have enough money to pay B, A is out of the game.

131. Derived Test Case 3 Precondition: Player A has $1200
Test ID
Description
Expected Result
Actual Result 3
Precondition:
Player A has $1200
A is 3 steps to B’s Cell: “Golden Garden”
Rent for “Golden Garden” is $100
A dice roll: 3
A clicks “End Turn” button
A moves to Cell: “Golden Garden”
Screen displays “Pay Rent to B: $100”
A’s money is $1100

132. Derived Test Case 3 Precondition: Player A has $50
Test ID
Description
Expected Result
Actual Result 3
Precondition:
Player A has $50
A is 3 steps to B’s Cell: “Golden Garden”
Rent for “Golden Garden” is $100
A dice roll: 3
A clicks “End Turn” button
A moves to Cell: “Golden Garden”
Screen displays “The Rent is $100, Player A is broke!”
A is kicked out of the game