1. Black Box Testing
Testing software against a specification of its external behavior without knowledge of internal implementation details
Can be applied to software “units” (e.g., classes) or to entire programs
External behavior is defined in API docs, Functional specs, Requirements specs, etc.
Because black box testing purposely disregards the program's control structure, attention is focused primarily on the information domain (i.e., data that goes in, data that comes out)
The Goal: Derive sets of input conditions (test cases) that fully exercise the external functionality

2. Black Box Testing
Black box testing tends to find different kinds of errors than white box testing
Missing functions
Usability problems
Performance problems
Concurrency and timing errors
Initialization and termination errors
Etc.
Unlike white box testing, black box testing tends to be applied later in the development process
Test cases are designed using only functional specification of the software:
without any knowledge of the internal structure of the software. For this reason, black-box testing is also known as functional testing.

3. The Information Domain: inputs and outputs
Individual input values
Try many different values for each individual input
Combinations of inputs
Individual inputs are not independent from each other
Programs process multiple input values together, not just one at a time
Try many different combinations of inputs in order to achieve good coverage of the input domain
Ordering and Timing of inputs
In addition to the particular combination of input values chosen, the ordering and timing of the inputs can also make a difference

4. The Information Domain: inputs and outputs
Defining the input domain
Boolean value
T or F
Numeric value in a particular range
99 <= N <= 99
Integer, Floating point
One of a fixed set of enumerated values
{Jan, Feb, Mar, …}
{Visa, MasterCard, Discover, …}
Formatted strings
Phone numbers
File names
URLs
Credit card numbers
Regular expressions

5. Equivalence Partitioning

6. Equivalence Partitioning
Typically the universe of all possible test cases is so large that you cannot try them all
You have to select a relatively small number of test cases to actually run
Which test cases should you choose?
Equivalence partitioning helps answer this question

7. Equivalence Partitioning
Partition the test cases into "equivalence classes"
Each equivalence class contains a set of "equivalent" test cases
Two test cases are considered to be equivalent if we expect the program to process them both in the same way (i.e., follow the same path through the code)
If you expect the program to process two test cases in the same way, only test one of them, thus reducing the number of test cases you have to run

8. Equivalence Partitioning
First-level partitioning: Valid vs. Invalid test cases
Valid
Invalid

9. Equivalence Partitioning
Partition valid and invalid test cases into equivalence classes

10. Equivalence Partitioning
Create a test case for at least one value from each equivalence class

11. Equivalence Partitioning
When designing test cases, you may use different definitions of “equivalence”, each of which will partition the test case space differently
Example: int Add(n1, n2, n3, …)
Equivalence Definition 1: partition test cases by the number of inputs (1, 2, 3, etc.)
Equivalence Definition 2: partition test cases by the number signs they contain (positive, negative, both)
Equivalence Definition 3: partition test cases by the magnitude of operands (large numbers, small numbers, both)
Etc.

12. Equivalence Partitioning
When designing test cases, you may use different definitions of “equivalence”, each of which will partition the test case space differently
Example: string Fetch(URL)
Equivalence Definition 1: partition test cases by URL protocol (“http”, “https”, “ftp”, “file”, etc.)
Equivalence Definition 2: partition test cases by type of file being retrieved (HTML, GIF, JPEG, Plain Text, etc.)
Equivalence Definition 3: partition test cases by length of URL (very short, short, medium, long, very long, etc.)
Etc.

13. Equivalence Partitioning
If an oracle is available, the test values in each equivalence class can be randomly generated. This is more useful than always testing the same static values.
Oracle: something that can tell you whether a test passed or failed
Test multiple values in each equivalence class. Often you’re not sure if you have defined the equivalence classes correctly or completely, and testing multiple values in each class is more thorough than relying on a single value.

14. Equivalence Partitioning - examples
Input
Valid Equivalence Classes
Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99 ?
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits

15. Equivalence Partitioning - examples
Input
Valid Equivalence Classes
Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
[1, 9]
[10, 99] ?
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits

16. Equivalence Partitioning - examples
Input
Valid Equivalence Classes
Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits ?

17. Equivalence Partitioning - examples
Input
Valid Equivalence Classes
Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
Prefix: (200, 999)
Suffix: Any 4 digits
(555)
200 <= Area code <= 999
200 < Prefix <= 999 ?

18. Equivalence Partitioning - examples
Input
Valid Equivalence Classes
Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
(555)
200 <= Area code <= 999
200 < Prefix <= 999
Invalid format , (555)(555)5555, etc.
Area code < 200 or > 999
Area code with non-numeric characters
Similar for Prefix and Suffix

19. Example 1
Consider an application A that takes an integer denoted by age as input. Let us suppose that the only legal values of age are in the range [1..120]. The set of input values is now divided into a set E containing all integers in the range [1..120] and a set U containing the remaining integers.
All integers
[1..120]
Other integers
© Aditya P. Mathur 2006

20. Example 1 (contd.)
Further, assume that the application is required to process all values in the range [1..61] in accordance with requirement R1 and those in the range [ ] according to requirement R2.
E is further subdivided into two regions depending on the expected behavior.
Similarly, it is expected that all invalid inputs less than or equal to 1 are to be treated in one way while all greater than 120 are to be treated differently.
U is further subdivided two categories.
<1
All integers
[62-120]
>120
[1..61]
© Aditya P. Mathur 2006

21. Example 1 (contd.)
Tests selected using the equivalence partitioning technique aim at targeting faults in the application under test with respect to inputs in any of the four regions, i.e. two regions containing expected inputs and two regions containing the unexpected inputs.
It is expected that any single test selected from the range [1..61] will reveal any fault with respect to R1.
Similarly, any test selected from the region [ ] will reveal any fault with respect to R2.
A similar expectation applies to the two regions containing the unexpected inputs.
© Aditya P. Mathur 2006

22. Example 2
This example shows a few ways to define equivalence classes based on the knowledge of requirements and the program text.
Consider a wordCount method which takes a word w and a filename f as input and returns the number of occurrences of w in the text contained in the file named f.
An exception is raised if there is no file with name f.
© Aditya P. Mathur 2006

23. Example 2 (contd.) begin String w, f Input w, f
if (not exists(f) {raise exception; return(0);}
if(length(w)==0) return(0);
if(empty(f)) return(0);
return(getCount(w,f));
end
Using the partitioning method described in the examples above, we obtain the following equivalence classes.
© Aditya P. Mathur 2006

24. Example 2 (contd.) Equivalence class w f E1 non-null exists, not empty
does not exist E3
exists, empty E4
null E5 E6
© Aditya P. Mathur 2006

25. Example 2 (contd.)
The number of equivalence classes without any knowledge of the program code is 2, while the number of equivalence classes derived with the knowledge of partial code is 6.
Of course, an experienced tester will likely derive the six equivalence classes given above, and perhaps more, even before the code is available
© Aditya P. Mathur 2006

26. Unidimensional partitioning
One way to partition the input domain is to consider one input variable at a time.
Thus, each input variable leads to a partition of the input domain.
We refer to this style of partitioning as unidimensional equivalence partitioning or simply unidimensional partitioning.
This type of partitioning is commonly used.
© Aditya P. Mathur 2006

27. Multidimensional partitioning
Another way is to consider the input domain I as the cross product of the domains of the input variables and define a relation on I.
This procedure creates one partition consisting of several equivalence classes. We refer to this method as multidimensional equivalence partitioning or simply multidimensional partitioning.
Multidimensional partitioning leads to a large number of equivalence classes that is difficult to manage manually.
Many classes so created might be infeasible.
Nevertheless, equivalence classes so created offer an increased variety of tests as is illustrated in the next section.
© Aditya P. Mathur 2006

28. Partitioning Example
Consider an application that requires two integer inputs x and y, expected to lie in the following ranges: 3x7 and 5y9.
For unidimensional partitioning we apply the partitioning guidelines to x and y individually. This leads to the following six equivalence classes.
E1: x<3
E2: 3x7
E3: x>7
y ignored.
E4: y<5
E5: 5y9
E6: y>9
x ignored.
For multidimensional partitioning we consider the input domain to be the cross product of x and y. This leads to 9 equivalence classes.
E1: x<3, y<5
E2: x<3, 5y9
E3: x<3, y>9
E4: 3x7, y<5
E5: 3x7, 5y9
E6: 3x7, y>9
E7: x>7, y<5
E8: x>7, 5y9
E9: x>7, y>9

29. Partitioning Example (contd.)
6 equivalence classes:
E1: x<3, y<5
E3: x<3, y>9
E2: x<3, 5y9
E4: 3x7, y<5
9 equivalence classes:
E5: 3x7, 5y9
E6: 3x7, y>9
E7: x>7, y<5
E8: x>7, 5y9
E9: x>7, y>9
© Aditya P. Mathur 2006

30. Boundary Value Analysis

31. Boundary Value Analysis
When choosing values from an equivalence class to test, use the values that are most likely to cause the program to fail
Errors tend to occur at the boundaries of equivalence classes rather than at the "center"
If (200 < areaCode && areaCode < 999) { // valid area code }
Wrong!
If (200 <= areaCode && areaCode <= 999) { // valid area code }
Testing area codes 200 and 999 would catch this error, but a center value like 770 would not
In addition to testing center values, we should also test boundary values
Right on a boundary
Very close to a boundary on either side

32. Boundary Value Analysis
Create test cases to test boundaries of equivalence classes

33. Boundary Value Analysis - examples
Input
Boundary Cases
A number N such that:
-99 <= N <= 99 ?
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits

34. Boundary Value Analysis - examples
Input
Boundary Cases
A number N such that:
-99 <= N <= 99
-100, -99, -98
-10, -9
-1, 0, 1
9, 10
98, 99, 100
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits ?

35. Boundary Value Analysis - examples
Input
Boundary Cases
A number N such that:
-99 <= N <= 99
-100, -99, -98
-10, -9
-1, 0, 1
9, 10
98, 99, 100
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
Area code: 199, 200, 201
Area code: 998, 999, 1000
Prefix: 200, 199, 198
Prefix: 998, 999, 1000
Suffix: 3 digits, 5 digits

36. Boundary Value Analysis - examples
Numeric values are often entered as strings which are then converted to numbers internally [int x = atoi(str);]
This conversion requires the program to distinguish between digits and non-digits
A boundary case to consider: Will the program accept / and : as digits?
Char / 1 2 3 4 5 6 7 8 9 : 47 48 49 50 51 52 53 54 55 56 57 58
ASCII

37. An Example – Testing a Triangle Analyzer
Topic: Black Box Software Testing
An Example – Testing a Triangle Analyzer
Testing a triangle analyzer:
Program specification:
Input: 3 numbers separated by commas or spaces
Processing:
Determine if three numbers make a valid triangle; if not, print
message NOT A TRIANGLE.
If it is a triangle, classify it according to the length of the sides as scalene
(no sides equal), isosceles (two sides equal), or equilateral (all sides equal).
If it is a triangle, classify it according to the largest angle as acute (less than
90 degree), obtuse (greater than 90 degree), or right (exactly 90 degree).
Output: One line listing the three numbers provided as input and the classification
or the not a triangle message.
Example: 3,4,5 Scalene Right
6,1,6 Isosceles Acute
5,1,2 Not a triangle
Jerry Gao Ph.D. 7/2002

38. An Example Functional Test Cases: Acute Obtuse Right
Topic: Black Box Software Testing
An Example
Functional Test Cases:
Acute Obtuse Right
Scalene: 6,5,3 5,6,10 3,4,5
Isosceles: 6,1,6 7,4,4 1,2, 2^(0.5)
Equilateral: 4,4,4 Not possible Not possible
Input Expected Results
4,4,4 Equilateral acute
1,2,8 Not a triangle
6,5,3 Scalene acute
5,6,10 Scalene obtuse
3,4,5 Scalene right
6,1,6 Isosceles acute
7,4,4 Isosceles obtuse
1,1,2^(0.5) Isosceles right
Jerry Gao Ph.D. 7/2002

39. An Example 5,5,A Character as a side 6,-4,6 Negative number as a side
Topic: Black Box Software Testing
An Example
Test cases for special inputs and invalid formats:
3,4,5,6 Four sides
646 Three-digit single number
3,,4,5 Two commas
3 4,5 Missing comma
3.14.6,4,5 Two decimal points
4,6 Two sides
5,5,A Character as a side
6,-4,6 Negative number as a side
-3,-3,-3 All negative numbers
Empty input
Jerry Gao Ph.D. 7/2002

40. An Example Boundary Test Cases:
Topic: Black Box Software Testing
An Example
Boundary Test Cases:
(1) Boundary conditions for legitimate triangles
1,1,2 Makes a straight line, not a triangle
0,0,0 Makes a point, not a triangle
4,0,3 A zero side, not a triangle
1,2, Close to a triangle but still not a triangle
9170,9168,3 Very small angle Scalene, acute
.0001,.0001,.0001 Very small triangle Equilateral, acute
, , Very large triangle, scalene, obtuse
Boundary conditions for sides classification:
,3,3 Very close to equilateral, Isosceles, acute
,4,5 Very close to isosceles Scalene, acute
Boundary conditions for angles classification:
3,4, Near right triangle Scalene, obtuse
1,1, Near right triangle Isosceles, acute
Jerry Gao Ph.D. 7/2002

41. Ad Hoc Exploratory Testing (Error Guessing)
Based on intuition, guess what kinds of inputs might cause the program to fail
Create some test cases based on your guesses
Intuition will often lead you toward boundary cases, but not always
Some special cases aren't boundary values, but are mishandled by many programs
Try exiting the program while it's still starting up
Try loading a corrupted file
Try strange but legal URLs:

42. Performance Testing Measure the system's performance
Running times of various tasks
Memory usage, including memory leaks
Network usage (Does it consume too much bandwidth? Does it open too many connections?)
Disk usage (Is the disk footprint reasonable? Does it clean up temporary files properly?)
Process/thread priorities (Does it play well with other applications, or does it hog the whole machine?)

43. Limit Testing Test the system at the limits of normal use
Test every limit on the program's behavior defined in the requirements
Maximum number of concurrent users or connections
Maximum number of open files
Maximum request size
Maximum file size
Etc.
What happens when you go slightly beyond the specified limits?
Does the system's performance degrade dramatically, or gracefully?

44. Stress Testing
Test the system under extreme conditions (i.e., beyond the limits of normal use)
Create test cases that demand resources in abnormal quantity, frequency, or volume
Low memory conditions
Disk faults (read/write failures, full disk, file corruption, etc.)
Network faults
Unusually high number of requests
Unusually large requests or files
Unusually high data rates (what happens if the network suddenly becomes ten times faster?)
Even if the system doesn't need to work in such extreme conditions, stress testing is an excellent way to find bugs

45. Random Testing Randomly generate test inputs
Could be based on some statistical model
How do you tell if the test case succeeded?
Where do the expected results come from?
Some type of “oracle” is needed
Expected results could be calculated manually
Possible, but lots of work
Automated oracles can often be created to measure characteristics of the system
Performance (memory usage, bandwidth, running times, etc.)
Did the system crash?
Maximum and average user response time under simulated user load

46. Security Testing
Any system that manages sensitive information or performs sensitive functions may become a target for intrusion (i.e., hackers)
How feasible is it to break into the system?
Learn the techniques used by hackers
Try whatever attacks you can think of
Hire a security expert to break into the system
If somebody broke in, what damage could they do?
If an authorized user became disgruntled, what damage could they do?

47. Usability Testing
Is the user interface intuitive, easy to use, organized, logical?
Does it frustrate users?
Are common tasks simple to do?
Does it conform to platform-specific conventions?
Get real users to sit down and use the software to perform some tasks
Watch them performing the tasks, noting things that seem to give them trouble
Get their feedback on the user interface and any suggested improvements
Report bugs for any problems encountered

48. Recovery Testing
Try turning the power off or otherwise crashing the program at arbitrary points during its execution
Does the program come back up correctly when you restart it?
Was the program’s persistent data corrupted (files, databases, etc.)?
Was the extent of user data loss within acceptable limits?
Can the program recover if its configuration files have been corrupted or deleted?
What about hardware failures? Does the system need to keep working when its hardware fails? If so, verify that it does so.

49. Configuration Testing
Test on all required hardware configurations
CPU, memory, disk, graphics card, network card, etc.
Test on all required operating systems and versions thereof
Virtualization technologies such as VMWare and Virtual PC are very helpful for this
Test as many Hardware/OS combinations as you can
Test installation programs and procedures on all relevant configurations

50. Compatibility Testing
Test to make sure the program is compatible with other programs it is supposed to work with
Ex: Can Word 12.0 load files created with Word 11.0?
Ex: "Save As… Word, Word Perfect, PDF, HTML, Plain Text"
Ex: "This program is compatible with Internet Explorer and Firefox"
Test all compatibility requirements

51. Documentation Testing
Test all instructions given in the documentation to ensure their completeness and accuracy
For example, “How To ...” instructions are sometimes not updated to reflect changes in the user interface
Test user documentation on real users to ensure it is clear and complete

52. References Sources: Code Complete, 2nd Ed., Steve McConnell
Software Engineering, 5th Ed., Roger Pressman
Testing Computer Software, 2nd Ed., Cem Kaner, et. Al.