Introduction to Software Testing

Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 2

What is Software Testing? 3

software testing: Evaluating software by observing its execution Primary goal of testing is to find bugs! During testing, should execute tests that find bugs. “Testing can only prove the presence of bugs, not their absence.” - Dijkstra When do you stop testing? What is Software Testing? 4

test case: A set of inputs, execution conditions, and a pass/fail criterion. – Example: height input is 5, width input is “Hello”, the test passes if the program produces an error message. test or test execution: The activity of executing test cases and evaluating their results. test suite: A set of test cases. Test Terminology 5

defect: A mistake in the source code (static) – Synonyms: fault, bug failure: An observable incorrect program behavior (output) debugging: The process of finding and correcting a defect in a program given a failure More Test Terminology 6

Software Testing is Hard! Why? 7

// Solve ax 2 + bx + c = 0 struct RootAnswer { float root_one; float root_two; } RootAnswer roots(float a, float b, float c); Which values for a, b, c should we test? Need to select test cases intelligently! Testing Inputs 8

Due to time constraints, only a small subset of actual inputs can be used to test. Need to choose a test suite such that… – Each “functionality” provided by a program is tested by at least one test case. – Tests target common programming mistakes. – Tests exercise areas of code that are likely to be “buggy”. Test Case Selection 9

Code Inspection: Looking at the source code directly for bugs. Tools – compiler – static analyzer – memory leak detector – profiler / performance monitors Other Techniques for Finding Bugs 10

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 11

Functional testing: Testing to determine if the software is working properly based on the specification (has the proper function). Non-functional testing: Testing other desirable properties of the software. – Examples: performance, usability – Testing is vastly different. Functional vs. Non-Functional Testing 12

Unit Testing: Testing of a “unit” in the program. – Unit: smallest part of the program – only assigned to one developer. Often a single function or class. – Focuses on the functionality of that unit. – Can be automated using unit testing tools Visual Studio Junit Google Testing Framework Unit Testing 13

Integration Testing: Testing the interactions of the different units. – Assumes unit testing is complete or will catch errors within the unit. – Focuses on the interactions between units. – Can be tested manually or can be automated. Integration Testing 14

System Testing: Testing the entire system. – Requires the entire system to be present. – Used to test common usage scenarios. – Used to test non-functional requirements. – Best tested manually. Testing approaches are vastly different for non- functional requirements. System Testing 15

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 16

Derive tests from the specification of the unit. Need to choose tests… – Intelligently – Systematically – Thoroughly Consider the unit under a test as a “function” with inputs and outputs – All unique and interesting inputs should be tested – All unique and interesting outputs should be produced – Focus on common errors Unit Testing Strategies 17

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 18

Equivalence Class Testing: Partition the input space into equivalence classes. The key is to divide the input space into equivalence classes such that all inputs within each equivalence class have the same functionality. – Very likely that each input within a equivalence class will exercise the same code within the function. Need to have at least one test in each equivalence class. – More tests are better. Equivalence Class Testing 19

Equivalence Class Testing Failure (valuable test case) Failures are sparse in the space of possible inputs but dense in some parts of the space If we systematically test some cases from each part, we will include the dense parts Functional testing is one way of drawing orange lines to isolate regions with likely failures The space of possible input values No failure 20

The division into equivalence classes is based on the desired / specified functionality of the subject under test. Equivalence classes may consist of: – one particular value – a range of values – a subset (possibly discontinuous) set of values The set of equivalence classes should include both valid and invalid inputs. Equivalence Class Testing 21

1.Volume discount if you buy more than 1000 units. 2.Function that returns the number of days in a month (1-12). 3.If the area code is 800 or 888, there is no charge for the call. 4.Shipping rates differ depending on what country you are in. Equivalence Class Testing Examples 22

Identify equivalence classes to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from Class Problem: Equivalence Class Testing 23

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 24

Boundary Value Testing: Test the values near and at the boundary of an ordered input. For ordered inputs, history suggests that values near the boundary are most error prone. – Programmers may forget to check for a boundary condition – Programmers may code the boundary incorrectly (use < instead of <=) – Special case or different code may be used at the boundary Boundary Value Testing Possible test case 25

For each boundary n, want to have three cases: n – 1, n, n +1 For a range of values from x to y, have at least six cases: x – 1, x, x + 1, y – 1, y, y + 1 Boundary value testing can be combined with equivalence class testing: – Boundaries mark the endpoints of equivalence classes. – Need to test the boundaries of each equivalence class. – Still desirable to test one or more values in the middle of the range. Boundary Value Testing 26

Some values are outside of the range: – May also be part of another equivalence class (worry about redundancy later) – May be invalid – should check to make sure it’s invalid Boundary analysis can be applied to more situations than handling numeric data: – string length – size of arrays (or other data structures) – data type limitations (size of integer) Boundary Value Testing 27

Class Problem: Boundary Value Testing 28 Identify boundary values to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 29

Equivalence class and boundary value testing can also be applied to the output of a function. Outputs can be divided into equivalence classes. – At least one test should be executed for each output equivalence class. – In many cases, but not all, tests created for output are redundant to equivalence class tests created for inputs. Boundaries of an ordered output should also be tested. Output Testing 30

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 31

System Testing focuses on the whole system: – Consists of not only software, but possibly hardware and other components. – Necessary to understand the solution / product as a whole. Encompasses functional and non-functional testing: – Functional: Does the software work? – Non-functional: Performance, usability, security, etc. System Testing 32

Scenario-Based Testing: Develop a scenario or a script that is important to execute. Scenarios tend to mimic realistic use of the system. – How would a normal user use the software? Include invalid input scenarios: – How would a user accidentally or intentionally misuse the system? Scenario-Based Testing 33

Exploratory Testing: Test the program without constraint. – Also called ad-hoc testing. – No script or scenario. Allows the tester to creatively try tests as they explore the application. Exploratory Testing 34

Introduction to Software Testing Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing System Testing Usability Testing Outline 35

What makes a software product usable? Usability Testing 36

Usability Testing: Tests the usability of a software product. Typically done with representative samples of real users to try out the software. Objective criteria: – Time and number of operations to perform a task – Frequency of user error blame user errors on the product! Usability Testing 37

Ask questions with the user in mind: – Does the user interface make sense? – Are the prompts / messages clear? – Is the response appropriate when an invalid input is entered? – Is the program intuitive? May want to have someone outside the class try out your programs. – Don’t let them read the specifications. Basic Usability Features 38