1 Introduction to Software Testing

Reading Assignment P. Ammann and J. Offutt “Introduction to Software Testing” ◦ Chapter 1 2

Outline What is Software Testing? What is the cost of software failures? Observability and controllability Black-box & White-box Testing Changing notions of testing 3

4 What is Software Testing? The process of finding evidence of defects in software systems. ◦ Establishing confidence that a program does what it is supposed to do. 1.Software testing is not debugging. 2.Software testing is not quality assurance

5 Software Testing Vs Quality Assurance (QA) Testing is necessary but not enough for QA process. ◦ Testing contributes to improve quality by helping to identify problems. QA sets standards that project members (including testers) should follow in order to build a better software.

6 What is Software Testing? Correctness of software with respect to requirements or intent; Performance of software under various conditions; Robustness of software, its ability to handle erroneous input and unanticipated conditions; Installation and other facets of a software release.

7 Test Engineer & Test Managers Test Engineer : An IT professional who is in charge of one or more technical test activities ◦ designing test inputs ◦ producing test values ◦ running test scripts ◦ analyzing results ◦ reporting results to developers and managers

8 Test Engineer & Test Managers Test Manager : In charge of one or more test engineers ◦ sets test policies and processes ◦ interacts with other managers on the project ◦ otherwise helps the engineers do their work

9 Test Engineer Activities Test Designs Output Executable Tests ComputerEvaluate P Test Manager Test Engineer Test Engineer design instantiate execute

10 Basic Definitions Software Fault ◦ A static defect in the software. Software Error ◦ An incorrect internal state that is the manifestation of some fault. Software Failure ◦ External, incorrect behavior with respect to the requirements or other description of the expected behavior. Faults in software are design mistakes and will always exist

Example public static int numZero (int[] x) { // Effects: if x == null throw NullPointerException // else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) { count++; } return count; } Fault numZero([0,5,0])=1 Failure What is the state at the if statement on the very first iteration of the loop (x,count,i,PC)? ([0,5,0], 0, 1, “if”) Error ([4,5,0], 0, 1, “if”) Error that does not result in a failure

12 Important Considerations Detect system failures by choosing test inputs carefully. Determine the faults leading to the failures detected. Repair the faults leading to the failures detected; and Re-test the module/system.

13 Why do Failure Occur?

14 Root Causes of Failures Inaccurate understanding of end user requirements. ◦ Inability to deal with changing requirements. Late discovery of serious project flaws. ◦ For example, modules that do not fit together. Untrustworthy build & release process. ◦ Implementation team’s chaos.

15 Failure Costs Disney’s Lion King CD

16 Observability and Controllability Software Observability – How easy it is to observe the behavior of a program in terms of its ◦ Outputs, effects on the environment and ◦ Other hardware and software components. Software that affects hardware devices, databases have Low observability.

17 Observability and Controllability Software Controllability – How easy it is to provide a program with the needed inputs, in terms of values, operations and behavior. ◦ Easy to control software with inputs from keyboards. ◦ Input from hardware sensors is harder.

18 Inputs to affect controllability and observability Prefix Values: ◦ Any inputs necessary to put the software into the appropriate state to receive the test case values. Post-Fix Values: ◦ Any inputs that need to be sent to the software after the test cases values.

19 Inputs to affect controllability and observability Two Types of postfix values ◦ Verification Values: Values necessary to see the results of the test case values. ◦ Exit Commands: Values needed to terminate the program or otherwise return it to a stable state. Execution Test Script ◦ A test case that is prepared in a form to be executed automatically on the test software and produce a report.

20 Top-Down and Bottom-up Testing Top-Down Testing: Test the main procedure, ◦ Then go down through procedures its calls, and so on. Bottom-up Testing: Test the leaves in the tree (procedures that make no calls), and move up to the root. ◦ Each procedure is not tested until all of its children have been tested.

21 Black-Box Testing requirements events input output

22 Black-box Testing Test cases are derived from formal specification of the system. Test case selection can be done without any reference to the program design or code. Only tests the functionality and features of the program. ◦ Not the internal operation.

23 Black-box Testing Advantages ◦ Test case selection is done before the implementation of a program. ◦ Help in getting the design and coding correct with respect to the specification.

24 White-Box Testing

25 White-Box Testing Test cases are derived from the internal design specification or actual code for the program. Advantages ◦ Tests the internal details of the code; ◦ Checks all paths that a program can execute. Limitations ◦ Wait until after designing and coding the program under test in order to select test cases.

26 Changing Notions of Testing Traditional View of testing is of testing at specific software development phases ◦ E.g. unit, module, integration etc. New view is in terms of structures and criteria. ◦ Graphs, logical expressions, syntax, input spaces.

27 Traditional Testing at different levels Class A method mA1() method mA2() Class B method mB1() method mB2() main Class P n Acceptance testing: Is the software acceptable to the user? n Integration testing: Test how modules interact with each other n System testing: Test the overall functionality of the system n Module testing: Test each class, file, module or component n Unit testing: Test each unit (method) individually

28 New Approach: Test Coverage Criteria Define a model of the software, then find ways to cover it. Test requirements: Specific things that must be satisfied or covered during testing. Test criteria: A collection of rules and a process that define test requirements.

29 Key Points Software testing is the process of discovering evidence of defects and failures in software systems. Test early, test often, test enough. Testers should have good understanding of the development process, product.

30 Key Points Testers should have good understanding of the development process, product. Inputs to affect controllability and observability New testing view is in terms of structures and criteria. ◦ Graphs, logical expressions, syntax, input spaces.