1 CEN 4072 Software Testing PPT3: Making the program fail

Two views of testing TESTING FOR VALIDATION Testing for unknown failures Goal is to uncover as many problems as possible TESTING FOR DEBUGGING Testing for known failures Goal is to reproduce a specific problem Used in many stages of debugging 2

Tests in debugging Tests are used for many stages in debugging. They are used for the following reasons: 1. To reproduce the problem 2. To simplify the problem 3. To observe the run 4. To verify successful fixes (Validation) 5. To ensure before each new release that similar problems will not occur in the future. (Known as Regression Testing) 3

Make the code fail There are several tests that can isolate failure causes automatically: 1. Failure-inducing input 2. Failure-inducing code changes 3. Failure-inducing thread schedules 4. Failure-inducing states (These are defined in depth in chapters 13 and 14) 4

Challenges of making the program fail When simulating program failures there are a few concepts to consider: 1. Synchronization: The simulation must follow the steps that cause the program to crash in exactly the same order that caused the crash the first time in order to monitor the outputs at every step. 2. Abstraction: There are different levels of abstraction that can be simulated in order to narrow down the possible causes of the failures. The higher the abstraction level, the easier it is to simulate interaction, however; this risks abstracting away the details that caused the failure. 3. Portability: Many programs have high portability, meaning they can run in a wide variety of environments. Some failures occur in some environments and not others, in which case, the tester must reproduce the same environment in order to reproduce the same failure. The goal is to fix all failures for all potential environments. 5

Interaction layers Programs can be broken down into three layers for testing: 1. Presentation layer: Handles the interaction with the user or the environment of the program. 2. Functionality layer: Includes only the functionality of the program. 3. Unit layer: Splits the functionality across multiple units that cooperate to produce a greater whole. 6

Presentation layer Low level: At the lowest abstraction level, user input can be captured and replayed as a stream of mouse and keyboard events. This can be recorded as a script, which includes exactly the input produced by the user and the time between each event. This makes it possible to simulate the user input in the exact way that initially caused the problem. System level: At the system level the goal is to simulate the hardware of a real machine using virtual machines in order to simulate things such as hardware defects. The scripts control the hardware directly. High level: At higher abstraction levels, the application is controlled using graphical user controls which are accessed via the script. 7

8 Low level interaction script Mouse and keyboard events

9 System level interaction script Simulating hardware, notice the HDL style syntax

10 Higher level interaction script Commands replace events normally controlled by the User Interface.

Scripting languages VBSCRIPT (Visual Basic Script): Scripting language widely used for Windows and Internet Explorer. PERL: General purpose scripting language. Runs on over 100 platforms and is “suitable for rapid prototyping and large scale development projects.” PYTHON: General purpose scripting language similar to PERL except that it emphasizes data structure design and object-oriented design over common application-oriented tasks. APPLESCRIPT: Scripting language widely used for Macintosh as well as Safari and Mozilla web browsers. 11

Functionality layer Programs may provide an interface that is designed for automation. They are generally designed for interaction with technical systems such as programming languages or scripting languages. For example an APPLESCRIPT program that uses scripting capabilities on the Safari web browser which uses such commands as: “set the URL of the front document”. This command works regardless of the user interface and is unaffected by changes to the user interface. Not every feature of Safari is scriptable in this way (ex: the print function) therefore; some functions still require testing at the presentation layer. The advantage of testing at the functionality layer is that the results can be easily accessed and evaluated. The difficulty with testing at the functionality layer is that it requires separation between the functionality and presentation layers, which is not always available, especially in older programs. 12

Unit tests Programs can be broken down into many units such as subprograms, functions, libraries, modules, classes, objects etc. depending on the programming language used. Unit tests are designed to execute just one of these units at a time rather than the program as a whole. A suite is commonly set up to run multiple unit tests at once and a main function is commonly included to run the tests and get the results. 13

Running unit tests Testing frameworks can be used to collect data from a number of individual unit tests automatically. When a single unit test executes the framework does three things: 1.Sets up an environment with only what is needed for the unit to execute. This may include other units the unit must interact with. 2.Tests the unit using enough test cases to cover every possible behavior of the unit, verifying that the results are as expected. 3.Closes the environment and goes back to the state that was initially encountered. 14

Unit test with JUNIT (JUNIT is a unit test framework in Java) Set up 3. Close 2. Tests To Run The “URLTest” class is a subclass of “TestCase”

Unit test with JUNIT (continued) 16 Set up a suite that runs all tests Main method starts JUNIT graphical Interface.

Unit test with PyUnit PyUnit is a unit test framework in Python. This example was found at: m/2011/11/getting-started- with-python-unit-testing- pyunit/ Notice that it includes the same basic principles as the JUNIT test. 17 Set up Close Tests to run Can add tests to a suite to run multiple tests Main function

Breaking circular dependenc y As shown in Example 3.11 the print_to_file function calls the confirm_loss function to first ask the user if overwriting is ok. Some programs have functionality that depends on presentation. To illustrate this the following is an example of this in which a function to print a web page to a file is called that asks the user if they want to overwrite the file if it already exists. 18 Circular dependence:

Breaking circular dependenc y (Continued) As shown in Figure 3.4 The presentation (user prints) invokes print_to_file which is part of the core functionality. The functionality (print function runs) invokes confirm_loss which is part of the user presentation. The consequence of circular dependence is that the presentation and functionality can’t be separated, which makes testing more difficult. 19 Circular dependence (continued):

Breaking circular dependenc y (continued) A good solution to this is to use inheritance to make the presentation an interface, or more specifically; an abstract super class that is instantiated in subclasses. In the previous case, we could make an automated presentation subclass that includes a confirm_loss function that automatically returns true, breaking the dependence on the user/presentation. 20

Design for debugging The main concept of designing for debugging is to reduce the number of dependencies by depending on abstractions rather than details. The goal is to separate as many dependencies as possible. This includes presentation and functionality at the application level as shown in the previous example. Remember that to do this we used inheritance to create a super class with subclasses that offer alternative automated versions of functions. One of the most popular examples of this technique is the model-view- controller architecture pattern shown in the next couple slides. 21

Model-View-Controller (MVC) The Model-View-Controller includes: 1.A model that holds and operates on core data. 2.A number of observers that register or attach to the model to get notified whenever core data changes. The observers are then divided into two subclasses: 1.A view class responsible for displaying the core data in any way needed. 2.A controller class to handle inputs and invokes the services of the model. 22

MVC (Continued) 23 The Model-View-Controller shown in figure 3.7 is quite beneficial for testing and debugging. Controllers can be added to automate execution of the services provided in the model. Thus avoiding dependencies between presentation and functionality by bypassing the need for user interaction.

General design rules High cohesion: The principle of grouping together parts of a system that work on common data into a single unit. For example object-oriented design groups data and functions into classes. Low coupling: The principle of reducing dependences. If two units do not operate on the same data, then they should exchange as little information as possible. For example the confirm_loss function from the previous slides would violate this principle by coupling presentation and functionality more than necessary. 24

Tools for automated testing JUNIT: Unit test tools ANDROID: A scripting language no longer available for download APPLESCRIPTS: Apple scripting language VBSCRIPT: Visual Basic scripting language FAUMachine: Virtual machines publically available which allows you to control the entire virtual machine via script. Virtual PC: Microsoft Virtual PC for various operating systems

References unit-testing-pyunit/ unit-testing-pyunit/ Zeller, A. (2009). Why Programs Fail: A Guide to Systematic Debugging. (2 nd ed.). Burlington, MA: Morgan Kauffman Publishers. 26