Presentation on theme: "14.1 Introduction Using test tools makes only sense if it is profitable in terms of time, money, or quality. For successful introduction of test automation."— Presentation transcript:
14.1 Introduction Using test tools makes only sense if it is profitable in terms of time, money, or quality. For successful introduction of test automation there must be a lifecycle and knowledge of all the test activities. The test process must be repeatable. Up to date information about available tools can be found through: conferences and exhibitions; vendor exhibitions for free trial versions of tools; research reports such as the CAST report (Graham et al., 1996); the internet – www.ovum.com (ovum reports) – www.testing.com (Brian Marick ’ s Testing Foundations) – www.soft.com/Institute/HotList (Software Research Incorporated) – most tool vendors have a website with detailed information about their (commercial) products.
14.2 Categorization of test tools Tools are available for every phase in the test lifecycle. Figure 14.1 shows how tools can be categorized according to where they are applied in the testing lifecycle (see Chapter 6). These tools are explained in more detail in the following sections – the list presented is not meant to be exhaustive.
14.2.1 Planning and control Most tools for the planning and control phase are project management based. These tools are not developed especially for testing, except for test and defect management tools. The following tools support planning and control: defect management tool; test management tool; configuration management tool; scheduling and progress monitoring tool.
188.8.131.52 Defect management tool Defects detected during the test process must be collated in an orderly way. For a small project, a simple file system with a few control procedures is sufficient. More complex projects need at least a database with the possibility of generating progress reports showing.
184.108.40.206 Test management tool The test management tools provided by tool vendors offer only the functionality to store test cases, scripts, and scenarios and sometimes integrate defect management. This type of test management tool is not very useful for planning and control activities. Tools with the ability to link system requirements to test cases are now available. These tools afford the ability to keep track of the coverage of test cases in relation to the system requirements. In addition, they become very useful if system requirements are changed or might change – the test manager is able to show the impact of these changes and this may give cause to block the changes.
220.127.116.11 Configuration management tool All testware has to be stored in a proper way. After quality checks, the testware should be frozen. The next step is to make the testware configuration items and store them in a configuration management tool. A configuration management tool provides the functionality to keep track of the configuration items and their changes.
18.104.22.168 Scheduling and progress monitoring tool For scheduling and progress monitoring, many different tools are available which are specially made for project management. Scheduling and progress monitoring are combined in most tools. These kinds of tools are very useful for a test manager – combined with information from the defect and test management systems it is possible to keep track of progress in very detailed manner if necessary.
14.2.2 Preparation phase-22.214.171.124 CASE tool analyzer CASE tools, such as tools for UML, are able to do consistency checks. These can be used to check whether or not the design has omissions and inconsistencies. In this way, a CASE tool can be used to support the “ testability review of the test basis ” activity – provided that the tool was used to design the system of course.
14.2.2 Preparation phase-126.96.36.199 Complexity analyzer The degree of complexity is an indicator of the chance of errors occurring and also of the number of test cases needed to test it thoroughly.
14.2.3 Specification phase-188.8.131.52 Test case generator Test design is a labor intensive job and using tools would make it simpler – unfortunately, not many are available. To use a test design tool, a formal language (mainly mathematically-based) has to be used for the system requirement specification.
14.2.4 Execution phase Many types of tools are available for the execution phase. Many tool vendors have generic and highly specialized tools in their portfolio. The following types of tools are described in this section: test data generator record and playback tool load and stress test tool simulator stubs and drivers debugger static source code analyzer error detection tool performance analyzer code coverage analyzer thread and event analyzer threat detection tool.
184.108.40.206 Test data generator This generates a large amount of different input within a pre-specified input domain. The input is used to test if the system is able to handle the data within the input domain.
220.127.116.11 Record and playback tool This tool is very helpful in building up a regression test. Much of the test execution can then be done with the tool and the tester can mainly focus on new or changed functionality. Building up these tests is very labor intensive and there is always a maintenance problem. To make this type of tool profitable in terms of time, money, or quality, delicate organizational and technical decisions have to be made.
18.104.22.168 Load and stress test tool Systems that have to function under different load conditions have to be tested for their performance. Load and stress tools have the ability to simulate different load conditions. Sometimes, one may be interested only in the performance under load or, sometimes more interesting, the functional behavior of the system under load or stress. The latter has to be tested in combination with a functional test execution tool. Another reason to use this type of tool is to speed up the reliability test of a system. A straightforward reliability test takes at least as long as the mean time between failures, which often is in the order of months. The test duration can be reduced to a matter of days, by using load and stress tools that can simulate the same amount of use in a much shorter period.
22.214.171.124 Simulator A simulator is used to test a system under controlled conditions. The simulator is used to simulate the environment of the system, or to simulate other systems which may be connected to the system under test. A simulator is also used to test systems under conditions that are too hazardous to test in real life. For example the control system of a nuclear power plant.
126.96.36.199 Stubs and drivers Interfaces between two system parts can only be tested if both system parts are available. This can have serious consequences for the testing time. To avoid this and to start testing a system part as early as possible, stubs and drivers are used. Standardization and the use of a testbed architecture (see Figure 14.2) can greatly improve the effective use of stubs and drivers. The testbed provides a standard (programmable) interface for the tester to construct and execute test cases. Each separate unit to be tested must have a stub and driver built for it with a specific interface to that unit but with a standardized interface to the testbed. Techniques for test automation, such as data-driven testing (see section 15.2.1), can be applied effectively. Such a testbed architecture facilitates the testing of any unit, the reuse of such tests during integration testing, and large-scale automation of low-level testing.
188.8.131.52 Debugger A debugger can detect syntactic failures. Debuggers have the ability to run the application stepwise and to set, monitor, and manipulate variables. Debuggers are standard elements of developer tools.
184.108.40.206 Static source code analyzer Coding standards are often used during projects. These standards define, for instance, the layout of the code, that pre- and post-conditions should be described, and that comments should be used to explain complicated structures. Additional standards which forbid certain error-prone, confusing or exotic structures, or forbid high complexity, can be used to enforce a certain stability on the code beforehand. These standards can be forced and checked by static analyzers. These already contain a set of coding standards which can be customized. They analyze the software and show the code which violates the standards. Be aware that coding standards are language dependent.
220.127.116.11 Performance analyzer This type of tool can show resource usage at algorithm level while load and stress tools are used to do performance checks at system level. The tools can show the execution time of algorithms, memory use, and CPU-capacity. All these figures give the opportunity to optimize algorithms in terms of these parameters.
18.104.22.168 Code coverage analyzer This type of tool shows the coverage of test cases. It monitors the lines of code triggered by the execution of test cases. The lines of code are a metric for the coverage of the test cases. Be aware that code coverage is not a good metric for the quality of the test set.
22.214.171.124 Thread and event analyzer This type of tool supports the detection of run-time multi-threading problems that can degrade Java performance and reliability. It helps to detect the causes of thread starvation or thrashing, deadlocks, unsynchronized access to data, and thread leaks.
13.3.2 Hardware/software integration tests In the hardware/software integration (HW/SW/I) test the test object is a hardware part on which the integrated software is loaded. The software is incorporated in the hardware in memory, usually (E)EPROM. The piece of hardware can be an experimental configuration, for instance a hard-wired circuit board containing several components including the memory. The goal of the HW/SW/I test is to verify the correct execution of the embedded software on the target processor in co- operation with surrounding hardware. Because the behavior of the hardware is an essential part of this test, it is often referred to as “ hardware-in-the-loop ”.
13.3.2 Hardware/software integration tests The test environment for the HW/SW/I test will have to interface with the hardware. Depending on the test object and its degree of completeness, the following possibilities exist: offering input stimuli with signal generators; output monitoring with oscilloscopes or a logic analyzer, combined with data storage devices; in-circuit test equipment to monitor system behavior on points other than the outputs; simulation of the environment of the test object (the “ plant ” ) in a real- time simulator.
126.96.36.199 Threat detection tool This type of tool can detect possible threats that can lead to malfunctioning. The type of threats are memory leaks, null pointer dereference, bad deallocation, out of bounds array access, uninitialized variable, or possible division by zero.
14.2.5 Completion phase Basically, the same tools as for the planning and control phase are used for this phase – although the focus here is on archiving and extracting conclusions rather than planning and control. Configuration management tools are used for archiving testware and infrastructure, and tool descriptions, calibration reports, etc. Scheduling and monitoring tools are used to derive project metrics for evaluation reports. Defect management tools are used to derive product metrics and sometimes provide information for project metrics.