Testing Tools Prepared by Stephen M. Thebaut, Ph.D. University of Florida Software Testing and Verification Lecture 14

Overview of Topics Why is test automation desirable? Types of test automation –test design tools –test implementation (including “testing frameworks”) tools –test execution and other run-time tools Sources for testing tools One other “Testing Tool:” Test Plans and Planning

Why is test automation desirable? A recent survey result reported in SoftwareTech News † (a DoD publication) confirms that software testing activity consumes a significant fraction of the overall development lifecycle. † October 2007 Vol. 10, Number 3

In the same article, a study undertaken by the US Navy showed that substantial time savings can be achieved via automated testing. Why is test automation desirable? (cont’d)

Some types of testing are not feasible without automation, e.g.: –load (“concurrency”) testing –long term “soak” testing to detect memory leaks/performance degradation More efficient build verification (“smoke”) testing Stand-alone “lights out” testing Increased repeatability Testers can focus on more creative tasks Much higher levels of coverage Other benefits of automated testing

Test Design Tools

Parameter value generators Boundary value, equivalence class, and special value generators Random data generation based on probabilistic CFGs (Context-Free Grammars) Fault tree coverage Cause-Effect analysis Database test data generators (for performance, load, and usability test) (cont’d)

Test Design Tools (cont’d) Statistical test generators (for reliability, availability, and performance testing) Structural test case design tools: –utilize code instrumentation (see Run- Time Tools) –source code specific –may incorporate symbolic execution capabilities to aid in path sensitization

Test Implementation Tools

Test Case Scripting Languages: –test scripts automate the steps involved in carrying out tests –suitable for tests that will be repeated (i.e., in support of regression testing) –scripts are maintained in a change-controlled Test Database (cont’d)

Test Implementation Tools (cont’d) Test Harness (scaffolding) Generators: –automates development of test drivers, stubs, and oracles (“assertions”) –often coupled with a test scripting capability (= “testing frameworks”) –useful for module and integration test

Run-Time Tools

Code Instrumentation: –inserts additional code to monitor cumulative logic/dataflow coverage, support trace and debug functions, and detect run-time errors. –aids in designing test cases to meet white-box coverage goals or to debug code. –alters execution performance and may mask or introduce timing errors. –many commercial (source code specific) tools available. –among the oldest of testing tool types. (cont’d)

Let’s Pause for a Moment… Imagine that it’s summertime and that a 3- day weekend is just starting… Wouldn’t it be great to just grab a fishin’ pole and head on out to the lake!

Let’s Pause for a Moment… (Take 2) There’s NOTHING like Paris in the the spring!

Run-Time Tools (cont’d) Data / Output Comparators –used to automatically compare output or data files for discrepancies –effective in dealing with the “Paris in the the spring” problem † –most are not “smart enough” to ignore “don’t care” data fields. –useful for regression testing when combined with a keystroke recorder and playback tool… † Also know as inattentional or perceptual blindness. (cont’d)

Run-Time Tools (cont’d) Keystroke Recorder and Playback Tools –records and plays-back keystrokes, pointer movement, and mouse clicks –useful for regression testing when combined with a data/output comparator

Sources for Tools An excellent source of information about testing tools is the WWW. Use your favorite search engine to find sites associated with “software testing tools” (about 94 million Google hits at last count). Several QA and testing related magazines, newsletters, and trade journals are available which publish tool reviews and comparisons. (cont’d)

Sources for Tools (cont’d) Relevant links can also be found at the websites of several QA and testing related organizations and societies, e.g., –SEI –IEEE –ACM –American Society for Quality –Society for Software Quality –Quality Assurance Institute –European Organization for Quality (cont’d)

Sources for Tools (cont’d) Some useful testing tools-related links: – – –

Test Plans and Planning Pezze and Young, Software Testing and Analysis, Wiley, 2008

Motivation Planning – just as important in testing as in any other engineering activity. Allows for the effective use of limited resources and management of activities. In a nutshell, it involves a disciplined, timely consideration of: –What needs to be done, –When and How it will be done, and –What resources (including time and effort) will be required.

The Master Test Plan Also called a Comprehensive Test Plan. Must be compatible with overall project plan. Developed early – during project planning and requirements stages. Defines requirements for lower level plans: –Unit Test Plan –Component Test Plan –Product Test Plan –System Test Plan

Master Test Plan Check-List Defines test stages (e.g., unit, integration, system) schedules, and responsibilities. Establishes objectives, strategies, and procedures. Identifies required tools, facilities, and test libraries. Sets criteria for test completion and success. Defines the integration test strategy and iden- tifies all scaffolding required with respon- sibilities and schedules for its production. (cont’d)

Master Test Plan Check-List (cont’d) Defines test coverage goals and methods for tracking progress. Establishes a regression test policy. Specifies required hardware and software, particularly non-standard or special purpose elements. Identifies estimated resource requirements including machine time, disk storage, physical / logical device access, etc. Identifies training required. Specifies test entry and exit criteria.

Typical Test Entry Criteria Test plan is complete and has been reviewed Design and implementation of test cases reviewed and all problems fixed Drivers, stubs, oracles in place Test environment (hardware and software) is ready

Typical Test Exit Criteria All problems discovered have been recorded and fixed All fixes have been re-tested Regression test(s) complete All test cases and associated documentation updated as necessary

IEEE Test Plan Template 1.Introduction and references 2.Test items (versions, media, references, etc.) 3.Features to be tested 4.Features NOT to be tested (and reasons) 5.Testing strategies and approach 6.Dependencies 7.Test case success/failure criteria 8.Pass/fail criteria for the complete test 9.Test entry/exit criteria (cont’d)

IEEE Test Plan Template (cont’d) 10.Test deliverables/status communication vehicles 11.Testing tasks 12.Hardware and software requirements 13.Problem determination and correction responsibilities 14.Staffing and training needs/assignments 15.Test schedules 16.Risks and contingencies 17.Approvals

Test Planning Summary Test planning allows for the effective use of limited resources and the effective manage- ment of test activities. A plan is only as good as the planning that goes into it!

Testing Tools Prepared by Stephen M. Thebaut, Ph.D. University of Florida Software Testing and Verification Lecture 14