Automated Testing Nathan Weiss April 23, 2007

Overview History of Testing Advantages to Automated Testing Types of Automated Testing Automated Testing Tools Disadvantages to Automated Testing

History of Testing

Four main phases in testing methodology:  Debugging (1950’s)  Exhaustive Testing/Prove it Works (1960’s)  Prove it Does Not Work (1970’s)  Defect Prevention & Testing Process (1980’s)

Debugging No distinction between debugging and testing. Consisted of making code compile and changing code so that it would not crash.

Exhaustive Testing/ Prove it Works Exhaustive testing in terms of code paths and given input.  Found to be impossible Write tests to prove the program works. Testing defined as “what is done to demonstrate correctness of a program.”

Prove it Does Not Work Testing not for proving correctness, but for finding defects. Started testing to see if a program did what it was not suppose to do, not just test to prove that it does what it was suppose to do.

Defect Prevention & Testing Process “Testing” traditionally referred to what was done to the system after working code was produced. Testing moved towards an activity done during a phase, not a phase by itself. Studies show that 50% of bugs are created in requirements and design stages.

Advantages to Automated Testing Test scenarios impossible/unfeasible using manual testing. Lowers testing time and costs in an iterative environment. Remove the human error factor. Introduces stability in regression tests. Increase in testers’ morale.

Types of Automated Testing Split into two categories  Black box tests  White box tests PC applications Embedded Systems

Black Box Automated Testing GUI testing Always been prime candidates for manual testing. Need to automate has created many commercial tools. Two ways to create the tests:  Scripts  Recordings

Black Box Tests - Scripts Less of a testing tool/environment and more of a testing library. Flexible. Easy to run batch scripts to run multiple tests.

Black Box Tests - Recordings Full testing tool/environment. Tester performs a test once while tools is recording his actions. Tester can add ending conditions to test the success of a test. Very useful for testing GUIs of stand alone applications.

White Box Automated Testing Module testing. More APIs then environments. Embedded systems introduce more complexity to white box testing.

White Box Tests – PC Apps Tests usually written in the same language as the system being tested. Tools usually come in the form of an API containing asserts and other testing utility methods. Tests can be performed on any level of abstraction required by the project.

White Box Testing – Embedded Systems Test code usually not stored/run on the same hardware as the system code. Do not have direct access to system code or memory. Requires an interface with an embedded system debugger.

Automated Testing Tools xUnit SilkTest Selenium Custom

xUnit Compilation of popular code driven testing frameworks. Frameworks based on the design by Kent Beck. Originally implemented for SmallTalk as SUnit.

xUnit Partial List of Frameworks  NUnit (.net)  JUnit (java)  CUnit ( C )  CPPUnit (C++)  SUnit (SmallTalk)

xUnit xUnit test execution  Setup  Test body  Teardown

xUnit

SilkTest Created by Borland. Works with:  Web Browsers(IE and Firefox)  Java GUIs .Net GUIs  Win32 GUIs Price : 4-6k

Selenium Testing tool for web applications. Comes in API and tool form. Supports:  Windows: IE, Firefox, Seamonkey, Opera  Mac OS X: Safari, Firefox, Camino, Seamonkey  Linux: Firefox, Konqueror

Selenium Can create tests using the Selenium library in many languages (Java,.NET, Perl, Python and Ruby) Or you can create tests using their Selenium IDE which is implemented as a Firefox extension.  Once created in Firefox, the tests can be run in any of the suported browsers.

Selenium

Custom Write your own custom testing environment or API. Used when you have very specific testing needs. Often used for embedded systems.

Disadvantages to Automated Testing Initial cost. Who tests the testers?

References Borland. (2007). Borland SilkTest: An Automated Regression and Functional Software Testing Tool. Retrieved April 2, 2007, from [2] Dustin, E., Rashka, J., Paul, J. (1999). Automated Software Testing: Introduction, Management, and Performance. Addison-Wesley. [3] Lewis, E., W., (2005). Software Testing and Continuous Quality Improvement. CRC Press LLC. [4] NUnit. (2006). Nunit. Retrieved April 2, 2007, from [5] OpenQA. (2006). OpenQA: Selenium. Retrieved April 2, 2007, from