1. Practical IT Research that Drives Measurable Results Assess the Appropriateness of Automated Functional Testing

2. Introduction IT shops who have a high degree of customized applications may be considering moving to tools for automating functional testing. The benefits of functional test automation are heavily weighted by the overhead cost of implementation. The high cost and labor commitment associated with automating functional testing requires that this investment be critically evaluated. Automated Functional Testing Overview Assess Organizational Appropriateness Make the Business Case

3. Executive Summary Avoid adding to your shelf-ware collection; separate hope from hype. Automation has high returns for the right shops, but the returns promised by vendors will only be realized by a select few. Automation can improve quality, but don’t do it to reduce costs. Tools enable testing that time and budget constraints would not permit without automation, but do not eliminate the need for testers or reduce development costs. Low maturity shops will not realize value. Tools do not replace process. Those who are most likely to benefit are shops with a highly mature QA process in conventional Waterfall development, or developers using Agile Test Driven Development (TDD) which relies heavily on the testing process for development. Look for long-term gain. Don’t expect returns in the first major project. Licensing costs, training and configuration mean break-even at best on initial six months. Enterprises who will benefit from automation will do so over the course of a number of years. Automate Functional Tests appropriately. Even shops that are automation appropriate should be careful how they use automation to avoid wasted scripting time and shaping development around testing tools instead of using them to enhance the QA process.

4. Automated Functional Testing Overview Assess Organizational Appropriateness Make the Business Case

5. Unit Testing Functional Testing Regression Testing System Integration Testing Acceptance Testing “ Automated functional testing” is a misnomer: tools not only test to ensure specific functions in the application work, but for functionality in a variety of tests across the SDLC. Testing Types Explained Unit Testing. Coders test their individual builds to make sure they work. Functional Testing. QA tests a feature within one software application. The feature tested will likely call to many parts of the code that exist in different modules produced by different programmers. Functional Testing covers: Regression Testing. Test functionality across the application each time a change is made to make sure changes did not adversely affect other components of the application. System Integration Testing. Tests for functionality of the application once it’s been integrated with other applications to ensure the application functions within the environment, and that the functionality of other applications is not affected by changes. Acceptance Testing. Validates with users that the app does what it was supposed to do according to the specification requirements. Performance Testing. Looks beyond functionality and examines how well the application performs under real life, load-bearing scenarios. Automated functional testing tools test for functionality beyond the functions

6. Steps in the Functional Testing ProcessHow Automation Helps Develop Testing Goals. For many projects, it is unrealistic to test every functional aspect of the software. Instead, project-appropriate functional testing goals are defined in terms of time, scope, and cost that will validate the requirements. Once a function is scripted it takes little effort to execute, making it easier to test more functions than would normally be called for in the test plan. Ideally, this leads to capturing more defects early in the development process. Write Test Cases. Functional tests should be written separately from the functionality implementation. In most cases, test cases and scripts are developed by the testing team based on requirements while development is occurring. Most commercial functional testing tools can automate the scripting of testing with a record and playback feature that facilitates the programming required for automation. Execute Test Cases. Using the determined testing method, execute each test against its implementation and record the result. Recognize that if the initialization criteria cannot be met, then the test cannot be executed; it will fail as designed. The primary function of the automated testing tool is to execute tests and record the results. Automation means each execution is identical, taking out potential for human error. Report, Review and Re-execute. Report the issues to the development team. Ensure the severity of the failure is included. Reporting is simplified by automation tools and the mundane, granular tasks of recording variances is addressed by the automation tools Functional testing verifies the ultimate testing question… “Does it work?” To learn more about functional testing refer to the Info-Tech research note, “Functional Testing Ensures Correct Requirements Implementation.”Functional Testing Ensures Correct Requirements Implementation Info-Tech Research Tools

7. Automated functional testing is not a replacement for QA analysts, but a solution for executing mundane regression tests Repetitive mundane tasks. As software is created, it goes through many versions, each of which must be tested. By creating test scripts once, automated testing allows reuse for future versions. Regression testing. Each time a change is made to a system, testers using a manual approach have to go through each function and make sure it does not affect any other part of the system. Automated regression testing uses test cases to do this more quickly. Granular function testing. Automated testing tools can provide more accuracy in testing. They can pick up granular differences that manual testing may not isolate. For instance, a slight color difference on a GUI may not be seen by the naked eye during manual testing, but would be caught during an automated test Multiple combinations. In more complex environments where many scenarios must be examined, automated testing can rapidly run many test case combinations using a single script. To learn more about the benefits of functional test automation refer to the Info-Tech research note, “Automate Software Testing for Efficiency Gains.”Automate Software Testing for Efficiency Gains Info-Tech Research Tools Automation Case Study A development group with a nine person QA team supports business-critical client-facing applications and uses IBM Rational Tester. They did not view automation as a cure all, but agreed that there was an improvement to quality. Experience with automation taught them to: – Only do automation on very basic sanity testing which does not require heavy scripting. – Run more complex functional cases manually. “ Think through your test process and your test cases, and look for ones that should be automated. And just as importantly, look for ones that don’t make sense to automate because they’re too dynamic. It’s just faster to continue to do it in a manual way.” ~ IT Manager, Professional Services Why Automate Functional Testing?

8. Functional testing automation allows for test execution frequency, which could not be done manually Testing budgets and timelines are fixed. Although automation won’t replace QA analysts, it allows for additional test execution which would otherwise be out of scope. The ability to increase functional testing within the budget and time constraints improves product quality. Benefits of Functional Testing Automation Repeatability. Automated tools allow testers to save scripts and test cases in an organized manner. Although planned manual testing can offer similar benefits, often human error can occur when the tester has forgotten which code, or how a piece of code, was tested; thus an automated tool can decrease the amount of rework. Reusability. Scripts that have been saved can be reused for future testing. This is a real boon to regression testing and also reduces test script development time for new projects. More tests means early defect detection. By starting tests early, bugs and defects can be caught, fixed, and retested in less time. This also lowers testing costs as bugs detected in later stages of the SDLC tend to cost more to fix. Scheduled testing. An automated solution can be set to run at a scheduled time enabling testing during non-work hours and overnight. Increased productivity. Automated testing frees up the tester’s time, allows testers to run more complex tests, spend more time developing tests, and gets the software to production faster.

9. …the use of an automated test tool to carry out regression testing will outweigh the initial setup costs after three or four re-tests. An investment has to be made into setting up a robust testing regime, and use of such tools demands solid knowledge of how best to make use of them.” IT Manager, Professional Service Implementers benefit most from improved quality, as well as planning & execution of scripts n= 30 “

10. Scripted high-level manual tests usually aren't very good tests to start with. At best, they're usually checks of business rules which would be more effectively and efficiently automated at lower levels. Then, to compound the problem, the automaters come along and turn those rotten manual tests into rotten automated tests, which have the virtue of at least being over with quickly. “ - Michael Bolton, Testing Expert We bought the tool and immediately sat down and thought, ‘Okay, great. Now that we have it, we can take all of these manual test cases and turn them into automated test cases.’ And quickly realized, ‘Holy cow, there’s a lot of work there.’” -IT Manager, Professional Services ““ Increases cost of maintenance. The commitment to functional automation tools is a commitment to keeping test scripts up to date. The more changes to an application the higher the maintenance cost of the tool. Complicate the QA process. Tools won’t improve a poor testing process – though they may compound the negative effects of a poor process by creating the opportunity to create more tests and generate more reports for analysts to look at. Automation shapes testing. When a test doesn’t lend itself well to automated testing, developers may reprogram software so it will pass automated tests. So, now you have software designed to pass automated functional tests instead of satisfying a business demand. In a worst case scenario, automation creates additional maintenance work without improving the QA process

11. The investment in automation tool licenses does not magically reduce testing costs. Functional Testing Costs Components Impact of Automation Staff to develop test cases Converting cases into scripts could wind up costing as much as three times the licensing costs, depending on how the tool is used. Open Source Tools will either require that staff have a development background or that external support is contracted, further raising the cost of implementation. Staff hours to run tests Staff will not be replaced by a tool, but tests run overnight can create greater efficiency leading to reduced overtime. Cost of licensing tool Commercial tools can cost anywhere from $3,000 - $8000 per named user, and $45,000 per user for full suite licensing (including test management and production testing software). Cost of fixing bugs Bugs found in the production stage are 100 times more costly to repair than in the development phase. Training Training on tools is a necessary cost of automation. HP training for 8 QA Analysts was quoted as $4,000. A large development shop running Quality Center gives new employees four weeks of training on tools. The cost increase from licensing & scripting will only be offset if additional testing catches more defects in development

12. Info-Tech Helps Professionals To: Sign up for free trial membership to get practical Solutions for your IT challenges “Info-Tech helps me to be proactive instead of reactive - a cardinal rule in stable and leading edge IT environment.” - ARCS Commercial Mortgage Co., LP