4 Myth #1 - Test Automation is simple, that every tester can do it This myth is promoted by the tool sales people. They are trying to promote the following test automation process:Record the scriptEnhance the script by adding functions and data drivingRun the scriptsReport resultsUnder the influence of this myth QA manager can proudly report: All our testers are developing test automation.
5 Reality - Test automation is a software development task Automation should be designed, developed and testedYou need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development.Test automation standards should be developedAutomated test components are assets that should be treated like application source code
6 Myth #2 – Commercial test tools are expensive Under the influence of this myth some companies, especially the small ones:Try to develop their own test automation toolsUse scripting languages like Perl and RubyUse shareware test toolsDo not consider test automation at all
7 Reality – Commercial tools are cheap Per seat license for most expensive automation tool is $8KThis tool will be used for 5 years.Maintenance/Support fees are 20% of tool cost or $1,800 per yearThe cost of this tool is $8K/5+$1,800 = $3,100 per yearThe automation developer cost with overhead is $100K per yearThe cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant
8 Commercial Tool Benefits: Customer support. Many of the open source tools come and go with little to no supportMost commercial tools are constantly being updated as technologies changeMost commercial tools usually have more functionality (QTP can test various GUI applications: Web, .Net, Java, VB, C/C++, PowerBuilder, etc. vs. WATIR – Web only)Commercial tools usually have a large community of users, which translates into better availability of qualified resourcesCommercial tools require less advanced programmingMore test automation frameworks are available for commercial toolsCommercial tools are integrated with Test Management tools which makes reporting and execution much simpler.
10 Dice search results across US (30 days) ToolSearch stringMatchesQTPQtp OR quicktest OR "quick test“613Functional Tester"rational robot" OR "functional tester”118SilkTestsilktest OR "silk test”86TestCompletetestcomplete OR "test complete”26TestPartnertestpartner OR "test partner“19VSTEVSTE AND test45WATIRwatirSelenium108
15 Classic ROI Calculation ROI = BENEFIT/COSTAutomation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution CostManual Testing Cost = Development Cost + Maintenance Cost + Execution CostROI = (Manual Testing Cost - Automation Cost)/Automation CostLooks right, isn’t it?
16 Problems with Classic ROI Calculation You can’t compare Automated Testing and Manual Testing. There are not the same and they provide different information about the AUT.You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
17 Automation Real ROIROI value IS NOT the value of Automation vs. Cost of executing these tests manuallyAutomation ROI value IS the benefit of this type of testing, and it can be:Reducing Time to MarketIncreased Test Efficiency (Productivity)Increased Test Effectiveness
18 Reduced Time to Market Can get a greater market share Makes people available to work on other projectsHigher margins, if no competitive products are currently available
19 Productivity and Effectiveness More testing gets done faster, increasing the odds of finding defectsDefects found early have better chances to be fixedManual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output.About 7% of bug fixes create new bugs, sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happened when testing done manually.
20 ROI summaryEach project requires different types of automation - there is often no easy formula available to calculate ROIPerforming ROI calculation can help to determine upfront what type of automation, what level of skills, what tools will be required.
22 Test Automation Framework A Test Automation Framework is a set of assumptions, concepts and tools that provide support for Automated Software Testing.Correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.
23 Types of Test Automation Frameworks ModularData-DrivenKeyword–DrivenModel-Based
24 Modular frameworkThe Modular framework is the natural progression from Record-and-PlaybackThe modular framework seeks to minimize this repetition of code by grouping similar actions into “modules” (e.g.: login)Test Data is in a scriptThis is what tool sales people are promoting.
25 Data-Driven/Keyword Driven These frameworks are similar in that the data is separated from the test scriptThe script is just a "driver" or delivery mechanism for the data.The difference:In keyword-driven testing, the navigation data and test data are contained in the data sourceIn data-driven testing, only test data is contained in the data source.
26 Model-Based testingModel-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system.Model-based testing for complex software systems is still an evolving field.
27 We’ll be concentrating on Keyword-Driven testing since it is a most beneficial framework for large-scale test automation.
28 Advantages of Keyword-Driven Test automation This Framework addresses the most common problem with test automation:Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise.When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity.Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.
29 Advantages of Keyword-Driven Test automation Reduced the cost and time spent maintaining and updating testsThe modular structure of keyword-driven testing means that new tests can easily be created from pre-existing modulesThe test team is capable of entirely automating tests, even without programming knowledgeCan be easily modified to use with different test toolReusability across different projects
30 Classic Keyword-Driven Example ObjectActionDataTextfield (username)Enter Text<username>
31 Recent Success with Keyword-Driven approach Estimate using Modular Framework: 1 yearEstimate with Keyword-Driven Framework: 6 monthsProject was completed in 4 months by 1 person, 2 month ahead of schedule.This was the first project for the automation developer utilizing the Keyword-Driven Framework.170 components were developed (1 component per Web page).109 Regression Test cases were automated.
32 Example of Test Data for Keyword-Driven test automation developed by Connected Testing Inc. for TestComplete and QTP
33 Resources“Everything you want to know about Test Automation…” by Brian Le Suer“Manager’s Guide to GUI Test Automation” by Yury Makedonov“The ROI of Test Automation” by Michael Kelly“What is my ROI?” by mVerify CorporationSQA Forums“An Overview of Test Automation Frameworks” by Nathaniel Ritmeyer“The benefits of keyword-based software test automation” by David W. JohnsonWikipedia
34 Igor Gershovich Company: Connected Testing, Inc. Phone:Website: