Presentation is loading. Please wait.

Presentation is loading. Please wait.

…or proving Einstein was right about insanity Presented by Jim Hazen.

Similar presentations

Presentation on theme: "…or proving Einstein was right about insanity Presented by Jim Hazen."— Presentation transcript:

1 …or proving Einstein was right about insanity Presented by Jim Hazen

2 Introduction Jim Hazen Veteran of the Software Testing Trenches Experience in software testing, both commercial and consulting work. Working with automation and tools since 1989. Implemented Automation Tools on DOS, OS/2, Windows and Web platforms. Experience with tools; including HP/Mercury, IBM/Rational, and other ‘scripting’ languages.

3 Sage Words to Remember “Insanity: doing the same thing over and over again and expecting different results.” – Albert Einstein

4 Agenda What is ‘Test Automation’ A Brief History Different types of test automation, tools and uses Automation Myths How to correct them Resources & Skills needed Setting expectations for Management & Others Summary Q&A

5 What is Test Automation?

6 It’s a Big Topic to Discuss

7 Bigger Than We Think!

8 Is It This?

9 Or More Like This?

10 Who Builds the Automation?

11 Some Are Seen This Way

12 Most Look Like This Instead

13 What ‘Test Automation’ Really Is “Automation” definition Webster’s – “the technique of making an apparatus, a process, or a system operate automatically” In Software Testing we use ‘tools’ to drive our processes People only think of tools that execute a test as ‘Automation’. Other tools can be used for Test Design, Test Data Generation, Defect Tracking, Test Management, etc. “Automation” is using a tool during any part of the process of testing.

14 Who Test Automator’s Really Are

15 A Brief History Test “Execution” tools have been around a long time! Mainframes (pre Mid-80’s) PC DOS (Mid 80’s – 90’s) PC GUI & Web (1990’s – present) Mobile (2008 – present) Mainframe – Home grown ‘macro’ tools 1, Hyperion PC DOS – AutoTester, SuperKey (macro tool) PC GUI – Automated Test Facility (ATF), Rational Robot, QA Run, Winrunner, etc. PC Web – QTP, SilkTest, Selenium, TestComplete, etc. Mobile – MonkeyTalk, Robotium, etc.

16 Other Types of Test Automation Tools Other Test Execution Tools xUnit (JUnit & NUnit), SOAPUI, FitNesse, etc. Test Design & Test Data Generation Hexawise, AllPairs, Tosca Test Suite, etc. Test Execution Management HP ALM, MS Team Foundation Server, TestLink, etc. Test Environment Management VMWare vCloud Director, etc. Defect Tracking & Reporting Bugzilla, JIRA, etc.

17 Automation Myths & Misconceptions

18 Old Myths You can automate 100% of your tests Automation will “cure” all problems with testing Automation tools are usable by anyone Record/Playback is all you need Automation allows Management to cut test staff One tool is all you need Automation will speed up testing process * Oldies, but goodies!!

19 New Myths You don’t have to know how to program Opensource tools are “free” Commercial tools are “expensive” GUI level automation is a waste of time “Codeless” and “Scriptless” tools Commercial tools can’t support agile Developers only need to focus on Unit Tests

20 How to Correct Them 100% Automation – Certain tests shouldn’t or can’t be ‘automated’. 80/20 Rule; automate 20% of functionality used 80% of time. Automation solves all testing problems – it helps, but it creates new problems too. Automation can be done by anyone – Not everyone has the ‘mindset’, it is both learned and inherent (logic). Record/Playback is all you need – Proven time & time again to be false, and to fail.

21 How to Correct Them cont. One tool is all you need – There isn’t a one-size fits all. Automation speeds up testing process – The “Illusion of Speed” problem. Both execution and creation. You don’t need to know programming – Anyone can Automate problem. Opensource tools are free – Costs due to ramp up time (money) and skilled staff to do it (more expensive). 2 Commercial = Expensive – Not really in grand scheme of things. 2

22 How to Correct Them cont. GUI level automation is a waste of time – Not really

23 How to Correct Them cont. Codeless/Scriptless tools – Just a different UI. You’re still writing a ‘test’ which is a set of ‘instructions’ (code). Commercial tools don’t do Agile – Yes they do, it’s how you use the tool that counts. Developers only create Unit Tests – Other tools require someone to ‘build’ the fixtures/code to ‘talk’ to the SUT.

24 Why Failures Happen Source: IDT Survey 3 85% Failure Rate! Too High!!

25 Einstein was right! We keep repeating history, why? Always a ‘new’ group of management and testers. Companies always looking to do testing faster, cheaper and with less resources. Vendors and others who sell “Automagic”. Instant gratification mindset. Automation takes time. Not learning from past mistakes. Software not designed with “testability” in mind. Testers not involved early with decisions, not included by management.

26 Resources for Automation Dedicated Staff Skilled & experienced staff Dedicated Equipment/Environments Reduce risk of competition of resource or teams stepping on each other Dedicated Budget Automation isn’t cheap, plan appropriately Dedicated Management Support Can’t have any of the other 3 without it

27 Skills for Test Automation It’s a different view and purpose for the script code. Creating ‘software’ to ‘test’ software; i.e., PROGRAMMING! Need basic understanding of programming methods. Understand different types of technology layers GUI, API, System, Web, Services, Database, etc. You need to know how to program in different languages when needed. C#, Java, VBScript, Jscript, XML, SQL, etc.

28 Test Automation Skills Cont. Need to understand how to ‘hook’ the software under test and interact with it, and at what level. API, or Object, or Service, or Protocol, etc. Understand how tool ‘interacts with’ the software under test; what functions/methods are to use. GUI Object ‘methods’ (functions) Ex: SwfWindow(“LoginForm").SwfEdit(“ID”).SET “Jim” Need to understand how to “pick apart” the software and “drive it” programmatically.

29 Axioms to remember SMART Standalone Test/Script able to run by itself, minimize dependencies Modular/Maintainable Scripts and code need to be modular in design and easy to maintain Appropriate Build according to need for project, don’t try to automate 100% Reusable Build for reuse; keep script code compact and focused. Testable & Traceable Make it easy to debug your code, and be able to link together.

30 Axioms to remember cont. SEARCH 4,5 Setup Test/Script needs to check for pre-test conditions and flag if needed. Execute The logic and data needed to run the test itself Analyze Validations, Error Handling, etc. Report Progress, Error, Summation of run information (Logfile) Cleanup Test/Script needs to ‘cleanup’ (temp files, reset data, etc.) Help (or Home)

31 Setting Realistic Expectations Management & Other Groups 100% automation is an unreachable goal. 6,7 Automation is its own form of software development. Ramp-up time will be required, longer than you think. Benefits realized over time, no immediate payback for investment.

32 Expectations cont. Management & Other Groups No tool supports all systems & technologies out of the box. Not all testers can write scripts. Specialized resources are a must and need to be dedicated to the work. Nothing is “Free”, there are always costs involved. Don’t believe the marketing hype! Don’t buy the snake oil. 8 There is no Magic!

33 Otherwise, the result

34 Summary What is ‘Test Automation’ – A lot more than you think! History – Lots of tools over the last 30 plus years. Different types of test automation tools – Not just for execution of a test only. Automation Myths – Old and New; lots of failures due to them. How to fix the problems – Be realistic, dispel the myths. Resources and Skills needed for test automation – Again, a lot more than you think. It is Programming! Setting expectations for Management & Others – Without proper expectations you are painted into a corner.

35 Remember… It’s automation, Not Automagic!

36 Q&A

37 Contact Info Jim Hazen Company: Freelance Contractor Home email: LinkedIn profile:

38 References 1) “Experiences of Test Automation: Case Studies of Software Test Automation”, Chapter 5, Graham & Fewster, Addison-Wesley, 2012 2) “Test Automation - Let's Talk Business”, Igor Gershovich, Connected Testing, 2006. 3) “Implementing Automated Software Testing”, IDT Survey, Elfriede Dustin et al., Addison-Wesley, 2009. 4) “How We Test Software At Microsoft”, page 187, Alan Page, Microsoft Press, 2009. 5) “How to Automate Testing: The Big Picture“, Keith Stobie & Mark Bergman, 1992. 6) “Seven Steps to Test Automation Success”, Bret Pettichord,, 2001. 7) “Success with Test Automation”, Bret Pettichord,, 2001. 8) “Test Automation Snake Oil”, James Bach, 1996

39 Thank you for attending this session. Please fill out an evaluation form.

Download ppt "…or proving Einstein was right about insanity Presented by Jim Hazen."

Similar presentations

Ads by Google