Sharing the good, the bad, the ugly & What can we do about it? GUI Test Automation Pierre Veragen The audience: How many are using GUI test automation today? How many feel good about the results? How many tried it and have given up? Sharing the good, the bad, the ugly & What can we do about it? 12/2/2018

What are we going to cover today? Why Consider Test Automation? Test Automation -- an Investment The Different Sides of Test Automation Forces Affecting Test Automation Three Case Studies The “No-Record-Playback” Approach An Iterative Approach to Implementation Be Selective The Deadliest Pitfalls Review QA 12/2/2018

Why Consider Test Automation? Our world today: Frequent changes Frequent builds Frequent deliveries to customer “To catch bugs early, we must test often” What Test Automation can bring: High speed cycle; code-test-analyze-fix-test… Better testing if designed right Repetitive, reliable testing Good coverage for Data-oriented and/or repetitive verification tasks 12/2/2018

Test Automation -- an Investment 1/2 Cost: Licenses Training Consulting Building the library Writing scripts The main cost: MAINTENANCE 95% Benefits: Catch bugs early when they happen… get the Duh effect Get bug ownership Free QA resource for other tasks Forces to write down what we actually test Can expand on it Can avoid the biggest embarrassment 12/2/2018

Test Automation -- an Investment 2/2 Our experience: App A - 210,000 lines of code App B - 37,000 lines of code App B again but a different test tool - ??? lines of code The easy traps: Volume is quality… Record now, clean up later “Don’t build/run automated tests now because the GUI is changing” Automated Testing is enough >>> All a bunch of fairy dust 8,000 loc shared by the two applications 12/2/2018

The Different Sides of Test Automation Component Type Main Test Tool Target Code Coverage Notes Methods, Classes, Components Unit Test High nUnit Test Driven Design API API driver, code in C++, .Net, VB6… High code coverage when combined with UT, but it is not what drives the test Test scenarios and test cases are driven by the expected usage of the system (Use Cases…) GUI GUI test tool In all cases use code coverage to pinpoint weaknesses in the testing Do not try to test components through a GUI 12/2/2018

Forces Affecting Test Automation “We may not know the details of future changes, but we can guess where it is going to change” Details of control will change >> attach names, enabled, visible, list… GUI Behavior will change >> added message, multilingual, new forms… Different OS, machine speed, network speed, file size Requirement updates See handout for design strategies …and we make errors too… 12/2/2018

Case Study 1: “Record & Playback” Demo 1 Demonstrate brittleness Demonstrate code duplication Demonstrate maintainability issues Code duplication: set window, control names Like hard coding values all over your code Also called Capture & Replay 12/2/2018

Case Study 1: “Record & Playback” The reality: AUT is always on the move Every time you record you duplicate code (buy now, pay later…) Every time you duplicate code you create increase your maintenance cost You’ll have to maintain it in hundreds of places Pretty soon we will record again to add clicking on a message box … in hundreds of places 12/2/2018

“Record & Playback” 12/2/2018

Case Study 2: “Encapsulate Recorded Scripts” Demo 2 Two methods to limit duplicating code Code approach: Think of GUI functionality Can’t keep up with methods names and arguments DB approach: Move name details to DB 12/2/2018

“Encapsulate Recorded Scripts” 12/2/2018

Case Study 3: “Development Project” Demo of an existing system Demo writing a script 12/2/2018

“Development Project” 12/2/2018

“Development Project” Business Requirements Mimic the User’s Actions Provide a Scripter Façade Encapsulate Attach functionalities Design for maintainability and scalability Provide timely Error Reporting of Errors Provide Database-Driven Tests for data and behavioral verifications Provide a framework to Extend the Test Tool’s Functionality A few good coding practices: Encapsulate No Code Duplication 12/2/2018

“No-Record-Playback” Approach The golden rules: Record only to understand how the test tool thinks and how it differentiates one control from another Then DELETE IT Never duplicate code Choose your testing goals wisely The Benefits: Fully encapsulated code – one place for the Click code, one place with the actual control name… Easy to upgrade and expand if well designed 12/2/2018

Iterative Approach to Implementation List what needs to be tested >> Test Scenarios at a high level, using an organized thought process (risk, UC, Requirements…) Earmark what could done through test automation Prioritize Set some reasonable goals (like a Build Smoke Test) to validate the automation approach (tool, encapsulation techniques, library design…) After the first script is running, reconsider your solution 12/2/2018

Be Selective in What to Automate 1/2 Repetitive to boredom Where we spent most of the time manually Test Scenario that would catch issues as they happen Unattended testing 12/2/2018

Be Selective… 2/2 2 to 5 times the time it takes to perform a test Some tests are nearly impossible to do by hand Run script 5 to 20 times before it becomes fully stable Build script stability with daily builds Build script stability with daily builds, i.e. when it has little effect on the project schedule 12/2/2018

The Deadliest Pitfalls Using Record & Playback Having no programming experience in the team Looking at it a back-burner project Ignoring the fact that you are building a framework “The test passes because the script ran without errors” Sloppy baselining practices 12/2/2018

Review for Success Consider building this test automation framework as a development project (requirement, architecture…) with a development team (SME, Automation Engineer, and Tester) Think reusable components Pick a small initial goal like automating the smoke test, then a basic Regression System Test Do not attempt to automate everything, think ROI Manage timers and synchronization issues in an encapsulated way Test your test tool 12/2/2018

Review Think maintainability as the AUT will change Think reusable components (script / library) Think scalability: use a library architecture to easily adapt to other applications being tested and other environments Get some developer involved at the get go Address synchronization problems early on Balance cool & productive features of the test tool Build a balanced team: SME, Automation Engineer, and Tester 12/2/2018

The End More material: Web sites Expected App changes and what to do about it? Functional requirements built on experience Web sites Fit and FitNesse … 12/2/2018