1. Accelerating Your Test Execution Pipeline
Bria Grangard

2. Who Am I? SmartBear Software
Automated UI functional testing tools & test management tools
Speaker
Joe Colantonio’s Automation Guild, 121 Automation Days, RTD, Better Software, and more!
Education
Went to Dartmouth: AB in Engineering, BE in Biomedical Engineering, MEM with a healthcare focus
What do I love to do?
Run, dance, play board games (Settlers of Catan anyone?)
@Bria_Grangard

3. Today’s Agenda Testing Overview The Importance of a Test Framework
What’s the Secret Sauce to Automation
Parallel Testing Overview
Benefits Of Parallel Testing
Q & A
@Bria_Grangard

4. There are bottlenecks in today’s development processes.
Iron triangle trade-off
Teams today are constantly feeling pressure to deliver software faster, without compromising quality
Automation might ramp up
There is only a certain point as to how scalable automation can be
Test environments are often the root-cause of the bottlenecks
They are very time consuming and costly
@Bria_Grangard

5. The promise of the new software delivery cycle
Design
Build
Test
Implement
Waterfall
Agile
DevOps
With the advent of the Agile Manifesto and the “maturation” of DevOps over the past few years, it seems that the promise of continuous deployments at a speed faster than light is almost fulfilled. Everywhere we look, development and ops team brag about how many times they ship a week, a day, or even an hour.
There isn’t less testing happening in agile or devops, the phase has simply been broke down, sped up, and spread out.
Week 1
Week 2
Week 3
Week 4
@Bria_Grangard

6. Time Consuming Nature Of Web Testing
More Features = More Testing
# of Tests
# of Browsers
To add to the innate problem of speed in the software delivery cycle, there are a few speed bumps unique to testing web applications.
As our product matures, we’ll have an increasing number of test cases to run in regression before each deployment. This can add exponential amounts of testing – and test execution time – in just a few months. Precious time most products do not have.
Also, as our product gains in popularity, we’ll have to support an increasing amount of browsers and devices. Customers now want to use their personal iPads to access your web portal that just 5 years ago was only accessed by the latest version of IE on…yep, you guessed it. Windows XP Service Pack 3. This again ads massive time to the time needed to sufficiently test across needed browsers.
LEAD: There are only a few ways to radically change your testing time to keep up with modern development.
Age Of Product
Popularity of Product
@Bria_Grangard

7. Diversify Your Testing
Test Less 1
More Bugs
Hire More Testers 2
Increased Cost
Diversify Your Testing 3
Run Tests In Parallel 4
There are only a few ways to radically change your testing time to keep up with modern development.
You can just test less. This results in more bugs, and we’d not recommend this strategy.
You can also increase the amount of testers on your team to keep up with demand. OR
You can diversify your testing suite with a mix of unit, API, and UI tests.
And run tests in parallel, executing tests in continuous batches of 10, 20, 50 tests at one time.
@Bria_Grangard

8. Now… How Can We Go Faster?
Test Frameworks
Automation
Parallel Testing
@Bria_Grangard

9. Level 1: Test Frameworks
@Bria_Grangard

10. The Basics of a Test Framework
Requirements
Tests
Defects
What do we make and how should it behave
Make sure it works as stated in the Requirement
Actual Results do not equal Expected Results
Definition
Sets
Environments
Let’s star high level with the software development lifecycle. While you all are probably very familiar with this let’s talk about some points.
Requirements. Where it all begins. What do we want to make, what will we make, and how should whatever we make really behave.
Need to have a solid foundation for success… you need to know what your requirements are and what you want them to accomplish. No confusion. It’s kind of like taking that moment to prepare for your essay before you write it.
For Tests you can have multiple types of tests.
Test Definition
Test set
What environments you want to run the test on-operating systems, browsers, resolutions, etc.
Defects—it’s inevitable. Your tests are going to find some bugs and there will be some problem. But if you have a defect you want to be able to log it properly and tie it to the correct test and requirement.
Releases—all of these things are encapsulated in a release. What are you pushing out for the next major or minor release/build.
@Bria_Grangard

11. What is a Test Framework?
Links tests to other SDLC items
Is NOT a Test Automation Framework but often contains one
Allows for rapid creation of tests from reusable components
Separates data from logic (REUSABILITY)
Provides a standardized test “language” and reporting structure for an application under test
@Bria_Grangard

12. Elements of a Test Framework
Library: A repository of all your decomposed scripts, separated into their components
Test Data Sources: A repository of all data sources
Helper Functions: A repository of all decomposed test scripts, automated or manual, that are inputs or checks
Test Environments: A list of all covered testing environments, broken out by type (OS, browsers…)
Modules: The combination of library items with any helper functions and test data sources–plus environments
Structure / Hierarchies: The “folder” structure of modules
@Bria_Grangard

13. Level 2: Figure Out What Tests Should be Automated
@Bria_Grangard

14. There are many types of testing that need to be done…
Browser: Chrome
HTML5, Angular JS
Network
Service/API/Database
There is some feedback you get only from a UI layer test. So while you should minimize the efforts of UI testing, you still need to have these tests to complete end-to-end testing workflows. In this case, the test starts at the browser level: Mozilla, chrome, firefox whatever you’re using. It then touchses on HTML5 or Angular JS depending on what you're using. It then touches on network level testing and services/API/Database test.
End-to-End Testing
@Bria_Grangard

15. UI Service Unit Automated Testing Pyramid Approach Manual
Mike Cohn’s Test Automation Pyramid is usually the go-to guide for deciding on how many tests should be automated.
(After discussing this in much detail as well as the recent debate)
The best way to incorporate a myriad of testing practices into your development cycle is to find and implement the right tools. Utilizing source control management tools (SCMS) such as Git or Subversion, CI tools like Jenkins, or defect management tools such as JIRA or Bugzilla, can speed up your processes as they integrate with a wide variety of automated test tools and provide more poignant feedback on which part of a test is failing.
@Bria_Grangard

16. Time To Test Behind Releases
Speeding Up Your Pipeline
Manual Testing
Record & Replay
Unit Testing
POM
Atomic Testing Continuous Testing
Time To Test Behind Releases
Longer
Shorter
In this image we are showing the evolution of the entire development pipeline.—transforming from manual testing to continuous testing. Even with the benefits a diversified testing process can provide, test automation and easier to debug pieces, there is still a cap on time to test and time to release. To reach the final two phases on the right, atomic testing and continuous testing, teams will have to run their tests in parallel. In most cases, this will require teams to move their testing to the cloud for cost and speed reasons.
Parallel Testing

17. With 5 manual testers, that is 3.5 weeks of testing
A Little Manual v Automated Math
Product v2
Product v2
# of Test Cases
1,000
# of Test Cases
1,000
# of Browsers Supported 10
# of Browsers Supported 10
Total Test Cases
10,000
Total Test Cases
10,000
Avg Test Run Time
4 Min
Avg Test Run Time
.5 Min
Total Test Time
666 Hrs
Total Test Time
83 Hrs
With 5 manual testers, that is 3.5 weeks of testing
With 2 QA engineers,
That is 1 week of testing
@Bria_Grangard

18. Decide on What to Automate
Environment Setup/Teardown
Data Entry
Form Filling
Varying data inputs in a repetitive process
Exposing backend data (APIs, DB table, etc.)
Repetitive/boring tasks that are prone to inattention errors
Tasks with high reuse value across many workflows
Tests with timing or screen responsiveness as a criteria for success
Many non-functional test types, such as performance testing
Capturing Results
@Bria_Grangard

19. But my Dev team says I have days to test, not weeks…
@Bria_Grangard

20. Level 3: Go Faster. Let’s take an example…
Let’s imagine a real world example: in release 3 of your product, you have 8 hours of sequential regression testing to perform before the team feels confident to deploy.
By release 5, this may be twice the amount of hours you’ll need to run your tests and as a bonus, your product is getting popular and is being used by more users on an increasing number of different devices. Before you were testing for only Chrome and FireFox, but now you see that you need Android and iOS devices, Safari and multiple versions of Internet Explorer. So you have 16 hours of tests and 10 different devices or browser to cover. 
This would take us 160 hours for complete test coverage before our deployment. With parallel testing environments, we can run our 16 hours of tests on 10 different devices at the same time, saving us 146 hours of testing time.
@Bria_Grangard

21. Running tests sequentially, we were able to run our tests in 1 week
Let’s Go Faster!
We can, with parallel testing
Test 1
Test 1
Test 1
Test 1
Test 1
Test 2
With 20 Parallel executions, we can run our entire test suite in only 4 hours
Test 2
Test 2
Test 2
Test 2
Test 2
Test 3
Test 3
Test 3
Test 3
Test 3
Running tests sequentially, we were able to run our tests in 1 week
Time is finite. So we need to maximize the time we have. Parallel testing allows you to test faster with a quicker turn around in deployments. No developer wants to spend more time testing their product than they did developing it.
How can parallel testing be done? By setting up multiple VMs and other infrastructure devices or by using a cloud test service.
Test 3
@Bria_Grangard

22. Types of test to run in parallel
Most effective tests to see ROI from for parallel testing
Cross Browser Testing
Regression Testing
Testing across different browsers and devices is one of the most time consuming aspects of testing the front end of your website or web application. Run more tests, against more browser configurations by running them in parallel.
Because deployments are happening at such a rapid pace, regression testing is on of the best ways to have a type of “testing version control” making sure the functionality of the new build, matches that of the last stable build. Running these tests in parallel allows more to be tested.
Unit Testing
Smoke Testing
According to the testing pyramid, Unit tests should be your most abundant test type in your entire testing suite. Because of this, running 14,000 unit tests in under an hour is really only possible with a massive parallel testing infrastructure investment.
Need to get your minimum testing done in the next 20 minutes while you push a hot fix? Only way to do that is to run them in parallel, allowing you to get the most testing done in the shortest amount of time.
Cross Browser Tests
Regression tests--regression testing is essential between deployments. The process is vital to ensure that a piece of software is still functional after a new update has been made prior to the next shipment. A challenge with running regression tests is the limited amount of time often provided and the growing nature of your test suite as your application matures. Developers looking to push their product out faster may give very little time between when they’ve finished their build process and the production date to conduct any testing. Running through the regression suite quickly with concurrent tests will get build engineers any necessary feedback faster and will keep your DevOps teams happy.
Unit Testing: unit tests are a great choice to run in parallel because they are naturally small since they test very particular or niche functions of an application, but can number in the thousands to tens of thousands.
Smoke tests or build verification testing: is the final testing practice teams should consider implementing in parallel. Since smoke testing focuses on ensuring that only the most important functions of an application work as expected, it is typically used to decide whether or not to proceed with further testing. A smoke test that passes is an indicator to go ahead with more testing. If it fails, teams will stop testing and ask for a new build with the initial required fixes.
Smoke testing is usually done to ensure that the minimal viable amount of testing is done with any quick fixes going into production immediately. Smoke tests can be conducted manually or through automation, but parallel testing will enable the process to go faster. Testing the few business critical processes, such as log-ins and checking out,. To get the product out the door in under five minutes can bring in revenue and make customers happy. This ensures enough time to test as thoroughly as they would like in between deployments—parallel testing accelerates this.
@Bria_Grangard

23. Benefits of Parallel Testing
Quick Deployments
Faster Feedback
Cross Browser Testing
Better Test coverage
Saves Valuable Time
Quick deployments—run as many tests as you want concurrently whether it’s 2, 10, 50, or 100.
Faster feedback—the faster you run your tests, the faster the feedback. In older development models and testing practices it could take weeks to receive feedback once a developer built a feature and sent it across to QA. By that time, developers have moved on and may not remember what the function that failed was. Without writing a comment for every piece of code, this can be a real challenge to work with. “shifting left” “continuous testing”
Cross Browser Testing—this is drastically more achievable with parallel testing. Typically, expanding coverage to incorporate all necessary environments can become very challenging and time consuming. With parallel testing you can test against multiple browsers and browser types all in a shorter timeframe than if you were running tests sequentially.
Better Test Coverage—enabling teams to test more in less time inherently means better test coverage. Environments and platforms
Saves valuable time
@Bria_Grangard

24. Key Takeaways Test frameworks are important and a key to success.
When automating—choose which tests to automate wisely
Parallel testing let’s you go faster.
@Bria_Grangard

25. Questions?
@Bria_Grangard

26. Thank You!
#C768
@Bria_Grangard