1. Building an Automation Framework around Open Source Technologies Charles Thangamuthu 20 th November 2010 Disclaimer – This is a personal case study and does not represent PTC’s view on the topic

2. 2 Thanks SiliconIndia!

3. 3 Before we step in…. What this presentation is not What this presentation is  Does not say open source technology is better than commercial testing tools  Does not say the components talked about are the best in their class  Does not talk about a PTC proprietary or certified framework  Does not claim all your automation woes are solved  An example of how open source tools can be used to form an automation framework  To say there are solutions readily available for many of your problems – don’t have to reinvent the wheel  More of a case study

4. 4 Topics  Automation Framework – Definition – Requirements – Functions  Generic Architecture  Framework components – Automation Tool – Version Control System – Continuous Integration – Test Runner – Data Manager  Advantages – Flexibility – Scalability  Benefits

5. 5  Automation Framework is not a single tool or process but a collection of tools and processes working together to support automated testing of an application  Is an integrated environment that sets the rules, assumptions, concepts and tools to aid automated testing  It integrates function libraries, test data, object details and various reusable modules  These components act as building blocks which can be assembled to represent a business process Test Automation Framework Definition - What is it?

6. 6 Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Distributed test execution Test Automation Framework Requirements – What do we need? Automation Framework Platform and OS independence

7. 7 Test Automation Framework Functions – What does it do?

8. 8 Test Automation Framework Architecture – What does it consist of? Application under test (AUT) Data Manager Test Runner Integration (CI) Version Controller Automation Tool Test artifacts Test Scripts Library Functions Config Files Data Files Loggers

9. 9 Automation Tool Automation Tool Selection - Considerations Operating system support Resource requirements Technical Information Features & Usability Event Capturing Scripting Support In-built Functions GUI abstraction features GUI spying features Capturing comfort Installation Test Execution Result reporting Supported languages Object orientation Compiler / Interpreter Browser handling XML handling Interaction with File System Wait /Sleep functions

10. 10 Automation Tool Automation Tool Selection – Considerations continued… Editor features Debugger Version management Integrated Development Environment Data Driven Testing Result Reporting Documentation Community Support Output formats Flexibility for customization Supported file formats Ease of use Users manual Online help FAQs WIKIs Forums Technical support

11. 11 Automation Tool Focus on Selenium Selenium

12. 12 AdvantagesDisadvantages Supports C#, Java, PHP, Perl, Phython Supported on Windows, Linux and Mac OS Highly active developer community - backed by Google Powerful methods to locate elements (Xpath, CSS, DOM) Challenges running on browsers other than Firefox Difficulty to automate AJAX Limitations to automate Applets, MS and Desktop Applications Automation Tool Why Selenium?

13. 13 Automation Tool Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Automation Framework Fulfills the following requirements  Win Runner  HP Quick Test Professional  IBM Rational Functional Tester  Parasoft SOATest  Rational Robot  Silk Test  QFTest  Test Partner Other popular Automation Tools Distributed test execution

14. 14 Version Control System Git Purpose  Is a way of ensuring the evolution of the tests corresponds to the evolution of the application features  Allows concurrent updates to the same test  Maintains the history and baselines of the automated test  Is a free, fast and open source Distributed Revision Control System  Can work offline  Everyone has a local sandbox  Branching and Merging are fast and easy

15. 15 Fulfills the following requirement Version Control System  Clearcase  Subversion  Fossil  Aegis  Bonsai  Concurrent Version System (CVS)  Revision Control System (RCS) Other Version Control Systems Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Automation Framework Distributed test execution

16. 16  Is a Free, Open Source CI server written in Java  Monitors executions of repeated jobs like – Building a software project – Executing automated tests – Sending e-mail notifications  Can execute – Apache Ant and Apache Maven based projects – External jobs on a remote machine  Supports SCM tools including CVS, Subversion, Git and Clearcase  Easy to install and configure  Supports plug-ins with RSS/E-mail/IM Integration Continuous Integration System Purpose HUDSON  Gives early warning of broken or incompatible code and conflicting changes  Does immediate unit testing of all changes in code  Initiates and monitors repeated jobs like test automation runs at predetermined schedule or event

17. 17 Fulfills the following requirements Continuous Integration System  CruiseControl  CruiseControl.NET  CruiseControl.rb  Apache Gump  Apache Continuum  BuildBot  CABIE Other CI Systems Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Automation Framework Distributed test execution

18. 18  Is an automation framework component that provides a simple workflow for execution of tests, result collection, re-running failures, and comprehensive reporting  Is an application built in-house on top of JUnit  Some of its features are – Compile APIs/Tests as needed – Generate Test Suite as per requirement – Selective execution of Tests in unattended mode – Re-runs the failed tests for the number of times specified – Creates comprehensive report with total number of tests passed and failed, time taken by each test, reasons for failure and logs – Localization Support – Even Video capture! Test Runner Features of RADAR

19. 19  An essential feature for repeating tests and testing in parallel  Is a set of APIs (developed in-house) allowing users to save and restore data in the database and LDAP  Creates multiple identical database schema clones from a Master and assigns them to multiple AUT installations Data Manager Features of ROSLIN Creates a schema with the required data loaded Creates multiple clones of the schema Restores the schema for automation runs as needed Distributes the clones to the requesting AUT instances Virtualization could be used as well

20. 20 Fulfill the following requirements Test Runner & Data Manager Platform and OS independence Data driven capability Customizable result reporting Ease of debugging and logging Version control of tests Execute unattended runs Repeat tests as needed Extensible and Customizable Status notification Automation Framework Distributed test execution

21. 21 Test Automation Framework Example AUT ROSLIN Test artifacts JUnit tests Scripts XMLs Java APIs Annotations Log4j, Dom4j Version Mgmt GIT Automation Tool SELENIUM RC Data Manager Test Runner RADAR Cont. Integrator HUDSON

22. 22 Advantages - Flexibility Framework design considerations What is a Solvent layer?  Reduce maintenance efforts to a minimum  Provide a means to help people with lesser programming expertise to create automated tests  A common infrastructure for the Development and QA teams to build and execute automated tests SSolvents are a set of API that help create Selenium tests, without executing Selenium commands directly SSolvents package the logic and Selenium commands required to perform various user actions into simple, reusable classes IIf you are wondering what’s in the name - Solvents reduce the amount of Selenium required to create a test (sometimes to none) thereby diluting Selenium, hence the name ‘Solvent’

23. 23 Automated Tests & Externalized Data Solvents Selenium JUnit ANT Advantages - Flexibility Layout with Solvent Layer Additional Layer

24. 24  It aids reusability of code  They can be ‘wired’ together to form test scenarios  Changes need to be made at a single place and not in multiple tests  Makes the test / script more readable Advantages - Flexibility Advantages of the additional layer POINT TO NOTE – The automation framework is flexible enough to accommodate an additional layer as needed

25. 25  The framework could be integrated with tools that help determine code coverage like Corbertura, EMMA and Jtest – Assess if the testing done is sufficient – Analyze if the testing focus is as planned Advantages - Scalability Other possibilities  Mechanism for code review like Gerrit could be integrated with the framework – Enforce standards, conventions and best practices for creating tests – Inspect the overall quality of the tests

26. 26 POINT TO NOTE – The automation framework can be enhanced by adding additional capabilities Advantages - Scalability Enhanced AUT ROSLIN Test artifacts XMLs Java APIs Annotations Log4j, Dom4j Eclipse Ant JUnit Tests Version Mgmt GIT Automation Tool SELENIUM RC Data Manager Test Runner RADAR Cont. Integrator HUDSON Gerrit EMMA Newly Added

27. 27 A consistent, flexible and scalable framework will help automation play a role at every stage in the software development process Benefits Software Development Ecosystem Not just test automation, some aspect of process automation is also in-built

28. 28  Maintainability – Significantly reduces the test maintenance effort  Reusability – Due to modularity of test cases and library functions  Manageability – Effective test design, execution, and traceability  Accessibility – To design, develop & modify tests whilst executing  Availability – Scheduled execution can run unattended on a 24/7 basis  Reliability – Due to advanced error handling and scenario recovery  Flexibility – Framework independent of AUT or environment  Measurability – Customizable reporting of test results ensure quality Benefits Key Advantages