1. There is an A for everything Theoretical side of things Definitions Vocab ISTQB Good practices Things to remember Lessons to learn Practical (realistic ) side of things Activities and their benefits Examples 1

2. Test Automation Frameworks Roxana Cracana December 4 th, 2013 Summary : Define test automation framework Types of automation frameworks Create your own framework Focus Points Layer your framework 2

3. A set of assumptions, concepts and practices that constitute a work platform and support for automated testing. What is a Test Automation Framework? Is responsible for: Creating a mechanism to hook into or drive the application under test Creating methods to automate test cases Defining the format in which to express input data and expected results Executing the tests Reporting results Properties: Correct and complete Reliable Reusable Scalable Easy to use Easy to maintain Fast 3

4. Types of automation testing frameworks Linear Modular Data Driven Keyword Driven Hybrid Test Driven Development Behavior Driven Development Agile Frameworks Automation Testing Frameworks 4

5. Record and Playback is not an framework What is… Capture interaction with system and replay it Popular approach among commercial automation tools Advantages Very easy and fast to create initially No programming skills needed Easiest way to learn the features of the testing tool Disadvantages Does not test anything unless checkpoints added Very fragile – often single change in UI can break all tests No modularity or reuse Hard to maintain 5

6. Linear testing framework What is… Non-structured scripts interact directly with the SUT Also produced by capture and replay tools Advantages Fast way to create script Automation expertise not required, just tool knowledge Disadvantages No modularity or reuse Test data is hard coded into the script Very fragile – one change in the system may break all scripts Plenty of separates scripts Never a good basis for large scale automation SUT Test script 1 Test script 2 6

7. Modular testing framework SUT Test script 1 Test script 2 Test library Test method 1 Test method 2 Test method 3 What is… Interaction with the SUT done by functions in a test library Advantages Higher level of code reuse Easier script maintenance Disadvantages Building test library requires initial effort: takes time and programming skills Test data embedded into scripts 7

8. Data-driven testing framework SUT Test Data 1 Test Date 2 Parser Test script 1 Test script 2 Test library Test method 1 Test method 2 Test method 3 What is… Test data taken out of test scripts – usually externalized in tabular format Advantages Changes to the test scripts do not affect the test data Test scripts can be executed with multiple sets of data Disadvantages Creating scripts requires programming skills and time Initial effort creating data parsers can be high May be an overkill for simple needs 8 Test data for InvalidLogin UsernamePasswordError Message The field E-mail is mandatory abc123*The field E-mail is mandatory john.smith@g mail.com The field Password is mandatory john.smithabc123* The format of the e-mail address entered is not correct. john.smith@g mail.comabc1 The password must have between 5 and 12 characters. john.smith@g mail.coma1b2c3d4e5f6g The password must have between 5 and 12 characters. john.smith@g mail.coma1b2c3d Your e-mail address or password is not correct or is not registered at Mango.com

9. Keyword-driven testing framework SUT Test script 1 Test script 2 Test library Keyword 1 Keyword 2 Keyword 3 What is… Utilizes data tables and self explanatory keywords to indicate the actions performed on SUT Advantages Provides high code re-usability Test tool independent Tests can be designed with or without SUT Very good for large scale use Disadvantages Initial effort can be really high 9 Test : ValidLogin ActionObjectParameters NavigateBrowserhttp://shop.mango.com/RO/mango SetEmailjohn.smith@gmail.com SetPassword123abc* VerifyTextUsernameJohn Smith CloseBrowser

10. Hybrid testing framework What is… Is the combination of modular, data- driven and keyword driven testing frameworks Advantages Integrates the advantages of all the other frameworks. Disadvantages Very complex 10

11. Which type of framework to choose? Test frameworkAdvantagesDisadvantages Modular Reusable functions Hierarchical structure Test data within the scripts limits reusability Test script is dependent on software Data driven Improved maintainabilityDependency on technical expertise Test script is dependent on software Keyword driven Ease of maintenance Scalability Less dependency of software Dependency on technical expertise Requires large effort Hybrid Integrates the advantages of all the other frameworks Increased complexity Create your own framework! 11

12. Focus point Good testing frameworks have layers Layer 1: Control level Layer 2: Business level Layer 3: Test level Build layers over existing automation tools! 12

13. Focus point Test error and failure handling Decide when to fail your test Decide when to fail your test suite Think about tests setup and tests recovery Soft assertion vs Hard assertion 13

14. Focus point The key artefact of a testing framework is REPORTING Summary report Detailed report Reports repository Traceability Metrics Integrate early logging and reporting into the framework! 14

15. Focus point Object Repository Also called Application map Stores AUT objects Multiple ways to represent it Identify the best way to represent OR! 15

16. Focus point Continuous automation Parallelize test execution Distribute test execution Schedule test execution Automate test machine setup Automate test precondition Automate notifications Integrate test execution into a continuous integration tool! 16

17. Layer your framework – Control Level(1) Step 1: Control types Identify all control types from AUT Standard controls e.g. : button, dropdown, edit, checkbox, radio button Composed controls e.g. : radio button group, checkbox group, menu, table, tab group Custom controls Implement one class for each type of control Step 2 : Control methods For each type of control identify user interaction Action e.g. : click button, select value in dropdown, type value in edit Verification e. g. : verify selected value in drop down, verify all values from table Implement one method for each user action Integrate in each method error handling and reporting 17

18. Layer your framework – Control Level(2) Dropdown actions Select value Select index Verify selected value Verify all values Verify value exist Verify value not exist Verify number of values Verify values are sorted Verify is displayed Verify is not displayed Verify is enabled Verify is disabled Verify background colour Verify label Verify tooltip Edit actions Type value Clear value Append value Enter value Verify value Verify is displayed Verify is not displayed Verify is enabled Verify is disabled Verify background colour Verify label Verify tooltip 18

19. Layer your framework – Control Level(3) Dropdown actions Select value Select index Verify selected value Verify all values Verify value exist Verify value not exist Verify number of values Verify values are sorted Select value Select index Verify selected value Verify all values Verify value exist Verify value not exist Verify number of values Verify values are sorted Edit actions Type value Clear value Append value Enter value Verify value Type value Clear value Append value Enter value Verify value General actions Verify is displayed Verify is not displayed Verify is enabled Verify is disabled Verify background colour Verify label Verify tooltip Verify is displayed Verify is not displayed Verify is enabled Verify is disabled Verify background colour Verify label Verify tooltip 19

20. Layer your framework – Control Level(4) Logging Error handling : Hard assertion Edit control – Verify Value method 20

21. Layer your framework – Business Level(1) Step 1: Business Units Identify business units for AUT (business unit : page, section, widget) Create one class for each business unit Step 2: Application Map For each business unit, identify all objects that have a business value For each object identified: Associate an unique alias map Associate properties that unique identify the object into the business unit Step 3: Business Flows For each business unit identify scenarios or business logic flows that can repeat into test case Implement one method for each flow 21

22. Layer your framework – Business Level(2) Business Unit: Header Business Unit: Register Business Unit: HomePage Business Unit: Start Session 22

23. Layer your framework – Business Level(3) Application MAP for Start Session Edit_Email Edit_Password Link_ClickHere Text_Errors CheckBox_AutoLogin Button_StartSession Button_Register 23

24. Layer your framework – Business Level(4) Start Session Business Unit 24

25. Layer your framework – Test Level(1) Test implementation : Establish setup and recovery scenarios Identify a test framework (e.g. NUnit, JUnit, PyUnit) and the best way to create test suits and test cases Parameterize test cases if necessary No waits or code into the tests, only calls to control and business methods Test execution : Identify tools for test execution Schedule execution Automate test preparation 25

26. Layer your framework – Test Level(2) TC01Verify invalid email in Login. Precondition Email "roxana.cracana@gmail" is already registered with password ‘123abc*’ Navigate to url http://shop.mango.com/RO/mangohttp://shop.mango.com/RO/mango Click link “START SESSION” Step NoActionsExpected results 1 Type "roxana.cracana" in field "E-mail" "E-mail" has value "roxana.cracana". 2 Type correct pasword "123abc*" in field "Password" "Password" has value "*******". 3 Click 'START SESSION' button Error is displayed :”The format of the e-mail address entered is not correct.” Test Case vs. Test Script Keyword driven test script 26

27. Layer your framework – Test Level(3) Data driven test script 27

28. Conclusion Characteristics: Modularity Configurability Business oriented Exception handling Easily maintainable Logging & reporting Is applicable for : Smoke tests Regression tests Internationalization tests Client side monitoring and performance tests Can be used to automate : UI Web application UI Desktop application UI Mobile application A hybrid automation testing framework 28

29. 29