1. The difference that makes the difference

2. Make your test automation project a success story Magnus Nilsson Lemontree

3. Agenda  The Test automation concept – Success factors – Components – Test cases – Framework  Walkthrough of a test automation project – Analysis phase and the business case – Maintenance of an automated solution

4. Test automation - success factors  Develop functional components  Decouple in- and output data from the script  Central data storage  Development project – strategy and – center of excellence  Combine components to test cases by actions as input data Functional components Test automation In-/outdata Central data storage Project Test Automation Concept

5. Test automation - building up the components  Functionality available in test – Development of the functionality as it comes available Requirement Design and development Test Phase 1 Phase 2 Function 1 Function 2 Function 1 Function 2 Function 1 Func. 2 Each functionality is part of a component (or a component itself) Func. 3 Func. 4 Test Automation Concept

6. 6 Test automation - building up the components CRM System  Function 1: Create customer using CRM CRMCustomer

7. 7 Test automation - building up the components http://www.webshop.se  Function 1: Create customer using Webshop WebCustomer

8.  Function 1: Create customer using mobile application 8 Test automation - building up the components AppCustomer

9. Test automation – test cases Login Customer Account [new] Inventory Login Credit check [create] Subscription Order [new] Account [new] Customer [open]Billing Test case 1: Test case 2: Test Automation Concept

10. Test automation – test cases Login Customer Inventory Login Credit check [create] Subscription Order [new] Account [new] Customer [open]Billing Test case 1: Test case 2: Account [new] ver 2.0 ver 1.0 Account [new] ver 1.0 Account [new] ver 2.0 Account [new] ver 2.0 Test Automation Concept

11.  1 st generation script – Using automation tool for capture and replay only – Hardcoded data  2 nd generation script – Code refactoring for maintainable scripts – Uses generic functions for iterations and error handling  3 rd generation script – Data driven test (DDT) where dynamic data is taken from data table rows – Keyword driven test (KDT) where action words controls the actions – Use of driver scripts for executing multiple test cases – SME for test designs and requirement – More complex scripting by developers Test automation - development of a framework Framework Development of a

12. Test automation - development of a framework  HP QuickTest Professional/Functional tester – Framework itself with test management connection ALM, keyword driven action scripts with data table, function libraries, Repositories, BPT, recovery scenario – Generates code  Library definitions, template of standard functions  Structured data handling, input and output data Repository objects Application GUI layer Application GUI layer List object Text object Table object Object recognition engine Data table M O Business ID will be validated if provided OutOut N/AN/A Cust omerI D_O ut BusinessIDBusinessID Fam ilyN ame First Na me InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M Offic ialN ame InIn Private or Business OutOut Script state _Out Status of execution Link Entit y_In InIn O Entity linked from previous component Function library A Function library B ALM Test resources Test plan Test lab Dashboard Recovery scenario Test action script Test action script Framework Development of a

13. Test automation - development of a framework  HP QuickTest Professional/Functional tester  Library definitions, template of standard functions – Application specific libraries – Standard actions in applications should be in application library – Component libraries enabling Configuration Management Branched Release Management for ST, AT and dev Multiple checkout and merge Only driver scripts – Framework libraries handling framework and best practices  Structured data handling, input and output data Framework Development of a

14.  HP QuickTest Professional/Functional tester  Library definitions, template of standard functions  Structured data handling, input and output data – Central data source – Decouple structure and data – Central structured data source Test automation - development of a framework Data table Component A sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component Component B sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component CreateCustomer sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component ALM Test resources Test plan Test lab Dashboard Data & Structure Framework Development of a Function library A Master DB Data Data & Structure Test action script Test action script

15. The test automation project Phase 1 Development System test Acceptance test Week Analysis Detailed design Requirement Plan next phase Documentation & Training Phase 2 Detailed design Development … … Walkthrough Project Phase 0 The Business Case

16. Test automation project - business case  Why do we need a business case? – Test automation cost time and money – The automated solution must at least ensure the time spent in development, maintenance and execution is less than a manual solution would cost to accomplish the same goal.  What is a test automation business case? – List of all costs and savings associated with test that possibly can be covered by test automation. – Not all drivers are possible to calculate directly as quality. But quality can be expressed as an estimation of the number of testing hours to ensure the desired level of quality. – Additional test automation drivers as repeated executions is calculated indirectly as increased quality.  How can a business case be used? – Business case can be used to predict effectiveness as automation deliveries should follow the calculated schedule. – ROI should be calculated at regular audits based on the business case. Walkthrough Project

17. Test automation project - business case Cost for manual regression tests Cost for error handling External and business costs for errors Production AT Number of errors found per year Handling time per error (report, reproduce, administer, test data, close defect) Number of FTE's for error handling per year Error handling cost per hour Annual error handling costs 1000 2 2 100 200000 Error source 500 4 2 100 200000 Total test case executions per cycle Number of test cycles Execution time per test, h Execution cost per hour 1000 4 0.5 100 Annual cost for execution of regression test 200000 Grand total Annual error correction costs for errors Business costs for productions errors 1) 0100000 2 400000 Cost multiplier for Production Errors 1) Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test 1100000

18. Test automation project - Business case Savings - Manual Test Labour Savings - Error Reduction Implementation Cost (Year 1) Operational costs (running) Description Reduction ratio Reduction ratio Annual reduction/cost Acc reduction 100000 70000 Phase 1 Phase 2 Phase 3 Total savings 270000 90000 180000 Year 1 Year 2 Year 3 10% 20% 30% 540000 90000 270000 Year 1 120000 160000 140000 Year 2 Year 3 60% 70% 80% 120000 420000 260000 Year 1 proj support, SME Year 2 TAC OP Year 3 TAC OP 50000 60000 Acc savings Year 1 Year 2 Year 3 -80 000 260000 370000 180000 550000 -80 000 Walkthrough Project

19. Test automation project - Business case Walkthrough Project Technology 2 1 1 Mixed Yes 5% 1.5 2 3 Create 5 1 2 Maintenance Reusability Simple TC Complex TC Framework Test data Integrations Selection Value 45 61 New  Implementation cost calculations – Initial estimates – Design estimates CRM Customer 20 10 15 10 20 Create Terminate WS Customer Create App Customer Create Analyis&Design Development 20 Component/Functions

20. Business case – follow up short term Analysis Detailed design Development System test Acceptance test Plan next phase Phase 1 Phase 2 Detailed design Development Week Delivered Planned Test cases Week Walkthrough Project

21. Business case – follow up long term ” T est automation strategy”  Description of the standard approach on how to use the tools  Company specific set of rules on how automation should be done  Generic solution that is not dependent on individuals  Maintenance focused  Encapsulated by test automation centre  Enforced by framework  Ensured by the automation project But we need to know what we are testing! We should not test manually what we execute automatically! Walkthrough Project

22. Analysis  Analysis – Analysis based on existing documentation, test cases, use cases and requirements (static test review) – Provides the prioritization to an analysis structure template  Disposition – Decision making: if we go live what does and does not work? – What business processes are working and not working or not tested? Often details on number of test cases passed and failed on integration and system level is not vital Test and defect status itself often only aids prioritizing of resources/focus  Structure – Common and clear reporting on a level that enables correct business decisions – Structured test design and coverage analysis – Will control the test automation deliveries – Used in test management to create common priorities for both manual and automated tests  Implementation – After the analysis and approved structure design, the structure may be implemented using available tools in HP QC/ALM Walkthrough Project

23. Analysis structure - disposition  what is the system used for? – how is it done? which are the requirements on process level? » which data conditions are covered by use cases? Use case Process Requirement Data condition 1 Data condition 2 Walkthrough Project Example Customer Web Self Care New Private Business

24. Analysis structure - disposition Requirements Folder area Process Description Prio Use cases/product Customer Account Subscription Create Terminate New Sales Web self care 1 Modify Cancel 3110Web shop Sales Customer operations 13113115CRM SalesDevice23App SalesDevice2111Online 1 ProductInventory1 Product configurations 3 1 1 1 Walkthrough Project

25. Requirement and component coverage Requirements Folder area Requirement Process Data condition Test case Data condition Test case Prio Components Customer Account Customer Create Terminate New Create Sales Web self care R1.1 1 Modify Cancel Create Customer Sales Customer op R2.3.4 1 Sales2 Terminate customer Sales2 Create bank account 1 CRM Create Customer Web Shop X Customer op R2.4.1 X Customer op R4.1.1 X X Sales Customer op R4.1.1 Create invoice account X Test case TC23 TC12 TC1 TC45 TC6 Walkthrough Project X

26. Coverage analysis and status report; BC follow up Customer Customer operations New Terminate New 2 1 0 3 2 1 1 No run Not completed Failed Passed Status Not covered 1 327 1 3 Passed 1 1 1 3 Not covered Not completed Web self care Mobile device New Failed 0 1 3 2 2 3 2

27. Coverage analysis and status report, HP ALM  Coverage in ALM  Test either – manual, – automated or – business component Walkthrough Project

28. Maintenance Test cases created Test data prepared Result reporting arranged Press the button Walkthrough Project Then what? X Automation Engineer Functional tester

29. Automation stakeholder Subject Matter Expert Test Automation Center Project organisation - Use roles to enforce competence Automation Engineer Automation Analyst Automation Solution Architect Automation Project Lead Walkthrough Project Project A Automation stakeholder Subject Matter Expert Project B Automation stakeholder Subject Matter Expert Project C Automation stakeholder Subject Matter Expert

30. Project organisation - Test automation center Development Execution Maintenence Walkthrough Project

31. Communicating Reporting Planned activities Test automation center TAC Service Management Management Test management Release management Project planning Change request TAC Execution TAC Execution TAC Maintenance TAC Maintenance TAC Development TAC Development Request Result Operations ST SIT AT Project A Project B Project C Maintenance Supplier Defect and Issue Management Resolution

32. Short about Lemontree  Swedish company established 1999  Awarded Gasell and Superföretag 2008-2012  Turnover +98 MSEK 2011  AAA-rated  About 100 employees, growing intensive  Owners are actively working in the company, no external capital  Offices in Stockholm and Oslo  QM trainee program since 2012  Focus areas Quality Management eGovernment Enterprise Solutions

33. Quality Management AQUA solutions

34. Thanks! Magnus Nilsson Lemontree Solution Manager magnus.nilsson@lemontree.se