The difference that makes the difference

The difference that makes the difference

Make your test automation project a success story
Magnus Nilsson Lemontree

Agenda The Test automation concept
Walkthrough of a test automation project Execution and maintenance of an automated solution Test strategy Development of a Framework Organizing your code

The test automation concept
Fundamentals behind test; a short review of Lifecycle activities Test levels Test methods Development models Test automation and test objectives Test automation success factors Test Automation Concept

Test as part of the lifecycle activities
Project Design Release Requirement Test Business process Architecture Development Defect Management Maintenance Monitoring Test Automation Concept

Connecting the activities Test levels
Integration test Unit test System test Integration test Acceptance test Unit test Acceptance test Test Development System test Functional Non functional Test types Test Automation Concept

Connecting the activities Test methods
Black box Black box Testing delivered functionality GUI test White box Testing internal structures Code Testing modules, classes, libraries, functions on code level White box Test Code Test Automation Concept

Connecting the activities Development models
Waterfall Waterfall The lifecycle approach Iterative Planned repeated cycles Agile Iterative and development adaptive to changes Practicing continuous Integration, CI Scrum Sprint-based development based on both on agile and iterative development methods Iterative Agile Development Scrum Test Automation Concept

The challenge of test Testing methods Test levels Development models
Code White box Black box Testing methods Unit test Integration test System test Acceptance test Test levels Development models Waterfall Agile Test Development Test Automation Concept

Test automation is all about unifying test.
Environment Development Increase code quality by implementing unit tests Development methodology TDD Automated builds, CI Environment Test environment creation Test data management, TDM Harness Stubbing interfaces Test Scripted tests Test interfaces GUI Performance Harness Test Automation Development Test Automation Concept

The challenge of test, connecting…
Testing methods Test levels TDD TDM Test automation Test Harness CI Automation Integration GUI Development models Waterfall Unit test Test test Development Acceptance test White box Agile Code System test Black box Test Automation Concept

System integration test
Example 1 - Test harness Test Harness Enables test before interfaces exists or are available, agile development but also project delivery regardless development model System integration test System test Acceptance test Project 1 X Project 2 Test harness X X Project 3 X Stable environment Start SIT Start AT Time Test Automation Concept Release

Example 1 - Test harness Each flow have a known expected result
Data driven solution Includes test of functions, web services and interfaces Calling system Test harness Simulated system 1. Call service 2. Evaluate simulator rules 3.1. Fetch simulator response 3.2.1 Call live test system 3.2.2 Return response to harness 4.Return response to calling system Calling system Test harness Simulated system Test Automation Concept

Test automation and test objectives
Test objectives depends on the testing level. But finding bugs is one of the main objectives. A bug is Wrong or missing requirement Bad design or architecture Coding error Poor implementation The objectives of test is to make sure none of the above takes place. Test Automation Concept

Test automation and test objectives
Use cases Requirement Test cases The automated solution must fulfill the test objectives Input is functional requirements, design use cases and test cases. Output is functional verifications interpreted as check points. Test objectives Test Automation Concept

Test automation success factors
Functional components Develop functional components Decouple in- and output data from the script Central data storage Development project strategy and center of excellence Combine components to test cases by actions as input data Project In-/outdata Test automation Central data storage Test Automation Concept

Test automation GUI – functional components
Product Customer Billing Account Order Configuration Subscription Login Task Credit check Inventory Calling system External systems (with or without test harness) Test Automation Concept

Test automation GUI – input data
Each component have a defined degree of freedom Population of mandatory data Trigger of validations Start customer form Address Info BusinessID Business Name Customer Name CustomerType Populate business customer form yes no Private customer? Validate customer Type Populate private customer form Complete customer form and save Test Automation Concept Next process

Test automation GUI – output data
Results of validations Identification data Start customer form Address Info BusinessID Business Name Customer Name CustomerType Populate business customer form yes no Private customer? Validate customer Type Populate private customer form Customer ID Status Complete customer form and save Test Automation Concept Next process

Test automation GUI – output data
Customer Account [new] CustomerID [create] Components are linked by use of identification data. Output data from one component is input data for another component Test Automation Concept

Test automation GUI – test cases
Login Customer Account [new] Credit check Subscription Order Test case 1: Inventory [create] [new] Login Account [new] Customer [open] Billing Test case 2: Test Automation Concept

Test automation GUI – changes
Login Customer Account [new] ver 2.0 Account [new] Account [new] ver 1.0 Credit check Subscription Order Test case 1: Inventory [create] [new] Login Account [new] Customer [open] Billing Test case 2: Account [new] ver 2.0 ver 1.0 Test Automation Concept

Building up the components
Functionality available in system test Development of the functionality as it comes available Func. 3 Func. 4 Phase 1 Phase 2 Requirement Function 1 Function 2 Design and development Function 1 Each functionality is part of a component (or a component itself) Function 2 System test Function 1 Test Automation Concept Func. 2

Building up the components
Func. 3 Func. 4 Function 1 Func. 2 Func. 5 Func. 6 Account Customer Systems ready Testing relations of functionality Regression, E2E, Acceptance Phase 3 Phase 4 System test Func. 3 Function 5 Func. 4 Func. 6 Regression test Function 1 Func. 2 Function 3 Func. 2 Function 1 Function 5 Function 1 Func. 4 Function 5 Test Automation Concept Func. 6

Example 2 - Agile Automation
Test Design Iteration 3-4 weeks Requirement Development Release Requirement D. Sprint release A. Product backlog B. Sprint backlog C. Sprint with automation Test Automation Concept

Example 2 - Agile Automation sprint schema
Requirement Function 1 Function 3 Function 2 Function 4 Function 5 Design and development Function 1 Function 3 Function 2 Function 4 System test Function 1 Function 3 Func. 2 Func. 4 Regression test Function 1 Function 2 Test Automation Concept

The test automation concept
Lifecycle activities Test levels Test methods Development models Test automation and test objectives Test automation success factors COVERED Test Automation Concept

Walkthrough of a test automation project
Project activities Project plan - phased implementation Project organisation Business case and follow up Walkthrough Project

Test automation project - activities
Analysis Design Requirement Development Test Documentation Training Execution Customer Product Order ? ? Customer Product Order Customer Product Order Customer Product Order ver 2.0 ver 1.0 Walkthrough Project

Project plan – phased implementation
Analysis Detailed design Development System test Acceptance test Plan next phase Phase 2 Detailed design Development Week Walkthrough Project

Project organisation - Use roles to enforce competence
Project A Automation stakeholder Subject Matter Expert Project B Project C Test Automation Center Automation Project Lead Automation stakeholder Automation Solution Architect Automation Analyst Subject Matter Expert Automation Engineer Walkthrough Project

Project organisation - Test automation center
Business Objectives Strategy Management Development Organization Template Routines Processes Execution Maintenence Walkthrough Project

Test automation center
Management Test management Release management Project planning Change request Supplier Defect and Issue Management Communicating Reporting Planned activities Resolution TAC Service Management Request Result Operations ST SIT AT Project A Project B Project C Maintenance TAC Execution TAC Maintenance TAC Development

Business case Why do we need a business case?
Test automation cost time and money The automated solution must at least ensure the time spent in development, maintenance and execution is less than a manual solution would cost to accomplish the same goal. What is a test automation business case? List of all costs and savings associated with test that possibly can be covered by test automation. Not all drivers are possible to calculate directly as quality. But quality can be expressed as an estimation of the number of testing hours to ensure the desired level of quality. Additional test automation drivers as repeated executions is calculated indirectly as increased quality. How can a business case be used? Business case can be used to predict effectiveness as automation deliveries should follow the calculated schedule. ROI should be calculated at regular audits based on the business case. Walkthrough Project

Business case Total test case executions per cycle
Number of test cycles Execution time per test, h Execution cost per hour 1000 4 0.5 100 Annual cost for execution of regression test 200000 Cost for manual regression tests Cost for error handling Number of errors found per year Handling time per error (report, reproduce, administer, test data, close defect) Number of FTE's for error handling per year Error handling cost per hour Annual error handling costs 1000 2 100 200000 Error source 500 4 1 AT Production External and business costs for errors Annual error correction costs for errors Business costs for productions errors1) 100000 2 400000 Cost multiplier for Production Errors 1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is times the cost for finding them in test Grand total

Business case Year 1 120000 160000 140000 Year 2 Year 3 60% 70% 80%
Description Reduction ratio Annual reduction/cost Acc reduction Savings - Manual Test Labour Year 1 120000 160000 140000 Year 2 Year 3 60% 70% 80% 420000 260000 270000 90000 180000 Year 1 Year 2 Year 3 10% 20% 30% 540000 Savings - Error Reduction 100000 70000 Phase 1 Phase 2 Phase 3 Implementation Cost (Year 1) Year 1 proj support, SME Year 2 TAC OP Year 3 TAC OP 50000 60000 Operational costs (running) Acc savings Year 1 Year 2 Year 3 260000 370000 180000 550000 Total savings Walkthrough Project

Business case – follow up
Analysis Detailed design Development System test Acceptance test Plan next phase Phase 1 Phase 2 Week Delivered Test cases Planned Walkthrough Project Week

Test automation strategy
Description of the standard approach on how to use the tools Company specific set of rules on how automation should be done Generic solution that is not dependent on individuals Maintenance focused Encapsulated by test automation centre Enforced by framework Ensured by the automation project We will now go through the test automation project Walkthrough Project

Analysis and design Evaluation Analysis Disposition Structure
Scope Analysis Analysis based on existing documentation, test cases, use cases and requirements (static test review) Provides the prioritization to a analysis structure template Disposition Decision making: if we go live what does and does not work? What business processes are working and not working or not tested? Often details on number of test cases passed and failed on integration and system level is not vital Defect status itself often only aids prioritizing of resources/focus Structure Common and clear reporting on a level that enables correct business decisions Structured test design and coverage analysis Will control the test automation deliveries Used in test management to create common priorities for both manual and automated tests Implementation After the analysis and approved structure design, the structure may be implemented using available tools in HP QC/ALM Walkthrough Project

Analysis and design - evaluation
Value Part of business case, added value by the test case, frequency and time to execute Stable Can the stability be impacted by sources out of control? Test data Is all test data needed defined or is a manual process involved? Reusability Can the business flow be reused in the automated component solution? Well defined Is the business flow well defined and no manual decisions needed? Fully automated flow No manual interventions needed? Maturity Is the functionality in maintainable where only the test data may be changed? Defect frequency Are there a lot of issues in the functional area? Walkthrough Project

Analysis and design - the evaluation star
Value Defect frequency Stable Prio 1 test case Prio 2 test case Test data Maturity Fully automated flow Reusability Walkthrough Project Well defined

Analysis structure - disposition
what is the system used for? how is it done? which are the requirements on process level? which data conditions are covered by use cases? Use case Process Requirement Example Mobile subscription Web self care New Free surf Free SMS Data condition 1 Data condition 2 Walkthrough Project

Analysis structure - disposition
Use cases/product Requirements Customer Account Subscription Folder area Process Description Prio Create Terminate New Modify Cancel New Sales Web self care Web shop 1 1 3 1 10 Customer operations Sales CRM 1 3 1 1 3 1 15 Sales Device App 2 3 Sales Device Online 2 1 1 1 Product configurations Product Inventory 1 Walkthrough Project

Requirement and component design - refresh
All processes requires several components to enable E2E automation Each component mainly automates one business process or interface Validations may be performed in interfacing systems A component may be developed in iterations Components are used together – keyword driven Components handle unlimited data conditions – data driven Map which processes require which components Walkthrough Project

Requirement and component coverage
Components Requirements CRM Web Shop Customer Account Customer Folder area Test case Requirement Process Data condition Test case Prio Create Terminate New Modify Cancel Create TC23 Web self care R1.1 Sales Create Customer 1 X Customer op R2.3.4 Sales TC12 Create Customer 1 X TC1 Customer op R2.4.1 Terminate customer Sales 2 X Customer op R4.1.1 Create bank account Sales TC45 2 X X TC6 Customer op R4.1.1 Create invoice account Sales 1 X Walkthrough Project

Coverage analysis and status report
Subscription requirement not completely covered New subscription requirement covered by web self care Status Not covered No run Not completed Failed Passed 1 Subscription 1 4 1 5 Customer operations 1 3 1 New Not completed 2 1 Modify Failed 1 1 Cancel Not covered 1 3 Web self care 3 New Passed 3

Coverage analysis and status report, HP ALM
Coverage in ALM Test either manual, automated or business component Walkthrough Project

Component design template
Coverage Covered test cases and requirements; Requirement and component coverage Component description Prerequisities Data Environment Design description Component type Flow/use case description Detailed steps Data table Input/output data Verification element Walkthrough Project

Component data table Entity linked from previous component In O In M
Field name Input/output Mandatory/optional Description Entity linked from previous component LinkEntity_In In O FirstName In M First name of customer FamilyName In M Family name M OfficialName In Private or Business BusinessID Business ID will be validated if provided In O CustomerID_Out Out N/A The created CRM customer ID Scriptstate_Out Out N/A Status of execution Walkthrough Project

X Generic input data Linked 1. Public function GetBusinessID(oID)
2. Dim oBusinessID 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. GetBusinessID = oBusinessID 13. End Function Manual Static oBusinessID = ”AB46” oBusinessID = oID oBusinessID = Parameter(”BusinessID”) oBusinessID = DataTable.value(”BusinessID”) oBusinessID = oBusinessID & "_" & cstr(now) oBusinessID = sqlQuery(”select businessID fr…) oBusinessID=DataTable.value(”BusinessID_In”) Dynamic oBusinessID Generated Data table Parameter Query, UDF, link Dynamic Input data Static Data table value Assigned value Parameter Table data, link Walkthrough Project

Generic input data, ALM Linked Manual Static Dynamic Generated
variable Generated Data table Data table Parameter

Development Development according to design and requirements
According to company specific test automation strategy The strategy is enforced by a framework including templates and best practices The tools included in the framework must be mastered by automation engineers. Experienced automation engineers not necessarily subject matter experts of the systems. Automation tool as HP QuickTest Professional/Functional Tester facilitate script creations and object handling. Configuration management used to maintain the scripts. Walkthrough Project

Execution Test cases created Ordering procedure set up
Test data prepared Result reporting arranged Press the button Automation Engineer Functional tester Then what? X Walkthrough Project

COVERED Walkthrough of a test automation project Week Phase 1 Analysis
Detailed design COVERED Development System test Acceptance test Plan next phase Phase 2 Week Detailed design Development Walkthrough Project

Development of a framework
Code test frameworks NUnit, JUnit Test-driven development (TDD) i agile development models GUI test frameworks Automation framework infrastructure to support use of different tools Data-driven testing (DDT) Keyword-driven testing (KDT) Hybrid testing Component/modular based Model based testing Framework Development of a

1st generation script Using automation tool for capture and replay only Hardcoded data Migration test 2nd generation script Code refactoring for maintainable scripts Uses generic functions for iterations and error handling 3rd generation script Data driven test where dynamic data is taken from data tables Use of driver scripts for executing multiple test cases SME for test designs and requirement More complex scripting by developers Framework Development of a

ALM Test resources Test plan Test lab Dashboard The Tool HP QuickTest Professional/Functional tester Framework itself with test management connection ALM, keyword driven action scripts with data table, function libraries, Repositories, Business Process Testing and recovery scenario Generates code Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Recovery scenario Repository objects Application GUI layer List object Text object Table object Object recognition engine Test action script Data table M O Business ID will be validated if provided Out N/A CustomerID_Out BusinessID FamilyName FirstName In The created CRM customer ID Family name First name of customer OfficialName Private or Business Scriptstate_Out Status of execution LinkEntity_In Entity linked from previous component Function library A Function library B Framework Development of a

'***************************************************** '* Function: [FunctionName] '* '* [Description] '* Parameters: '* [Param1] - [Description] '* [Param2] - [Description] '* Returns: '* [Returnvalue] '* Author: [Author] '* See Also: '* <[DependentFunction]> '****************************************************** The Tool Code standard Templates secures a common use of functionality Unified naming convention Keep it simple Function declaration to extract common documentation Library definitions, template of standard functions Shared repositories Structured data handling, the use of input and output data Error handling Execution flow Reporting Framework Development of a

The Tool Code standard Library definitions, template of standard functions Application specific libraries Standard actions in applications should be in application library Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Framework Development of a

1. … 2. … 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. … 13. … result = Browser("Home").Page("Customer").WebElement("LoadingAnimation_left").exist(1) Do While Not( result) thisWait = thisWait +1 If thisWait > maxWait Then Err.Raise vberror + 100, " WaitLoading()", "Timeout loading page" Exit Do End If Loop Repository objects Development of a framework Extract from action script CreateCustomer Framework Development of a

1. … 2. … 3. 4. … WaitLoadingObject(Browser("Home").Page("Customer").WebElement("LoadingAnimation_left"), timeout) Extract from action script CreateCustomer 1. Public Sub WaitLoadingObject(theObject, maxWait) 2. 3. 4. 5. 6. 7. 16. 17. End Sub Dim maxWait, thisWait maxWait = theTimeout thisWait = 0 result = theObject.exist(1) Do While Not( result) … CRM_utility.qfl Framework Development of a

The Tool Code standard Library definitions, template of standard functions Application specific libraries Standard actions in applications should be in application library Component libraries enabling Configuration Management Branched Release Management for ST, AT and dev Multiple checkout and merge Only driver scripts Framework libraries handling framework and best practices Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Framework Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories Objects as well as code needs to be shared in order to avoid redundancy and maintenance overhead Local and shared repositories, application area Structured data handling, input and output data Error handling Execution flow Reporting ALM Test resources Test plan Test lab Dashboard Test action script CreateCustomer Local Repository objects UpdateCustomer CRMCustomer Framework Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Central data source Structure and data Error handling Execution flow Reporting ALM Test resources Test plan Test lab Dashboard Structure Data Data table Component A sheet M O Business ID will be validated if provided Out N/A CustomerID_Out BusinessID FamilyName FirstName In The created CRM customer ID Family name First name of customer OfficialName Private or Business Scriptstate_Out Status of execution LinkEntity_In Entity linked from previous component Component B sheet CreateCustomer sheet Master DB Data Framework Development of a

11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. On Error Resume Next Do While row < rowCount + 1 rowFunction If err.Number <> 0 Then LogError Err.Clear Else LogRun End If row = row + 1 Loop On error goto 0 CreateCustomer.qfl The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Essential in script execution; defect, automation or script language Recovery scenarios handle all unexpected events in execution, calling framework functions Handle user defined errors in framework template Execution flow Reporting 56. 57. 69. 70. 71. 72. 73. 74. 75. 76. 77. 85. Public Function rowFunction … WaitLoading timeout 'Wait for customer table retVal =waitForObjectComplete(Browser("Home")…), timeout) If not retVal Then Err.Raise vberror , "Customer not found", "CreateCustomer" End If 'Retrieve customerID in customer table customerID = GetCustomerID(privateCust) end function Customer.qfl Recovery scenario Test action script Framework library Exception Framework Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Combining components to test cases by Driver scripts and Test data Reporting Componentlibrary A Componentlibrary B ALM Test resources Test plan Test lab Dashboard Component B action script Driver action script Framework Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Use built in reporting mechanism to HP Run Results Upload attachments Report using OTA API to other ALM resources User logs Run Result Frameworklibrary ALM Test resources Test plan Test lab Dashboard Log file Framework Development of a

ALM Test resources Test plan Test lab Dashboard COVERED Recovery scenario Repository objects Application GUI layer List object Text object Table object Object recognition engine Test action script Data table M O Business ID will be validated if provided Out N/A CustomerID_Out BusinessID FamilyName FirstName In The created CRM customer ID Family name First name of customer OfficialName Private or Business Scriptstate_Out Status of execution LinkEntity_In Entity linked from previous component Function library A Function library B Framework Development of a

Agenda Session 1 The Test automation concept
Walkthrough of a test automation project Execution and maintenance of an automated solution Company test strategy Session 2 Development of a Framework Organizing your code

Short about Lemontree Swedish company established 1999
Awarded Gazelle company and Supercompany Turnover +98 MSEK 2011 AAA-rated About 100 employees, growing intensive Owners are actively working in the company, no external capital Offices in Stockholm and Oslo QM trainee program since 2012

Lemontree Industries Focus areas Bankgirot Telecom Bank & Finance
Insurance Public sector Focus areas Quality Management eGovernment Enterprise Solutions Bankgirot

And yes, we do have a framework called LTE, Lemontree Test Enabler!

Download Course Presentation

Lemontree Solution Manager
Thanks! Magnus Nilsson Lemontree Solution Manager

The difference that makes the difference

Similar presentations

Presentation on theme: "The difference that makes the difference"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

The difference that makes the difference

Similar presentations

Presentation on theme: "The difference that makes the difference"— Presentation transcript:

Similar presentations

About project

Feedback