Presentation is loading. Please wait.

Presentation is loading. Please wait.

The difference that makes the difference. Make your test automation project a success story Magnus Nilsson Lemontree.

Similar presentations


Presentation on theme: "The difference that makes the difference. Make your test automation project a success story Magnus Nilsson Lemontree."— Presentation transcript:

1 The difference that makes the difference

2 Make your test automation project a success story Magnus Nilsson Lemontree

3 Agenda The Test automation concept Walkthrough of a test automation project – Execution and maintenance of an automated solution – Test strategy Development of a Framework – Organizing your code

4 The test automation concept Fundamentals behind test; a short review of – Lifecycle activities – Test levels – Test methods – Development models Test automation and test objectives Test automation success factors Test Automation Concept

5 Requirement Project Business process Test Defect Management Release Maintenance Monitoring Architecture Design Development Test Automation Concept Test Test as part of the lifecycle activities

6 Connecting the activities Test levels Integration test Acceptance test Unit test Test Automation Concept Test Development System test Unit test System test Integration test Acceptance test Functional Non functional Test types

7 Connecting the activities Test methods Black box Test White box Code Black box – Testing delivered functionality – GUI test White box – Testing internal structures Code – Testing modules, classes, libraries, functions on code level Test Automation Concept

8 Connecting the activities Development models Waterfall Development AgileIterative Scrum Waterfall – The lifecycle approach Iterative – Planned repeated cycles Agile – Iterative and development adaptive to changes – Practicing continuous Integration, CI Scrum – Sprint-based development based on both on agile and iterative development methods Test Automation Concept

9 The challenge of test Development Test Test Automation Concept Code White box Black box Testing methods Unit test Integration test System test Acceptance test Test levels Development models Waterfall Agile

10 Test automation is all about unifying test. Environment Automation TestHarness Development – Increase code quality by implementing unit tests – Development methodology TDD – Automated builds, CI Environment – Test environment creation – Test data management, TDM Harness – Stubbing interfaces Test – Scripted tests Test interfaces GUI Performance Test Automation Concept

11 Development Test The challenge of test, connecting… Unit test CodeIntegration test GUI White box System test Acceptance test Black box Test Automation Concept Testing methods Test levels Development models TDD TDM Test automation Test Harness CI Automation Waterfall Agile

12 Example 1 - Test harness Enables test before interfaces exists or are available, agile development but also project delivery regardless development model System test Project 1 Test Automation Concept System integration test Acceptance test Project 2 Project 3 Stable environment Time Release Start SIT Start AT X X X Test harness X X Test Harness

13 Example 1 - Test harness Each flow have a known expected result Data driven solution Includes test of functions, web services and interfaces Calling system Test harness Simulated system Calling system Test harness Simulated system 1. Call service 2. Evaluate simulator rules Call live test system Return response to harness 3.1. Fetch simulator response 4.Return response to calling system Test Harness Test Automation Concept

14 Test automation and test objectives Test objectives depends on the testing level. But finding bugs is one of the main objectives. A bug is – Wrong or missing requirement – Bad design or architecture – Coding error – Poor implementation The objectives of test is to make sure none of the above takes place. Test Automation Concept

15 Test automation and test objectives The automated solution must fulfill the test objectives Input is functional requirements, design use cases and test cases. Output is functional verifications interpreted as check points. Test objectives Use cases Requirement Test cases Test Automation Concept

16 Test automation success factors Develop functional components Decouple in- and output data from the script Central data storage Development project – strategy and – center of excellence Combine components to test cases by actions as input data Functional components Test automation In-/outdata Central data storage Project Test Automation Concept

17 Test automation GUI – functional components Product Customer Billing Account Order Configuration Subscription Login Task Credit check Product Inventory Calling system External systems (with or without test harness) Test Automation Concept

18 Each component have a defined degree of freedom Population of mandatory data Trigger of validations Test automation GUI – input data Next process Start customer form Private customer? Populate private customer form Address Info BusinessID Business Name Customer Name Validate customer Validate customer Validate Type Validate Type Complete customer form and save CustomerTy pe Populate business customer form yes no Test Automation Concept

19 Results of validations Identification data Test automation GUI – output data Next process Start customer form Private customer? Populate private customer form Address Info BusinessID Business Name Customer Name Customer ID Customer ID Customer Status Customer Status Validate customer Validate customer Validate Type Validate Type Complete customer form and save CustomerTy pe Populate business customer form yes no Test Automation Concept

20 Test automation GUI – output data [create] Customer Account [new] CustomerID Components are linked by use of identification data. Output data from one component is input data for another component Test Automation Concept

21 Test automation GUI – test cases Login Customer Account [new] Inventory Login Credit check [create] Subscription Order [new] Account [new] Customer [open]Billing Test case 1: Test case 2: Test Automation Concept

22 Test automation GUI – changes Login Customer Inventory Login Credit check [create] Subscription Order [new] Account [new] Customer [open]Billing Test case 1: Test case 2: Account [new] ver 2.0 ver 1.0 Account [new] ver 1.0 Account [new] ver 2.0 Account [new] ver 2.0 Test Automation Concept

23 Building up the components Functionality available in system test – Development of the functionality as it comes available Requirement Design and development System test Phase 1 Phase 2 Function 1 Function 2 Function 1 Function 2 Function 1 Func. 2 Each functionality is part of a component (or a component itself) Func. 3 Func. 4 Test Automation Concept

24 Func. 3 Func. 4 Function 1 Func. 2 Func. 5 Func. 6 Account Customer Building up the components Systems ready – Testing relations of functionality Regression, E2E, Acceptance System test Phase 3 Phase 4 Function 5 Func. 6 Regression test Function 1 Func. 2 Function 3 Func. 4 Function 1 Func. 2 Function 5 Func. 6 Func. 3 Func. 4 Function 1 Function 5 Test Automation Concept

25 Example 2 - Agile Automation Test Development Requirement Design Requirement A. Product backlog B. Sprint backlog C. Sprint with automation Release Iteration 3-4 weeks D. Sprint release Agile Test Automation Concept

26 Example 2 - Agile Automation sprint schema Requirement Design and development System test Regression test Sprint 0 Sprint 1 Sprint 2 Function 1 Function 2 Function 1 Function 2 Function 1 Function 2 Function 1 Func. 2 Function 3 Function 4 Function 3 Function 4 Function 3 Func. 4 Function 5 Test Automation Concept

27 Lifecycle activities Test levels Test methods Development models Test automation and test objectives Test automation success factors COVERED The test automation concept Test Automation Concept

28 Agenda The Test automation concept Walkthrough of a test automation project – Execution and maintenance of an automated solution – Test strategy Development of a Framework – Organizing your code

29 Walkthrough of a test automation project Walkthrough Project Project activities Project plan - phased implementation Project organisation Business case and follow up

30 ? ? ? ? ? ? ? ? ? Test automation project - activities Analysis Design Requirement Development Test Documentation Training Execution Customer Product Order Customer Product Order Customer Product Order Customer Product Order ver 2.0 ver 1.0 Walkthrough Project

31 Project plan – phased implementation Walkthrough Project Analysis Detailed design Development System test Acceptance test Phase 1 Week Plan next phase Phase 2 Detailed design Development

32 Automation stakeholder Subject Matter Expert Test Automation Center Project organisation - Use roles to enforce competence Automation Engineer Automation Analyst Automation Solution Architect Automation Project Lead Walkthrough Project Project A Automation stakeholder Subject Matter Expert Project B Automation stakeholder Subject Matter Expert Project C Automation stakeholder Subject Matter Expert

33 Project organisation - Test automation center Development Execution Maintenence Walkthrough Project

34 Communicating Reporting Planned activities Test automation center TAC Service Management Management Test management Release management Project planning Change request TAC Execution TAC Execution TAC Maintenance TAC Maintenance TAC Development TAC Development Request Result Operations ST SIT AT Project A Project B Project C Maintenance Supplier Defect and Issue Management Resolution

35 Business case Why do we need a business case? – Test automation cost time and money – The automated solution must at least ensure the time spent in development, maintenance and execution is less than a manual solution would cost to accomplish the same goal. What is a test automation business case? – List of all costs and savings associated with test that possibly can be covered by test automation. – Not all drivers are possible to calculate directly as quality. But quality can be expressed as an estimation of the number of testing hours to ensure the desired level of quality. – Additional test automation drivers as repeated executions is calculated indirectly as increased quality. How can a business case be used? – Business case can be used to predict effectiveness as automation deliveries should follow the calculated schedule. – ROI should be calculated at regular audits based on the business case. Walkthrough Project

36 Business case Cost for manual regression tests Cost for error handling External and business costs for errors Production AT Number of errors found per year Handling time per error (report, reproduce, administer, test data, close defect) Number of FTE's for error handling per year Error handling cost per hour Annual error handling costs Error source Total test case executions per cycle Number of test cycles Execution time per test, h Execution cost per hour Annual cost for execution of regression test Grand total Annual error correction costs for errors Business costs for productions errors 1) Cost multiplier for Production Errors 1) Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is times the cost for finding them in test

37 Business case Savings - Manual Test Labour Savings - Error Reduction Implementation Cost (Year 1) Operational costs (running) Description Reduction ratio Reduction ratio Annual reduction/cost Acc reduction Phase 1 Phase 2 Phase 3 Total savings Year 1 Year 2 Year 3 10% 20% 30% Year Year 2 Year 3 60% 70% 80% Year 1 proj support, SME Year 2 TAC OP Year 3 TAC OP Acc savings Year 1 Year 2 Year Walkthrough Project

38 Business case – follow up Analysis Detailed design Development System test Acceptance test Plan next phase Phase 1 Phase 2 Detailed design Development Week Delivered Planned Test cases Week Walkthrough Project

39 Test automation strategy Description of the standard approach on how to use the tools Company specific set of rules on how automation should be done Generic solution that is not dependent on individuals Maintenance focused Encapsulated by test automation centre Enforced by framework Ensured by the automation project Walkthrough Project We will now go through the test automation project

40 Analysis and design Evaluation – Scope Analysis – Analysis based on existing documentation, test cases, use cases and requirements (static test review) – Provides the prioritization to a analysis structure template Disposition – Decision making: if we go live what does and does not work? – What business processes are working and not working or not tested? – Often details on number of test cases passed and failed on integration and system level is not vital – Defect status itself often only aids prioritizing of resources/focus Structure – Common and clear reporting on a level that enables correct business decisions – Structured test design and coverage analysis – Will control the test automation deliveries – Used in test management to create common priorities for both manual and automated tests Implementation – After the analysis and approved structure design, the structure may be implemented using available tools in HP QC/ALM Walkthrough Project

41 Analysis and design - evaluation Value – Part of business case, added value by the test case, frequency and time to execute Stable – Can the stability be impacted by sources out of control? Test data – Is all test data needed defined or is a manual process involved? Reusability – Can the business flow be reused in the automated component solution? Well defined – Is the business flow well defined and no manual decisions needed? Fully automated flow – No manual interventions needed? Maturity – Is the functionality in maintainable where only the test data may be changed? Defect frequency – Are there a lot of issues in the functional area? Walkthrough Project

42 Analysis and design - the evaluation star Value Stable Test data Reusability Well defined Defect frequency Maturity Fully automated flow Prio 1 test case Prio 2 test case Walkthrough Project

43 Analysis structure - disposition what is the system used for? – how is it done? which are the requirements on process level? » which data conditions are covered by use cases? Use case Process Requirement Data condition 1 Data condition 2 Walkthrough Project Example Mobile subscription Web self care New Free surf Free SMS

44 Analysis structure - disposition Requirements Folder area Process Description Prio Use cases/product Customer Account Subscription Create Terminate New Sales Web self care 1 Modify Cancel 3110Web shop Sales Customer operations CRM SalesDevice23App SalesDevice2111Online 1 ProductInventory1 Product configurations Walkthrough Project

45 Requirement and component design - refresh All processes requires several components to enable E2E automation Each component mainly automates one business process or interface Validations may be performed in interfacing systems A component may be developed in iterations Components are used together – keyword driven Components handle unlimited data conditions – data driven Map which processes require which components Walkthrough Project

46 Requirement and component coverage Requirements Folder area Requirement Process Data condition Test case Data condition Test case Prio Components Customer Account Customer Create Terminate New Create Sales Web self care R1.1 1 Modify Cancel Create Customer Sales Customer op R Sales2 Terminate customer Sales2 Create bank account 1 CRM Create Customer Web Shop X Customer op R2.4.1 X Customer op R4.1.1 X X Sales Customer op R4.1.1 Create invoice account X Test case TC23 TC12 TC1 TC45 TC6 Walkthrough Project X

47 Coverage analysis and status report Subscription Customer operations New Modify Cancel Web self care New No run Not completed Failed Passed Status Not covered Subscription requirement not completely covered New subscription requirement covered by web self care Passed Failed Not completed

48 Coverage analysis and status report, HP ALM Coverage in ALM Test either – manual, – automated or – business component Walkthrough Project

49 Component design template Coverage – Covered test cases and requirements; Requirement and component coverage Component description – Prerequisities Data Environment – Design description Component type Flow/use case description Detailed steps – Data table Input/output data – Verification element Walkthrough Project

50 Component data table M O Business ID will be validated if provided Mandatory/opt ional Description Field name Input/output Out N/A CustomerID_Out BusinessID FamilyName FirstName In M N/AThe created CRM customer ID Family name First name of customer M OfficialName In Private or Business Out Scriptstate_Out Status of execution LinkEntity_In In O Entity linked from previous component Walkthrough Project

51 Generic input data oBusinessID 1. Public function GetBusinessID(oID) 2. Dim oBusinessID GetBusinessID = oBusinessID 13. End Function Static Parameter Data table Generated Dynamic Query, UDF, link Dynamic Input data Static Data table value Assigned value Parameter Assigned value Table data, link Manual X Linked oBusinessID = AB46 oBusinessID = oID oBusinessID = Parameter(BusinessID) oBusinessID = DataTable.value(BusinessID) oBusinessID = oBusinessID & "_" & cstr(now) oBusinessID = sqlQuery(select businessID fr…) oBusinessID=DataTable.value(BusinessID_In) Walkthrough Project

52 Generic input data, ALM variable Static Parameter Data table Generated Dynamic Manual Linked Data table

53 Development Development according to design and requirements According to company specific test automation strategy The strategy is enforced by a framework including templates and best practices The tools included in the framework must be mastered by automation engineers. Experienced automation engineers not necessarily subject matter experts of the systems. Automation tool as HP QuickTest Professional/Functional Tester facilitate script creations and object handling. Configuration management used to maintain the scripts. Walkthrough Project

54 Execution Test cases created Ordering procedure set up Test data prepared Result reporting arranged Press the button Walkthrough Project Then what? X Automation Engineer Functional tester

55 Analysis Detailed design Development System test Acceptance test Plan next phase Phase 1 Phase 2 Detailed design Development Week COVERED Walkthrough of a test automation project Walkthrough Project

56 Agenda The Test automation concept Walkthrough of a test automation project – Execution and maintenance of an automated solution – Test strategy Development of a Framework – Organizing your code

57 Development of a framework Code test frameworks – NUnit, JUnit – Test-driven development (TDD) i agile development models GUI test frameworks Automation framework infrastructure to support use of different tools Data-driven testing (DDT) Keyword-driven testing (KDT) Hybrid testing Component/modular based Model based testing Framework Development of a

58 1 st generation script – Using automation tool for capture and replay only – Hardcoded data – Migration test 2 nd generation script – Code refactoring for maintainable scripts – Uses generic functions for iterations and error handling 3 rd generation script – Data driven test where dynamic data is taken from data tables – Use of driver scripts for executing multiple test cases – SME for test designs and requirement – More complex scripting by developers Development of a framework Framework Development of a

59 Development of a framework The Tool – HP QuickTest Professional/Functional tester – Framework itself with test management connection ALM, keyword driven action scripts with data table, function libraries, Repositories, Business Process Testing and recovery scenario – Generates code Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Repository objects Application GUI layer Application GUI layer List object Text object Table object Object recognition engine Data table M O Business ID will be validated if provided OutOut N/AN/A Cust omerI D_O ut BusinessIDBusinessID Fam ilyN ame First Na me InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M Offic ialN ame InIn Private or Business OutOut Script state _Out Status of execution Link Entit y_In InIn O Entity linked from previous component Function library A Function library B ALM Test resources Test plan Test lab Dashboard Recovery scenario Test action script Test action script Framework Development of a

60 Framework Development of a Development of a framework The Tool Code standard – Templates secures a common use of functionality – Unified naming convention – Keep it simple – Function declaration to extract common documentation Library definitions, template of standard functions Shared repositories Structured data handling, the use of input and output data Error handling Execution flow Reporting '***************************************************** '* Function: [FunctionName] '* '* [Description] '* '* Parameters: '* '* [Param1] - [Description] '* [Param2] - [Description] '* '* Returns: '* '* [Returnvalue] '* '* Author: [Author] '* '* See Also: '* '******************************************************

61 Development of a framework The Tool Code standard Library definitions, template of standard functions – Application specific libraries – Standard actions in applications should be in application library Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Framework Development of a

62 Framework Development of a Development of a framework 1. … 2. … … 13. … result = Browser("Home").Page("Customer").WebElement("LoadingAnimation_left").exist(1) Do While Not( result) result = Browser("Home").Page("Customer").WebElement("LoadingAnimation_left").exist(1) thisWait = thisWait +1 If thisWait > maxWait Then Err.Raise vberror + 100, " WaitLoading()", "Timeout loading page" Exit Do End If Loop Repository objects Extract from action script CreateCustomer

63 Framework Development of a Development of a framework 1. … 2. … … WaitLoadingObject(Browser("Home").Page("Customer").WebElement("LoadingAnimation_left"), timeout) Extract from action script CreateCustomer 1. Public Sub WaitLoadingObject(theObject, maxWait) End Sub Dim maxWait, thisWait maxWait = theTimeout thisWait = 0 result = theObject.exist(1) Do While Not( result) … CRM_utility.qfl

64 Development of a framework The Tool Code standard Library definitions, template of standard functions – Application specific libraries – Standard actions in applications should be in application library – Component libraries enabling Configuration Management Branched Release Management for ST, AT and dev Multiple checkout and merge Only driver scripts – Framework libraries handling framework and best practices Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting Framework Development of a

65 The Tool Code standard Library definitions, template of standard functions Shared repositories – Objects as well as code needs to be shared in order to avoid redundancy and maintenance overhead – Local and shared repositories, application area Structured data handling, input and output data Error handling Execution flow Reporting Development of a framework ALM Test resources Test plan Test lab Dashboard Test action script CreateCustomer Test action script CreateCustomer Local Repository Local Repository objects Test action script UpdateCustomer Test action script UpdateCustomer CRMCustomer Repository CRMCustomer Repository objects Framework Development of a

66 The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data – Central data source – Structure and data Error handling Execution flow Reporting Development of a framework Data table Component A sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component Component B sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component CreateCustomer sheet M O Business ID will be validated if provided OutOut N/AN/A CustomerI D_Out B us in es sI D FamilyNa me FirstNam e InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M OfficialNa me InIn Private or Business OutOut Scriptstate _Out Status of execution LinkEntity _In InIn O Entity linked from previous component ALM Test resources Test plan Test lab Dashboard Structure Data Master DB Data Framework Development of a

67 Framework Development of a The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling – Essential in script execution; defect, automation or script language – Recovery scenarios handle all unexpected events in execution, calling framework functions – Handle user defined errors in framework template Execution flow Reporting Development of a framework Recovery scenario Test action script Test action script Framework library Exception On Error Resume Next Do While row < rowCount + 1 rowFunction If err.Number <> 0 Then LogError Err.Clear Else LogRun End If row = row + 1 Loop On error goto 0 CreateCustomer.qfl Public Function rowFunction … WaitLoading timeout 'Wait for customer table retVal =waitForObjectComplete(Browser("Home")…), timeout) If not retVal Then Err.Raise vberror , "Customer not found", "CreateCustomer" End If 'Retrieve customerID in customer table customerID = GetCustomerID(privateCust) … end function Customer.qfl

68 The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow – Combining components to test cases by Driver scripts and Test data Reporting Development of a framework Componentli brary A Componentli brary B ALM Test resources Test plan Test lab Dashboard Component B action script Driver action script Framework Development of a

69 Framework Development of a ALM Test resources Test plan Test lab Dashboard The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting – Use built in reporting mechanism to HP Run Results – Upload attachments – Report using OTA API to other ALM resources – User logs Development of a framework Log file Run Result Frameworkli brary

70 Development of a framework Repository objects Application GUI layer Application GUI layer List object Text object Table object Object recognition engine Data table M O Business ID will be validated if provided OutOut N/AN/A Cust omerI D_O ut BusinessIDBusinessID Fam ilyN ame First Na me InIn InIn InIn M N/AN/A The created CRM customer ID Family name First name of customer M Offic ialN ame InIn Private or Business OutOut Script state _Out Status of execution Link Entit y_In InIn O Entity linked from previous component Function library A Function library B ALM Test resources Test plan Test lab Dashboard Recovery scenario Test action script Test action script Framework Development of a COVERED

71 Agenda Session 1 – The Test automation concept – Walkthrough of a test automation project Execution and maintenance of an automated solution Company test strategy Session 2 – Development of a Framework Organizing your code

72 Short about Lemontree Swedish company established 1999 Awarded Gazelle company and Supercompany Turnover +98 MSEK 2011 AAA-rated About 100 employees, growing intensive Owners are actively working in the company, no external capital Offices in Stockholm and Oslo QM trainee program since 2012

73 Lemontree Industries Telecom Bank & Finance Insurance Public sector Focus areas Quality Management eGovernment Enterprise Solutions Bankgirot

74 And yes, we do have a framework called LTE, Lemontree Test Enabler!

75 Download Course Presentation

76 Thanks! Magnus Nilsson Lemontree Solution Manager


Download ppt "The difference that makes the difference. Make your test automation project a success story Magnus Nilsson Lemontree."

Similar presentations


Ads by Google