Presentation on theme: "End to end testing (E2E): a unique test method!"— Presentation transcript:
1 End to end testing (E2E): a unique test method! Rob SmitSenior Test ManagerConsultant E2ESogeti’s Second Testing Academy 29 April 2009
2 Agenda Complex stuff, these chains Facts from the research in 2008 What is E2E? (the theoretical picture)What’s so special about E2E?The E2E project modelLet’s make an E2E test case!Some highlights from the E2E test methodReference materialIf you need us, just call us!
3 How a storm in Germany caused Complexity in chainsHow a storm in Germany causeddarkness in Morocco2006/11/06
4 Facts from the research in 2008 E2E tests exceeds planning and budget by 200 – 600%No special test strategy for E2ENo risk analysis for E2ENo ownership of the chain of processesNo knowledge how to set up an E2E test environmentNo special techniques available for E2E testingBusiness processes keep growing, and as they grow, they get more complex
5 The theoretical picture of E2E testing Interface testXABYZABInterface testTesting connectivitybetween two systemsE2E testTest type where one or more processes are tested on a uninterrupted chainof systems and platforms.E2E test:Test a process from start to finish. Test purpose: system and process integrationInterface test:Sending data from A to B, sending data from B to A
6 E2E testing is complex because the processes are complex RiskRiskRiskRiskRiskRiskRiskRiskRiskRiskE2E testing is complex because the processes are complexRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskLife insuranceinsurance policyRiskRiskBANKACCOUNTFIODCONTROLRiskRiskRiskMyWebINTERNET DOMAINRisk5xbRiskrTRAVELinsurance policyRiskRiskRisk9RiskxRiskRiskxRiskxRiskRiskRiskRiskuyRiskRiskRiskRiskRiskRiskRisk7RiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskDAMAGEinsurance policyRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskx11CURRENTACCOUNTFIN. ADMRiskRiskRiskRiskFIN. ACCOUNTINGACCOUNTINGFIN. ADMGENERALLEDGERRiskRiskRiskRiskRiskRisk6RiskRiskRiskdataRiskSTOCKinsurance policyRiskRiskSYSTEMSTANDALONERiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskRiskLIVINGinsurance policyRisk1234RiskRiskRiskRiskRiskRiskRiskRiskRiskRiskzRiskRiskRiskRiskRiskRisk10RiskRiskRiskRiskRiskRiskRiskRiskRiskRiskCARinsurance policyRiskRiskRiskDWHMISRiskRiskRiskRiskRiskRiskxpRiskRiskRiskRiskxRisk8RiskRiskRiskRiskRiskTesting software was easy in those good old days…Testing a change was simply a matter of testing the system and maybe some regression in parts that didn’t change…No more…All systems are connected, sending, receiving, processing data! Where do we begin our E2E test and where do we stop? What and where are the risks???xqx
7 Finding risks, failures and dependencies is sometimes very difficult 12V3WA4XBY5CZShould the risks in the secondary process be considered ? If so, and suppose risks are found, what do we do? Is the secondary process in scope? Can it be tested anyhow (is the knowledge accessible, do we have the budget and mandate?)Finding E2E risks is sometimes very difficult. What’s happening further up the process? And do we need the impact, or is the E2E chance sufficient?
8 End to end testing according TMap AdaptiveMake aninventory ofprocessesand systemsAnalyze E2E- risks and determine feasibilityDetermine-Scope- ClusteringStrategyInfraDesignIntegral E2E test planPlanning& BudgetingGONO GODetermine limitations and measures- Determine strategy-PlanningBudgetDesignPartial E2E test planDetailing data flowsSetting up Infra-structureBuild up E2E test environmentIntakeInfra-structureRound upExecuteSpecify logical test casesSpecify physical test cases and initial situationsDesignE2E scriptPreparationThird level: E2E test environment coordinator, responsible for the setting up and building the E2E test environment.Fourth level: E2E test specialist, responsible for the E2E test script and the test executionSecond level: E2E test manager, responsible for the partial E2E test, with it’s own test strategy, planning and budget in a E2E test planSpecial attention for the three GO/ NO GO’s! In every phase of the E2E proces, full commitment of ALL stakeholders is mandatory. Remember the Chain principle (the weakest link…)First level: E2E director, responsible for the integral E2E test plan, which also contains the business case for the E2E test
9 Example: an online store Actor(Customer)Actor(Customer)Invoice in financial systemActor(Sales emp)Actor(Shipping agent)Work backwards (against the process flow)Order in ordering systemLetter of contents in logistics systemDetermine the IT-products that supports the business processA proces is triggered by an actor and finishes at an actor
10 Example: designing a E2E test case Initial situationActionInputProcessingOutputPredicted resultFinally a test case has a predicted result. In this example, the result would be a received order (in a order management system or some kind of database. Notice: predicted result = OrderThe next thing you must describe are the actions (the process steps) that must be executed to get the predicted result (a received order).This is the first test case. This test case describes the initial situation (in which circumstances can the customer places his order?)For example: a business process with a customer and an online store. The process starts when the customer (= actor) places his order.
11 Example: designing E2E test cases End to end test caseAn E2E test case must test the process from start to finish, using data flows, products and actors. The next product is the invoice, followed by the shipment bill.So all test cases (new designed and found in testware) must be connected like this example. The predicted result of test case 1 + primary data (payment) is the initial situation of test case 2.
13 Using partial E2E tests and connecting the tests Logical starting pointLogical finishing pointSystems, processes or data flowsABCDEFGHIJKLMNOPProces flowXOOOOOXXOOOOOOOXAvailability for E2E testX: not availableO: availablePartial E2E test 1Partial E2E test 2Partial E2E test 3E2E testStart with?End with?Partial E2E test 1Result AResult prediction FPartial E2E test 3Initial situation IResult prediction HResult PResult prediction E Initial situation FResult prediction IE2E is almost impossible to execute as an integral test. Max. 5-8 systems are manageable, otherwise use partial E2E and connect!Major findings in E2E: design errors. These kind of errors are very expensive, the sooner found, the better! So start your E2E partial tests. This would be your advise ofcourse…Imagine, systems G + H are not available (in building phase for another 6 months). What to do with the E2E test? Wait until G and H are ready, or start with partial E2E testing?
14 knowledge accessibility Feasibility modelProjectenvironmentTestKnowledge &knowledge accessibilityOrganizing &Management2/2Within reach of the organizationOut of reach of the organization1= number of processes/systems per circle12Within reach of the department11/12In project621/12/1321/11/111/11/11= number of processes/ systems with complexity increasing factors22/2= number of processes/ systems with number of blocking issues
15 Fortis Insurance-ASR (launching customer in 2008) Reference materialFortis Insurance-ASR (launching customer in 2008)Elapse time User Acceptance Test: 40% lessNo more incidents caused by design errorsE2E in critical path: not happened sinceKPN Telecom (pilot started in march 2009)Still in design phase of the E2E test cases, but already many possible risks and design issues are foundAchmea (pilot started in march 2009)Clear view on risks in the chainsProject quality already much better
16 The business processes are increasingly complex ConclusionThe business processes are increasingly complexSupporting IT-processes evidently grow in complexityThe E2E test method provides the method, activities, tools and examples to handle that complexity
17 Thank you for your attention E2E test method has many new techniquesWhite paper release May 2009Book July 2009English version August 2009Course materials available May 2009Commercial materials available May 2009Support for any E2E issue: any time!ContactsRob SmitRob Baarda
18 Sogeti’s 2nd Testing Academy 29 April 2009 www.sogeti.com
19 System integration test is executed in 3 steps E2E in V-modelwish, law, policychance, problemuse&maintenancerequirementsacceptancetestsE2E testfunctionaldesignInterfacetesttechnicaldesignsystem-testsInterface testdeveloperstestsrealisationSITSystem integration test is executed in 3 stepsInterface test: by comparing the interface documentation.Moment of execution: design phaseInterface test: dynamic testing the technical and functional interface behaviour.Moment of execution: system testEnd to end test: dynamic testing the business processes over the integrated systems and platforms. Moment of execution: in system test or in acceptance test. Preferably as soon as possible!