1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group

2 Abstract There is great pressure upon developers today to improve productivity and effectiveness. To achieve this there is a move away from the traditional structured methodologies towards more dynamic, iterative and RAD approaches. This is being combined with Object and Component based techniques, and delivered with a new generation of IDE’s, to produce thin client, web based, voice and data products. So how do we test it?  Propose Time Boxed Testing  Outline an approach  Discuss the problems  Give a flavour of the fun ?

3 Agenda m Development Lifecycle m Testing Lifecycle m Testing Helix m Testing Stage by Stage m Traceability m Testing Process m How to sell time boxed testing m Lessons Learnt

4 The Project m Telephony based project m 2 types of proprietary hardware m Client Server running Windows NT m Interfaces to legacy systems m Rational Rose case tool (using UML) m Objectory process m Visual Studio m Was considering Java & thin clients m Incremental/Iterative development

5 Incompatible Models? Sys Test Classic V Model Rqmts. Analysis Design Build Unit Test Int Test UAT Breadth Depth Iterative (RAD) Model

6 Difficult to Represent Test Stages Breadth Depth unit Component Integration Component Integration System Acceptance Component Integration

7 Project Lifecycle ­ ¬ ®...

8 Testing Lifecycle ­ ¬ ®...

9 P D C E A dual spiral life cycle ? ­ ¬ ® Plan Design Construct Execute Rqmts Analysis Design Build... R A D B

10 Testing Lifecycle m Unit Testing m Component Integration Testing m System Testing m Acceptance Testing m Year 2000 Millenium Compliance m Testing Processes and Activities m Tools & Traceability m Resources m Planning

11 Scope of Test Phases Component Integration Testing Unit Testing System Testing User Acceptance Testing Conical Model Iterative Release

12 Tuned Test Lifecycle System Testing - Non functional Requirements - Performance Analysis Component Integration Testing Unit Testing Iterative Release User Acceptance Testing - Scenario Testing - Acceptance Criteria Testing Activities Brought Forward

13 Unit Testing Static Analysis Dynamic Analysis Unit Testing m Iterative Time-Boxed Activity m Specific to delivered components m ‘White Box’ testing techniques employed m External Interfaces tested m Use of tools for Static & Dynamic analysis Quality Assurance External Interface Testing Unit Integration Testing Sub Unit Testing

14 Unit Testing Sub Unit Package Unit Integration Testing Sub Unit Sub Unit Package Unit Unit Testing Sub Unit Sub Unit Sub Unit Sub Unit Sub Unit Sub Unit Testing Sub Unit Stubs, Drivers, Harnesses Previously tested & checked-in

15 Component Integration Testing Component Integration Non Functional Requirements Functional Requirements Regression Test Component Integration Testing m Iterative Time-Boxed Activity m Testing prioritised by Business m End to End process threads where possible m Contains elements of following test phases m Build regression test packs Scenario Regression Test Performance Analysis Package Integration

16 Package Component Integration Testing Package Package Integration must precede Component Integration Component Package Step 1 - Package Integration

17 Component Integration Testing Component Drop1 Component Drop2 Component... Step 2 - Component Integration

18 Component Integration Testing Automated Regression Test Pack ¬ Component ­ ®... Component Drop by Drop

19 System Testing Functional Requirements Regression Test Non Functional Requirements Performance Analysis Design Verification System Testing m One off activity m Structured testing techniques m End to End process threads m Ties together the development iterations m Regression test already constructed

20 Acceptance Testing Acceptance Criteria Testing Scenario Regression Testing Business Process Testing Usability Testing Acceptance Testing Data Take On & Conversion Testing m One off activity m Test that the computer system integrates with business processes m Structured testing techniques m End to End process threads m Regression test already constructed

21 Y Millennium Compliance 6 Date Scenarios 2 Business Tests This Century Across Century Boundary Wholly next Century Jan 1stFeb 29thDec 31st Financial Year ? Fiscal Periods ? m Separate test runs m Carried out in sequence m Some tests can be exercised post Y2000

22 Testing Lifecycle Traceability High Level Function Functional Requirement Use Case Scenario Test Case Test Script Non FR Development Repository Test Planning User Requirements

23 Testing Processes and Activities Test Management Test Reporting Monitor Measure Manage Exit Test Stage Sign off Test Planning Test Design Test Preparation Analysis of Change Test Execution Test Automation Regression Test Quality Interface Problem Mgmt Conf. Mgmt. Interface Process Entry Activity

24 Processes m Essential l Configuration Management l Impact Analysis using a Test Planning repository l Quality Management m Good practice l Fault Reporting - on the Intranet - Live l Gathering metric information

25 Tools m Full set of testing tools is required l Static analysis l Dynamic Analysis l Interactive debugging l Test Automation/Capture Replay essential l Test Planning Repository l Performance monitoring (stress/load) l Fault Reporting system

26 Monitoring

27

28

29

30 Test Resources Unit Testing Component Integration Testing System Testing User Acceptance Testing TEST TEAM Testers Business Users Developers Business Users Developers Secondment

31 Test Planning Definition Test Strategy Testing Process Development Non Functional Testing Test drop ¬ Test drop ­ Test drop ® Decision Test Approach Delivery System Test Acceptance Test Y2000 Test Time-boxed Testing

32 Selling the Approach m Target the following; l Test Team l Systems Architect l Development team l Standards & Methods l Business Representatives l Project Sponsor m Via l Animated Presentation l Test Strategy document

33 So how easy was this?

34 So What went well? m Systems Architect liked it  m The presentation was well received 

35 What was difficult? m Initial cool response from other members of the test team m Development Team were uneasy l Dev. Why test anyway, our code is wonderful and our test harnesses will do all the testing? m Business Representatives were unhappy about; l Anything less than 100% testing l Bringing their effort forward in the development lifecycle l The difference between the ‘Use Case’ modeling they had done, and the test planning that they would need to carry out

36 What was difficult? (2) m Standards & Methods Saw nothing wrong with their approach, it was either; structured in which case their approach worked fine, or it was incremental, in which case their approach worked fine. They did not do iterative development, but if they did their approach would still work. This took several meetings to gain approval. m Project Sponsor l Cancelled the project. (Time Boxed Testing was not cited as the reason !!!)

37 So what fun did we have ?

38 What did I learn from this? m Organisations change their development approach more quickly than their testing approach m There is a reluctance to do anything less than 100% testing m Testing is a key and integral part of any development lifecycle m The more you think about it, the more sense Time Boxed Testing makes

39 Contact Information Graham Thomas OSI the management support company Phone: Copies of slides including animated gifs at