1. ©2003 Systeme Evolutif LtdSlide 1 we’ll change the way you think about testing What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director Systeme Evolutif Systeme Evolutif Limited 9 Cavendish Place London W1G 0QD, UK Tel: +44 (0)20 7636 6060 email: paulg@evolutif.co.uk http://www.evolutif.co.uk/

2. ©2003 Systeme Evolutif LtdSlide 2 we’ll change the way you think about testing Paul Gerrard Paul is the Technical Director and a principal consultant for Systeme Evolutif. He has conducted consultancy and training assignments in all aspects of Software Testing and Quality Assurance. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Co-Programme Chair for the BCS SIG in Software Testing, a member of the BCS Software Component Test Standard Committee and Founding Chair of the IS Examination Board (ISEB) Certification Board for a Tester Qualification whose aim is to establish a certification scheme for testing professionals and training organisations. He is a regular keynote speaker at both development and testing seminars and conferences in Europe, US and Australia, and won the "Best Presentation" award at EuroSTAR ’95 and "Best Presentation of 2002" award for the BCS SIGIST. Paul's book, Risk-Based E-Business Testing, written with Neil Thompson, is the first book focusing on risk-based test approaches and aimed at E-Business.

3. ©2003 Systeme Evolutif LtdSlide 3 we’ll change the way you think about testing Themes “[A cynic is] A man who knows the price of everything but the value of nothing” Oscar Wilde (1854-1900), Lady Windermere’s Fan. “To win without risk is to triumph without glory” Pierre Corneille (1606-84), The Cid.

4. ©2003 Systeme Evolutif LtdSlide 4 we’ll change the way you think about testing Agenda Let’s play Marketing Quantity, Quality, Process or Product – Where is the Value? The Most Valuable Product of Testing is Project Intelligence Influences on the Value of Testing Increasing the Value of Testing Process “Improvements” and the Value of Testing Summary.

5. ©2003 Systeme Evolutif LtdSlide 5 we’ll change the way you think about testing Let’s Play Marketing We need to consider: – Who are the customers of testing? – What exactly does the customer value? – How can we increase the value of our testing? – How can we increase their appreciation of our testing? – How do we select an “improvement” to make? – How do we judge the value of an “improvement”?

6. ©2003 Systeme Evolutif LtdSlide 6 we’ll change the way you think about testing Quantity, Quality, Process or Product – Where is the Value?

7. ©2003 Systeme Evolutif LtdSlide 7 we’ll change the way you think about testing Mine clearing According to captured records, there are 100 mines in that minefield We have identified and removed seven mines I crossed it once but don’t remember how Is it safe?

8. ©2003 Systeme Evolutif LtdSlide 8 we’ll change the way you think about testing Test activity and information I ran some tests – But I can’t remember what happened or how I could repeat those tests – Was this testing valuable? I ran a hundred tests and the system failed 23 times – I can’t remember how I did it or where the system failed – Was this testing valuable? The ACTIVITY of testing only has VALUE if it generates INFORMATION (this is the deliverable).

9. ©2003 Systeme Evolutif LtdSlide 9 we’ll change the way you think about testing How much detail? Which is best? – “We ran lots of tests” – “We found lots of faults” – “10% of our tests caused the system to fail” – “This test case caused a failure” – “This set of test steps caused the system to fail at precisely this point, in precisely this way…” Test records should be detailed enough to plan, design, implement, execute tests and interpret test results and reproduce failure – AND NO MORE.

10. ©2003 Systeme Evolutif LtdSlide 10 we’ll change the way you think about testing Quantity or quality? Which would you pay more for? – 100 page test plan or 200 page test plan? – 100 incident reports or 200 incident reports? – 100 requirements faults or 100 coding faults? – 100 random tests or 100 ‘designed’ tests? Quantity is one thing, but the nature and quality of the information and the way it was prepared makes a huge difference to its value.

11. ©2003 Systeme Evolutif LtdSlide 11 we’ll change the way you think about testing Testing generates “project intelligence” If you don’t know where you are, a map won’t help If you don’t know the status of the deliverables, the best plan won’t help you manage your project A project is like driving off-road in the dark – Testing provides the headlights for the journey Testing provides data on the status of deliverables and generates “PROJECT INTELLIGENCE” Stakeholders and management need intelligence to make decisions, command and control the project.

12. ©2003 Systeme Evolutif LtdSlide 12 we’ll change the way you think about testing Does testing ‘change’ anything? Product Testing Product (unchanged) Valuable Stuff Testing doesn’t change the product in any way. We do it to get the ‘valuable stuff’.

13. ©2003 Systeme Evolutif LtdSlide 13 we’ll change the way you think about testing What is the valuable stuff? Test deliverables (infrastructure, test plans, specifications, scripts, data, results etc…) TEST RECORDS – test documentation (but do only testers read it?) PROJECT INTELLIGENCE – rarely documented as such though.

14. ©2003 Systeme Evolutif LtdSlide 14 we’ll change the way you think about testing Military intelligence?

15. ©2003 Systeme Evolutif LtdSlide 15 we’ll change the way you think about testing The Most Valuable Product of Testing is Project Intelligence

16. ©2003 Systeme Evolutif LtdSlide 16 we’ll change the way you think about testing Documentation PRODUCTS – Test Strategy – Test plans, designs, specifications – Procedures/scripts, expected results – Actual results – Incident reports – Log – End of Phase Report – Etc. TANGIBLE, BUT LESS VALUABLE.

17. ©2003 Systeme Evolutif LtdSlide 17 we’ll change the way you think about testing Test infrastructure PRODUCTS – Test process – Configured test hardware – Test databases – Tools – Manual and automated tests – Skilled resources – Etc… TANGIBLE, USUALLY AN OVERHEAD – (unless these are required deliverables).

18. ©2003 Systeme Evolutif LtdSlide 18 we’ll change the way you think about testing TEST RECORDS DATA – How the product SHOULD work (test specs) – How the product ACTUALLY works (test results) – Where the product is STABLE (incident trends) – Where the product is UNSTABLE (incident trends) – What they have COVERED in the product (test log) – What they have NOT COVERED (test plan – test log) – How (and where) the product FAILS (incidents) TANGIBLE AND VALUABLE, BUT… – (valuable to testers, developers, NOT management).

19. ©2003 Systeme Evolutif LtdSlide 19 we’ll change the way you think about testing PROJECT INTELLIGENCE KNOWLEDGE OF PROJECT RISKS/BENEFITS – RISKS » Requirements are poor and/or unstable » Less of the product is ready, product is not as good as planned » They should have been less ambitious, done it differently » The project will slip – BENEFITS » Some features/benefits are available » Other features/benefits are blocked by risk uncertainty INTANGIBLE AND VALUABLE – Crucial to management and stakeholders – Intelligence often arrives TOO LATE TO BE OF USE.

20. ©2003 Systeme Evolutif LtdSlide 20 we’ll change the way you think about testing Where the value of testing lies LOW VALUE – Documentation – Test infrastructure **UNLESS these are specific objectives or deliverables of project HIGH VALUE – Test Records – Project Intelligence We can increase the value of testing MOST by: Improved INTELLIGENCE Providing INTELLIGENCE EARLIER.

21. ©2003 Systeme Evolutif LtdSlide 21 we’ll change the way you think about testing Influences on the Value of Testing

22. ©2003 Systeme Evolutif LtdSlide 22 we’ll change the way you think about testing Intangible value, tangible cost Testers think testing is good and has undeniable value, but we are not independent! Not everyone thinks that (software suppliers, in-house developers?) If you are not a tester, the value seems intangible – But the cost IS tangible – Exercise, flossing, house maintenance, life insurance, testing can all be put off till another day – If you don’t do it now you store up problems for later A bit like a lifestyle choice – it’s human nature to put things off.

23. ©2003 Systeme Evolutif LtdSlide 23 we’ll change the way you think about testing Key factors in the value of testing Adding value depends on how well we achieve and communicate: 1. Coverage 2. Incidents 3. Risk-based testing 4. Benefits-based testing Time to deliver that value depends on the test process (more later) Willingness of the organisation to accept and act upon project intelligence is key (more later).

24. ©2003 Systeme Evolutif LtdSlide 24 we’ll change the way you think about testing Coverage Breadth of coverage – Are all aspects of the product tested? Depth of coverage – Are critical aspects of the product tested more thoroughly? Can the measure be applied systematically? Is the measure relevant to the faults to be detected or the risks of concern? Can we communicate coverage in a meaningful, understandable way?

25. ©2003 Systeme Evolutif LtdSlide 25 we’ll change the way you think about testing Incidents Are the failures of most concern being found and logged as incidents? Can these failures be reproduced easily? Are the incidents being addressed in the right order? Are the incidents of most concern being resolved quickly?

26. ©2003 Systeme Evolutif LtdSlide 26 we’ll change the way you think about testing Risk-based testing Are product risks identified early in the project? Are stakeholders involved in the decision to rank these risks? Is the decision to include risks in scope for testing done by consensus? Is the decision to de-scope risks done by consensus? Is test design driven by the need to address risk? Is test progress reporting based on risks addressed?

27. ©2003 Systeme Evolutif LtdSlide 27 we’ll change the way you think about testing Benefit-based testing Does the project have cardinal objectives (benefits) defined? Are all risks related to the benefits? Are tests related to the risks that block the benefits? Is test progress reporting based on the benefits that are threatened or available?

28. ©2003 Systeme Evolutif LtdSlide 28 we’ll change the way you think about testing Increasing the Value of Testing

29. ©2003 Systeme Evolutif LtdSlide 29 we’ll change the way you think about testing Contribution of the test process Does testing generating valuable intelligence? Does testing generate intelligence early enough? Is testing flexible enough to react to new risks? Are the test communication channels clear? Improvement types: – Effectiveness – Efficiency – Perceived value of testing.

30. ©2003 Systeme Evolutif LtdSlide 30 we’ll change the way you think about testing Effectiveness improvements Improvements to “intellectual” test activities enhance the quality of intelligence Implementing these activities earlier make intelligence available earlier Examples: – Formal test design improves coverage, and improves the quality of test records – Improvements that promote earlier test activities tend to make the value available earlier (e.g. more reviews).

31. ©2003 Systeme Evolutif LtdSlide 31 we’ll change the way you think about testing Efficiency improvements These reduce the cost of doing testing Improvements to the mechanical or clerical aspects of our test process make us more efficient (but don't affect the value of testing) For example – Using a tool to automate tests might make it cheaper to do a task, but does it add value? – It might save time, allowing us to do other useful activities Ultimately, effectiveness improvements are more valuable.

32. ©2003 Systeme Evolutif LtdSlide 32 we’ll change the way you think about testing Organisational culture Willingness to accept and act on the intelligence that testers generate is key Primarily a cultural issue If the intelligence isn’t valued by the culture - it doesn't matter what value it has We need to EDUCATE our customers.

33. ©2003 Systeme Evolutif LtdSlide 33 we’ll change the way you think about testing Increasing the perceived value of testing No such thing as “absolute value” of testing We must work on the PERCEIVED VALUE to the customers of our testing: 1.Improve the value in line with their expectations – Provide richer, deeper, focused, relevant information – “Create more complex, subtle, interesting wine” 2.Increase their appreciation of what we do – Educate them to understand testing products – “Train their palate to detect the complex flavours” Improvements must address both issues.

34. ©2003 Systeme Evolutif LtdSlide 34 we’ll change the way you think about testing Process “Improvements” and the Value of Testing

35. ©2003 Systeme Evolutif LtdSlide 35 we’ll change the way you think about testing Test process or test product assessment? Assessments of the “Test Process” may miss the point “Process healthchecks” are useless because they don’t focus on the VALUE of test deliverables Assessment of the perceived value of Test Products must be a key driver We must focus on process changes that: – add value to the test products – enhance the perceived value of test products – make test products available earlier.

36. ©2003 Systeme Evolutif LtdSlide 36 we’ll change the way you think about testing The challenges of testing improvement Barriers are organisational, personal, not technical: – changing management perceptions so they support testing and improvement – overcoming management and practitioner resistance to change – design and implementation of workable processes and management controls People, not tools, implement test strategies Change is easier if people see VALUE being added.

37. ©2003 Systeme Evolutif LtdSlide 37 we’ll change the way you think about testing Problems or symptoms? Need to separate problems from symptoms: – “Management doesn't understand the value of testing” – “The cost of testing is high but difficult to pin down” – “Developers, testers, users never trained” – “The quality of the product delivered into testing is poor, so takes longer to system test” – “The perceived value of testing is low” Need to understand symptoms to identify the source of problem and fix the problem Improvements often tackle the symptoms only.

38. ©2003 Systeme Evolutif LtdSlide 38 we’ll change the way you think about testing Right improvement mix is key A mix of improvements is most likely to be required: – management awareness – tester training – developer training – improved definition of the test stages and their objectives – measurement of the quality of the product at each stage – etc. etc.

39. ©2003 Systeme Evolutif LtdSlide 39 we’ll change the way you think about testing Constraints and expectations Not all improvements a good idea straight away – some improvements are expensive – some save time, but bring dramatic change – some improve the quality of the testing, but take longer to implement – some tidy up ‘loose ends’ but have negligible value Very few improvements save time, improve quality, cause minimal change and pay back after two weeks Need to set/agree clear management expectations.

40. ©2003 Systeme Evolutif LtdSlide 40 we’ll change the way you think about testing Do tools increase the value of testing? With tool vendors present, I’d better be careful! Need to distinguish between – Quantity, quality and the information provided A tool might allow you do to testing faster, but it probably won’t increase the value of tests It might make a task interesting, so a valuable task gets done It might give you more time to do other valuable tasks.

41. ©2003 Systeme Evolutif LtdSlide 41 we’ll change the way you think about testing To Summarise…

42. ©2003 Systeme Evolutif LtdSlide 42 we’ll change the way you think about testing How to increase the value of testing The value of testing lies in the project intelligence it generates Produce documentation with purpose: – To allow testers to do their job (THE MEANS) – To communicate intelligence (THE END) Process ‘improvements’ selected to deliver better intelligence, earlier, faster and cheaper You need to train your customers to appreciate the project intelligence that testers produce.

43. ©2003 Systeme Evolutif LtdSlide 43 we’ll change the way you think about testing Themes revisited “A poor tester knows the challenge and cost of testing but not its value” – Don’t be cynical – testing isn’t just a costly activity to be gotten through, testers should aim to deliver value “To win without risk is to triumph without glory” – Risks are inevitable - the testers’ role isn’t to avoid them, but inform management and reduce their uncertainty – Satisfaction (and recognition) comes from overcoming risks, not doing the mundane tasks of development – Make your management aware of software product risks and report progress towards their goal, not just incidents. Testers deal in success, not just failure.

44. ©2003 Systeme Evolutif LtdSlide 44 we’ll change the way you think about testing www.evolutif.co.uk www.riskbasedtesting.com What Is the Value of Testing and How Can We Increase It? Paul Gerrard