2. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-1 Exploratory Test Management Outline Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management Exploratory Test Team Management Risk-Based Test Management Session Based Test Management 1. 2. 3. 4. 5.

3. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-2 Exploratory Test Management These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by- sa/2.0/legalcode.

4. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-3 5. Exploratory Test Management 5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management 1. 2. 3. 4. 5.

5. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-4 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Traditional Test Teams… …are built on… Available knowledge and skills Business needs Employee’s ambitions …provide a career path through… Execution  coordination  planning Defect reporting  analysis  Strategy This might not work for Exploratory Testing

6. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-5 How to build and lead an effective test team Slides and Questionnaire from Lloyd Roden, Grove Consultants, Lloyd@grove.co.ukLloyd@grove.co.uk Copyright © 2002 Lloyd Roden, Grove Consultants (Lloyd@grove.co.uk)Lloyd@grove.co.uk 1. 2. 3. 4. 5.

7. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-6 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The questionnaire Things to note No right or wrong answer Try not to think too much Helps us assess our strengths Can be used with other management questionnaires (Belbin, 16PF etc) Plot values on the graph Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

8. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-7 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The questionnaire - how to complete Friendly __ Approachable __ Casual __ Open __ Unstructured __ Social __ Intuitive __ Random __ Warm __ Perceptive __ __ Formal __ Retiring __ Business Like __ Guarded __ Organised __ Introvert __ Logical __ Focused __ Cool __ Insensitive __ X - Axis XXXXXX6XXXXXX6 XXXXXXXX Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

9. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-8 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The questionnaire - how to complete To the point __ Challenging __ Quick __ Insistent __ Lively __ Impatient __ Adventurous __ Confronting __ Competitive __ Strong Minded __ __ Indirect __ Accepting __ Leisurely __ Thoughtful __ Relaxed __ Patient __ Cautious __ Receptive __ Co-operative __ Analytical __ Y - Axis XX2XX2 XXXXXXXXXXXXXXXX Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

10. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-9 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The Grid X (6,2) Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk 1 2 3 4 5 6 7 8 9 10 9 8 7 6 4 3 2 1 0 0

11. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-10 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The Model The Pragmatist The Pioneer The Analyst The Facilitator Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

12. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-11 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Pragmatic’ Style Tester Likes strategic / goals positive results / brief practical efficiency tasksDislikes indecision vagueness time-wasting unproductive Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

13. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-12 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The ‘Pragmatic’ style tester will… be good for setting and monitoring short/long term goals for the team be good at documenting factual ‘test reports’ remain positive through pressure be keen to adopt ‘Most Important Tests’ first principle be a strong driving force - ensure a task is done want to implement efficiency into the team be self-motivated and task oriented will make quick decisions enjoy challenging testing tasks Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

14. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-13 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Pioneer’ Style Tester Likes new / ideas change openness results/efficiency involving others risksDislikes standards detail ‘norm’ paper-work Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

15. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-14 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The ‘Pioneer’ style tester will… be good at ‘ad-hoc’ testing / bug hunting / error- guessing/ exploratory testing be good at challenging and improving things to make more efficient and effective enjoy “GUI” type testing/lateral tester have good ideas be good at brainstorming Test Conditions share ideas about different ways to approach testing identify and take necessary risks when required have creative test ideas - how to find more faults Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

16. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-15 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Analysing’ Style Tester Likes accuracy attention to detail proof standards reliable all alternativesDislikes new / change untested / risks brief / speed letting go Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

17. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-16 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The ‘Analysing’ style tester will… be good at defining and documenting test cases be good at producing test standards and procedures analyse problems and finding root cause produce work which is accurate and complete enjoy logical tests scenarios provide proof when faults are found document thorough test reports complete work regardless of what it takes challenge requirements Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

18. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-17 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Key words for the ‘Facilitating’ Style Tester Likes networking positive team oriented consensus / sharing building bridges status quoDislikes pressure / deadlines confrontation isolation dictated Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

19. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-18 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The ‘Facilitating’ style tester will… be good in a RAD environment or a ‘buddy’ test team often ask opinion before raising issues be good at documentation co-operate well with other departments often see the ‘other side’ be good at defusing ‘us’ v ‘them’ syndrome be popular make things happen - eventually! will provide support in testing to other team members Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

20. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-19 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Tester Style - patterns usually operate within a certain boundary what if you are on the line/centre flexible between styles can be difficult to manage opposites repel maybe reasons for team tension! analysts & pragmatists tend towards ‘Tasks’ facilitators & pioneers tend towards ‘People’ Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

21. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-20 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management The Ideal Test Team… A MIXTURE IS THE BEST BUT IT DEPENDS... Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)www.grove.co.uk

22. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-21 5. Exploratory Test Management 5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management 1. 2. 3. 4. 5.

23. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-22 1. 2. 3. 4. 5. ”Ordinary” Project Plan including Testing SpecificationDesignCodingTest Plan: SpecificationDesignCoding Test In the Real World:

24. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-23 1. 2. 3. 4. 5. ”Incremental” Project Plan with Testing Spec.DesignCodingTest Spec.DesignCodingTest Spec.DesignCodingTest Plan: In the Real World: Spec.DesignCodingTest? Spec.DesignCodingTest? Spec.DesignCodingTest......

25. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-24 1. 2. 3. 4. 5. Why Risk Based Testing? It’s about controlling: Time Resources New Technology Lack of knowledge Lack of experience Quality Demands Scope ???

26. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-25 1. 2. 3. 4. 5. What is Risk? “A risk is an unwanted event that has negative consequences.” Shari Lawrence Pfleeger In other words, a risk is a problem waiting to happen. Shari Lawrence Pfleeger (2000)

27. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-26 1. 2. 3. 4. 5. What is Risk Management? Plans to avoid these unwanted events or, if they are inevitable, minimize their negative consequences. Shari Lawrence Pfleeger Shari Lawrence Pfleeger (2000)

28. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-27 1. 2. 3. 4. 5. Problems vs. Risk Problem (Issue): Something that has or will happen Risk Something that might happen (in the future)

29. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-28 1. 2. 3. 4. 5. Types of Risk Project / Process Risk Resourcing Planning Contracts Etc. Business / Product Risk Stability Performance Quality / Errors (Quality Risk = Potential Errors) Etc.

30. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-29 1. 2. 3. 4. 5. Business Risk – Why is it so difficult? (1) Flaw in the Pentium chip (1994) 6 million PCs sold, potential USD300 loss per chip Intel’s risk impact USD 1.8 billion Intel: ”Average” computer user: wrong answer every 27.000 years ”Heavy user”: every 270 years Conclusion: The flaw is not meaningful to most users IBM: ”Average” computer user: a problem every 24 days A large company (500 PCs): 20 problems per day! Conclusion: stopped selling Pentium PCs IBM’s assessment 400.000 times worse than Intel’s! Pfleeger (2000)

31. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-30 1. 2. 3. 4. 5. Business Risk – Why is it so difficult? (2) Evaluating the risk of an accident at a small ammonia storage plant (1988 to 1990) 11 EU countries (national teams) + several private firms The national teams’ assessment varied by a factor of 25.000, reaching wildly different conclusions. Pfleeger (2000)

32. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-31 1. 2. 3. 4. 5. How to identify risk? 1. A loss associated with the event 2. The likelihood that the event will occur 3. The degree to which we can change the outcome

33. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-32 1. 2. 3. 4. 5. What is Risk-Based Testing? ”Risk-based testing carries at least two major meanings:” Risk-Based Test Management To determine what things to test next (prioritisation) Doing Risk analysis for the Purpose of Finding Errors Lessons Learned in Software Testing, Cem Kaner et al 2001b

34. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-33 1. 2. 3. 4. 5. RBT: The purpose of finding errors Make a prioritised list of risks. Perform testing that explores each risk. As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop. James Bach 1999a

35. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-34 1. 2. 3. 4. 5. Risk-Based Test Management 1.Define all requirements to be tested 2.Based on risk assessment – prioritise the requirements 3.Plan and define tests according to requirement prioritisation (coverage to be defined in test plan) 4.Execute test according to prioritisation and acceptance criteria (as defined in the test plan)

36. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-35 1. 2. 3. 4. 5. Risk-Based Test Management Statistical Risk Analysis (Hans Schaefer) Failure Mode and Effect Analysis (FMEA) Software Reliability engineering, John D. Musa Risk based test strategy (TMap, Iquip.nl)) Decision Theory (Bayesian Belief Nets) Rational Unified Process (RUP)...and many, many more

37. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-36 1. 2. 3. 4. 5. Statistical Risk Analysis – Theory Equation: Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f RBT Management - Statistic Risk Analysis – Hans Schaefer Schaefer 1998, Amland 1999

38. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-37 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Statistical Risk Analysis – Matrix RBT Management - Statistic Risk Analysis - Hans Schaefer Schaefer 1998, Amland 1999 Weighted

39. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-38 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Statistical Risk Based Testing Plan: Identify Elements to be Tested Logical or physical Functions, Modules etc. Identify Risk Indicators What is important to predict the probability of faults? Identify Cost (consequence) of faults Identify Critical Elements I.e. functions, tasks, activities etc. based on Risk Analysis (Indicators and Cost) Execute Improve the Test Process and Organization: Schedule and Track Schaefer 1998, Amland 1999

40. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-39 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Example – Statistical Risk Analysis Matrix RBT Management - Statistic Risk Analysis Interest Calc. Close Account Customer Profitablty C(s)C(c)Avrg. New Func. Desgn Qual.Size Com- plexity Weigh. Sum Risk Exposure 5513 333 132 211,5 233337 222331 332341 111 62 61,5 Other Probability Factors might include: Function Points, Frequency of Use etc. Cost *Probability =Re Schaefer 1998, Amland 1999

41. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-40 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Risk Based Testing - Reporting LowMedium High Consequence Probability

42. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-41 5. Exploratory Test Management 5.1 Exploratory Test Team Management 5.2 Risk-Based Test Management 5.3 Session Based Test Management 1. 2. 3. 4. 5.

43. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-42 1. 2. 3. 4. 5. Introducing the Test Session 1) Charter 2) Time Box 3) Reviewable Result 4) Debriefing vs. From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

44. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-43 1. 2. 3. 4. 5. Charter – summary… “Architecting the Charters” i.e. Test Planning Brief information / guidelines on: Mission: Why do we test this? What should be tested? How to test (approach)? What problems to look for? Might include guidelines on: Tools to use Specific Test Techniques or tactics to use What risks are involved Documents to examine Desired output from the testing Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

45. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-44 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Time Box: Focused test effort of fixed duration Brief enough for accurate reporting. Brief enough to allow flexible scheduling. Brief enough to allow course correction. Long enough to get solid testing done. Long enough for efficient debriefings. Beware of overly precise timing. Short: 60 minutes (+-15) Normal: 90 minutes (+-15) Long: 120 minutes (+-15) From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

46. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-45 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Reviewable Result: The session sheet Charter #AREAS Start Time Tester Name(s) Breakdown #DURATION #TEST DESIGN AND EXECUTION #BUG INVESTIGATION AND REPORTING #SESSION SETUP #CHARTER/OPPORTUNITY Data Files Test Notes Bugs #BUG Issues #ISSUE CHARTER ----------------------------------------------- Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREAS OS | Windows 2000 Menu | View Strategy | Function Testing Strategy | Functional Analysis START ----------------------------------------------- 5/30/00 03:20 pm TESTER ----------------------------------------------- Jonathan Bach TASK BREAKDOWN ----------------------------------------------- #DURATION short #TEST DESIGN AND EXECUTION 65 #BUG INVESTIGATION AND REPORTING 25 #SESSION SETUP 20 From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

47. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-46 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Debriefing: Measurement begins with observation The manager reviews session sheet to assure that he understands it and that it follows the protocol. The tester answers any questions. Session metrics are checked. Charter may be adjusted. Session may be extended. New sessions may be chartered. Coaching / Mentoring happens. From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

48. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-47 1. 2. 3. 4. 5. Exploratory Test Management Summary Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management Exploratory Test Team Management Risk-Based Test Management Session Based Test Management

49. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-48 Exercise 5 Based on available information; Discuss how you would manage and track a test team working on testing StarOffice. What would your ideal test team look like? What management strategy would you use? Why? Outline a risk matrix Select a few areas or functions Identify probability indicators How to define Consequence of an error?

50. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-49 Summary (1) Strengths: Customer-focused, risk-focused Takes advantages of each tester’s strengths Responsive to changing circumstances Well managed, it avoids duplicative analysis and testing High Bug find rates Blind Spots: The less we know, the more we risk missing Limited by each tester’s weaknesses (can be mitigated with careful management) This is skilled work, juniors aren’t very good at it Copyright © 1996 – 2002 Cem Kaner

51. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-50 Summary (2) Exploratory testing in pairs, is a very effective test approach Use it as a complementary testing approach (it depends…) Skilled testers can become very good exploratory testers. Novice testers need mentoring to do exploratory testing.

52. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-51 Learn more… Test Training www.testingeducation.org Exploratory Testing: Rapid Software Testing, by James Bach, www.satisfice.com james@satisfice.comwww.satisfice.com Black Box Software Testing, by Cem Kaner, www.kaner.com kaner@kaner.comwww.kaner.com kaner@kaner.com Amland Consulting, www.amland.no, stale@amland.nowww.amland.nostale@amland.no People Issues, test techniques, inspections: Grove Consultants, www.grove.co.ukwww.grove.co.uk + many, many more

53. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-52 Presentation - Summary Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management Introduction: Exploratory Testing and Risk, Why and when to use Exploratory Testing. Test Management, Test Techniques and Test Process ET Planning, Documentation and Execution. How to do Exploratory Testing Exploratory Test Styles How to do Exploratory Testing, continued Exploratory Test Management How to Manage Exploratory Testing and Teams 1. 2. 3. 4. 5.

54. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-53 References/Bibliography (1) Amland, S., 1999. Risk Based Testing and Metrics, EuroSTAR '99 Preceding, Barcelona, Spain, http://www.amland.no/articles Amland, S., 2000. ”Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Volume 53, Issue 3, 15.9.2000, Elsevier publishing, UK. Argus, C. & Johnson, B., Ad Hoc Software Testing; Exploring the Controversy of Unstructured Testing, http://www.testingcraft.com/ad_hoc_testing.pdfhttp://www.testingcraft.com/ad_hoc_testing.pdf Bach, J., 1999a. Risk-Based Testing. How to conduct heuristic risk analysis, Software Testing & Quality Engineering Magazine, November/December 1999, vol. 1, issue 6, http://www.stqemagazine.com. See also satisfice.com Bach, J., 1999b. What is Exploratory Testing? www.satisfice.comwww.satisfice.com Bach, Jonathan, 2000, Session-Based Test Management, STQE Magazine, 11/00, www.satisfice.comwww.satisfice.com Beizer, Boris, Software Testing Techniques, Van Nostrand Reinhold. NY. 1990. Beizer, Boris, 1995, Black Box Testing, Wiley

55. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-54 References/Bibliography (2) Bereza-Jarocinski, B., 2000, Test and Decision Theory, Enea Data AB, bogb@enea.se, +46 50 709 714 293 (www.bbj.com.pl)www.bbj.com.pl Black, Rex, 1999. Managing the Testing Process, Microsoft Press. Buwalda, Hans, Janssen, Dennis, and Pinkster, Iris, 2001. Integrated Test Design & Automation Using The TestFrame Method, Addison Wesley, Copeland, Lee, Exploratory Planning, Sep. 3, 2001, StickyMinds.com Collard, Ross, 2002, to be published, rcollar@attglobal.net. Fenton, N.E. & Pfleeger, S.L., 1997. Software Metrics, a rigorous & practical approach, 2nd edition, International Thomson Computer Press. Fewster, M., Graham, D., 1999, Software Test Automation: Effective Use of Text Execution Tools, Addison-Wesley Gerrard, P., 1999, Risk-Based Testing, TEST Congress, London, UK, www.evolutif.co.uk Gilb, T., Principles of Software Engineering Management, Addison-Wesley, Wokingham

56. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-55 References/Bibliography (3) Kaner, C., Falk, Nguyen, 1999, Testing Computer Software, John Wiley & Sons Kaner, C., 2000. Rethinking Software Metrics, Evaluating measurement schemes, Software Testing & Quality Engineering Magazine, March/April 2000, vol. 2, issue 2. Kaner, C., 2001a, Black-Box Software Testing, training at Satisfice July 2001. Kaner, C., Bach, J., Pettichord, B., 2001b, Lessons Learned in Software Testing, John Wiley & Sons; http://www.testinglessons.com/, ISBN: 0471081124http://www.testinglessons.com/ Kaner, C., Bach, J. 2001c, Exploratory Testing in Pairs, presentation at STAREast, Orlando, FL, www.kaner.comwww.kaner.com Karolak, Dale Walter, “Software Engineering Risk Management”, IEEE Computer Society Press, 1996. Keefer, G., 2002. Extreme Programming Considered Harmful for Reliable Software Development, www.avoca-vsm.comwww.avoca-vsm.com Keith, Geordie, 2002, “All Hands on Deck: How we used our whole company to test”, STQE Magazine, vol. 4, issue 4, July / August 2002. Lyndsay, James and Eeden, Niel van, 2002, “Adventures in Session-Based Testing”, paper at StickyMinds.com presented at EuroSTAR 2002

57. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-56 References/Bibliography (4) Marick, Brian, 1995, The Craft of Software Testing, Prentice Hall PTR, ISBN 0- 13-177411-5 Myers, Glenford, 1979. The Art of Software Testing, John Wiley & Sons Neumann, P.G., Computer-Related Risks, Addison-Wesley/ACM Press, ISBN 0- 201-55805-X, 1995, http://www.csl.sri.com/users/neumann/http://www.csl.sri.com/users/neumann/ Nguyen, 2000, Testing Applications on the WEB, John Wiley & Sons Ottevanger, Dr. I.B., 1999. A Risk-Based Test Strategy, Presented at STARWest 1999, http://www.sqe.com/startwest. Part of TMAP, IQUIP Informatica B.V. PO Box 263, 1110 AG Diemen, The Netherlands (www.iquip.nl).www.iquip.nl Pfleeger, S.L., 2000. Risky Business: what we have yet to learn about software risk management, Journal of Systems and Software, issue 11, 2000, Elsevier Publishing, UK. Robinson, H., Microsoft, Exploratory Modeling, http://www.testingcraft.com/exploratory-robinson.html, home page: http://www.geocities.com/model_based_testing/ http://www.testingcraft.com/exploratory-robinson.html http://www.geocities.com/model_based_testing/ Roden, Lloyd, 2001. How to build and lead an effective test team, tutorial EuroSTAR 2001, Grove Consultants, www.grove.co.uk, Lloyd@grove.co.ukwww.grove.co.ukLloyd@grove.co.uk

58. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-57 References/Bibliography (5) Schaefer, H., 1998. Surviving under time and budget pressure, keynote STAR West '98, schaefer@c2i.net, http://www.sqe.com/starwest, http://home.c2i.net/schaefer/ http://home.c2i.net/schaefer/ Våga, J., Amland, S., 2002. High Speed Web Testing, contribution to the book ” Software Quality and Testing in Internet Times”, Editor: Dirk Meyerhoff et al, Springer-Verlag, Berlin Heidelberg, Germany 2002, ISBN:3-540-42632-9 Whittaker, J. and Jorgensen, A., 1999. Why software fails. ACM Software Engineering Notes, July. http://se.fit.edu/papers/ and www.stickyminds.comhttp://se.fit.edu/papers/ www.stickyminds.com Whittaker, J. and Jorgensen A., 2002, “How to Break Software: A Practical Guide to Testing”, Addison-Wesley, ISBN: 0-201-79619-8, presented at EuroSTAR 2000 Øvstedal, E. Ø. and Stålhane, T., 1992. A goal oriented approach to software testing, Reliability Engineering and System Safety. Elsevier Science Publishers Ltd., UK.

59. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-58 Links www.stickyminds.com www.satisfice.com www.kaner.com www.testingeducation.org www.pettichord.com www.amland.no home.c2i.net/schaefer/ www.bbj.com.pl http://www.stqemagazine.com http://www.testinglessons.com/ http://www.context-driven-testing.com/ www.grove.co.uk

60. ET Workshop v. 1.20 - Test Management©2002 Amland Consulting5-59 Ståle Amland, Amland Consulting Hulda Garborgsv. 2, N-4020 STAVANGER Norway Phone: +47 905 28 930 Fax: +47 51 58 55 24 e-mail: stale@amland.nostale@amland.no WEB: www.amland.nowww.amland.no Contact Details 1. 2. 3. 4. 5.