1. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-1 These slides are distributed under the Creative Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by- sa/2.0/legalcode.

2. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-2 Planning, Documentation and Execution Outline Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management How much planning is done with ET? How do you execute the tests with ET? What kind of documentation is produced with ET? How to use Pair Testing with ET? How do you use ET with eXtreme Programming (XP)? 1. 2. 3. 4. 5.

3. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-3 3. Exploratory Test Planning, Execution and Documentation. 3.1 Planning, Tasks and Documentation 3.2 Exploratory Testing in Pairs 3.3 Exploratory Testing with eXtreme Programming 1. 2. 3. 4. 5.

4. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-4 ET Planning “We plan as much as we can (based on the knowledge available), when we can (based on the time available), but not before.” “When we apply Exploratory Planning to Testing, we create Exploratory Testing.” Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Lee Copeland, Exploratory Planning, 2001

5. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-5 1. 2. 3. 4. 5. Exploratory Testing Tasks Explore Execute Tests Design Tests Product (coverage) Techniques Quality (evaluation) Discover the elements of the product. Discover how the product should work. Discover test design techniques that can be used. Decide which elements to test. Observe product behavior. Speculate about possible quality problems. Evaluate behavior against expectations. Configure & operate the product. Select & apply test design techniques. Testing notes Tests Problems Found From Rapid Software Testing, copyright © 1996-2002 James Bach

6. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-6 1. 2. 3. 4. 5. Preparation Decompose the Product into elements Areas / Groups of functions or features you expect to be able to test within a day or two Identify the Quality Criteria Ref. Satisfice Testing Model Check your toolbox Test techniques Application To be tested Decomposed Into elements Element Each element is expected to be tested within a day or two.

7. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-7 1. 2. 3. 4. 5. ET Planning – Architecting Charters Pick an element to test Decompose further – into smaller elements Each new element should be tested within an hour or two Prepare a charter for each element Element Charter Element To be tested Decomposed Into smaller elements /Charters Each Charter is expected to be tested within an hour or two. Charter

8. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-8 1. 2. 3. 4. 5. Doing Exploratory Testing Keep your mission clearly in mind. Keep notes that help you report what you did, why you did it, and support your assessment of product quality. Keep track of questions and issues raised in your exploration (logging and debriefing). To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time. From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

9. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-9 1. 2. 3. 4. 5. Charter: A clear mission for the session A charter may suggest what should be tested, how it should be tested, and what problems to look for. A charter is not meant to be a detailed plan. General charters may be necessary at first: “Analyze the Insert Picture function” Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.” From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

10. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-10 1. 2. 3. 4. 5. Lateral Thinking but periodically take stock of your status against your mission Let yourself be distracted… ‘cause you never know what you’ll find… From Rapid Software Testing, copyright © 1996-2002 James Bach

11. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-11 1. 2. 3. 4. 5. Charter – summary… “Architecting the Charters” i.e. Test Planning Brief information / guidelines on: Mission: Why do we test this? What should be tested? How to test (approach)? What problems to look for? Might include guidelines on: Tools to use Specific Test Techniques or tactics to use What risks are involved Documents to examine Desired output from the testing Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

12. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-12 1. 2. 3. 4. 5. Taking Notes Test Coverage Outline/Matrix Evaluation Notes Risk/Strategy List Test Execution Log Issues, Questions & Anomalies It would be easier to test if you changed/added… How does … work? Is this important to test? How should I test it? I saw something strange… From Rapid Software Testing, copyright © 1996-2002 James Bach Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

13. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-13 1. 2. 3. 4. 5. ET Execution - Sessions Session Basic testing work unit Uninterrupted e-mail, meetings, telephone calls etc. Reviewable A report should be produced Chartered Mission associated with this session; What are we testing? What are we looking for? Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

14. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-14 1. 2. 3. 4. 5. Exercise 1a Prepare a test charter for testing one selected area of StarOffice 5.2 Bullet and Numbering or Tables Work in pairs.

15. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-15 1. 2. 3. 4. 5. ET Documentation Planning: Charters – overview of what to test (plan) Migh be a flip chart on the wall Missions – What are we looking for? Execution Notes – what happened during testing? What did I do? Why did I do it? Used to assess product quality after test. Data files – input data used for testing Bug reports – enough details to recreate the test / bug Track questions and Issues Debriefing Charter/ Mission Notes, Risks, Issues, Questions and Errors Execution: Pairs & Sessions Debriefing

16. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-16 1. 2. 3. 4. 5. Key Ideas of Exploratory Testing The results from the tests you design and execute influence the next test you will choose to design and execute. You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave. You test what you know about, and you are alert for clues about behaviors and aspects of the product that you don’t yet know about. From Rapid Software Testing, copyright © 1996-2002 James Bach

17. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-17 1. 2. 3. 4. 5. Challenges of High Accountability Exploratory Testing Architecting the system of charters (test planning) Making time for debriefings Getting the metrics right Creating good test notes Keeping the technique from dominating the testing Maintaining commitment to the approach For example session sheets and metrics see http://www.satisfice.com/sbtm From Rapid Software Testing, copyright © 1996-2002 James Bach

18. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-18 1. 2. 3. 4. 5. Problems to be Wary of Habituation may cause you to miss problems. Lack of information may impair exploration Expensive or difficult product set-up may increase the cost of exploring Exploratory feedback loop may be too slow Old problems may pop up again and again High MTBF may not be achievable without well defined test cases and procedures, in addition to exploratory approach. From Rapid Software Testing, copyright © 1996-2002 James Bach

19. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-19 1. 2. 3. 4. 5. Exercise 1b Prepare a test charter for testing one selected area of MiniTest Work in pairs.

20. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-20 3. Exploratory Test Planning, Execution and Documentation. 3.1 Planning, Tasks and Documentation 3.2 Exploratory Testing in Pairs 3.3 Exploratory Testing with eXtreme Programming 1. 2. 3. 4. 5.

21. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-21 ET and Pair Testing Observations made at Satisfice, several independent companies and Florida Tec – spectacular results: Testing workgroups: High productivity High creativity Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

22. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-22 What is Pair Testing? Two testers and (typically) one machine Typically (as in XP): Pairs work together voluntarily. One person might pair with several others during a day A given testing task is the responsibility of one person, who recruits one or more partners (one at a time) to help out. Some stable pairs have worked together for years One tester strokes the keys (keyboard might pass back and forth in a session) while the other suggests ideas or tests, pay attention and takes notes, listen, asks questions, grabs reference material etc. Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

23. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-23 A Paired Testing Session Start with a charter Testers might operate from a detailed project outline, pick a task that will take a day or less Might (instead or also) create a flipchart page that outlines this session’s work or the work for the next few sessions An exploratory testing session lasts about 60-90 minutes The charter for a session might include what to test, what tools to use, what testing tactics to use, what risks are involved, what bugs to look for, what documents to examine, what outputs are desired etc. Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

24. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-24 Benefits of Pair Testing Generate more ideas. More fun. Helps the tester stay on task. Also, the fact that two are working together limits the willingness of others to interrupt them… Better Bug Reporting. Great Training. Additional technical benefit. Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

25. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-25 Risks and Suggestions Senior / junior pairs: The pairs are partners; the junior tester is often the one at the keyboard, and he / she is always allowed to try out his / her own ideas. Accountability must always belong to one person. Some people are introverts. They need time to work alone and recharge themselves for group interaction. Some people have strong opinions and don’t work well with others. Coaching may be essential. Learn from XP experiences, Laurie Williams of NCSU (collaboration.csc.ncsu.edu/laurie/) Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

26. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-26 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Pair Programming vs. Testing Pair Testing is developed independently of paired programming, but many of the same problems and benefits apply. The eXtreme Programming community has a great deal of experience with paired work and offers many lessons: Kent Beck, Extreme Programming Explained Ron Jeffries, Ann Anderson & Chet Hendrickson, Extreme Programming Installed Laurie Williams of NCSU does research in pair programming. For her publications, see http://collaboration.csc.ncsu.edu/laurie/ http://collaboration.csc.ncsu.edu/laurie/ Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)

27. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-27 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management Pair Programming – Problems? Paper by Gerold Keefer, AVOCA GmbH: “So far there is no evidence about the usefulness of this practice [pair programming] in general,….In addition there are several reports that indicate that pairing leads to faster exhaustion… On the other hand no one will deny that pairing in certain situations, such as educational settings or debugging sessions, is good common sense.” Alternative to pair programming: Mutual Programming: A pair of developers does mutual QA on each others work products. Mutual Testing?

28. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-28 1. 2. 3. 4. 5. Exercise 1c Prepare a test charter for testing the WEB-page: www.amland.no/et_testwww.amland.no/et_test Work in pairs.

29. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-29 3. Exploratory Test Planning, Execution and Documentation. 3.1 Planning, Tasks and Documentation 3.2 Exploratory Testing in Pairs 3.3 Exploratory Testing with eXtreme Programming 1. 2. 3. 4. 5.

30. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-30 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management ET and XP analogy… Exploratory Testing is to Scripted Testing what XP is to traditional “requirements and specifications” based system development methods

31. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-31 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management What is eXtreme Programming? (1) Agile methodology Working in small development groups Two roles: Customer Developer Everybody participate in design, programming and test Iterative development Iterations not more than 2 – 3 weeks Create executable code every night

32. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-32 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management What is eXtreme Programming? (2) Key areas: Communication in the team, I.e. with the customer Simplicity Keep everything simple, should be possible to implement within hours (days) Reduce risk Feedback From customers and others Continuous system improvement Accept Changes Low threshold to changes Continuously specify, design, code and test

33. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-33 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management What is eXtreme Programming? (3) Testing Automated testing Create test before coding Test Driven Design The test will document requirements Daily Build and Test Unit Test, Integration Test and System Test

34. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-34 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management What is eXtreme Programming? (4) Note, please, that in the XP context, only the Customer Test can /validate/ that the program does the right thing. The Programmer Tests merely /verify/ that the program does what the programmers intended. Ron Jeffries, (e-mail on agile-testing list, April 29, 2002)

35. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-35 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management XP and Exploratory Testing (1) “You develop the application test-first, where automated customer tests specify what the system is to do. At some points, you invite James Bach in to do exploratory testing. Perhaps at the ends of iterations. If you’re so inclined, you say that James is part of the Customer Role.” Brian Marick (e-mail on agile-testing list, March 7, 2002)

36. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-36 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management XP and Exploratory Testing (2) “I’ve found that exploratory testing helps me to think of additional acceptance tests to propose to the customer. Lots of times you don’t think of everything about a system until you use it.” Lisa Crispin (e-mail on agile-testing list, March 8, 2002, writing book on XP and Testing)

37. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-37 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management XP and ET – Problems? Paper by Gerold Keefer, AVOCA GmbH: Extreme Programming Considered Harmful for Reliable Software Development (2002): Discussing pros and cons of XP (including Pair Programming) “The most recent study … suggests that … pair programming is rather expensive technology.. and...XP-like pair programming appears less efficient than it is reported” (with references) Reviewing XP’s most well known reference project, C3 at DaimlerChrysle Company Launched in 1995, Kent Beck (XP) took over 1996 Supposed to be used by 87.000 employees, never used by more then 10.000 The project was cancelled in 2000 after 4 years in XP mode! How should this effect Exploratory Testing?

38. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-38 Introduction Test Management and Techniques ET Planning and Documentation ET Styles ET Management ET – The Vanguard The object of your mission is to explore the Missouri river, & such principal streams of it, as, by its course and communication with the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for the purposes of commerce. - Thomas Jefferson's letter to Meriwether Lewis, June 1803 Robinson, H., Microsoft, Exploratory Modeling

39. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-39 1. 2. 3. 4. 5. Planning, Documentation and Execution Summary Introduction Test Management and Techniques ET Planning, Exec. and Documentation ET Styles ET Management How much planning is done with ET? How do you execute the tests with ET? What kind of documentation is produced with ET? How to use Pair Testing with ET?

40. ET Workshop v. 1.20 - How to?©2002 Amland Consulting3-40 1. 2. 3. 4. 5. Exercise 2 Based on previously produced charter and available information about StarOffice: Select one (or more) testing technique(s) and test the application. Explain your mission / reasoning Explain technique(s) used and errors found.