Thoughts on Systematic Exploratory Testing of Important Products James Bach, Satisfice, Inc.

Exploratory Testing and Risk  If the stakes are high, how can you justify using unproductive approaches to testing?  Highly scripted testing means testing without using your brain. This is unproductive.  If you are using your brain; if you are designing tests throughout the project based on all the information available; then you are doing exploratory testing.

Exploratory Testing and Risk  I predict that within ten years it will widely be considered reckless to ship a product that has not been tested systematically using exploratory methods.  Exploratory testing is extremely productive. All testers do it. The problem is that very few take it seriously, or have any training in it.

 The “questions” consist of various ways of configuring and operating the product. (Asking questions about the product without operating it is usually called review rather than testing)  The product “answers” by exhibiting behavior, which the tester observes and evaluates.  To evaluate a product is to infer from its observed behavior how it will behave in the field, and to identify important problems in the product. Testing is questioning a product in order to evaluate it. I teach testing as a sort of martial art.

Exploratory Testing Defined Exploratory testing is simultaneous learning, test design, and test execution. pure scripted freestyle exploratory When I say “exploratory testing” and don’t qualify it, I mean anything on the exploratory side of this continuum. chartersvague scripts fragmentary test cases roles

Exploratory Testing Can Be Done…  …on an idea before it’s a product.  …on a product to model it for test planning.  …to augment or improvise upon scripts.  …to investigate a problem uncovered by some other test process.  …to test a product after changes are made.  …as the principal method of finding problems on a complex product.

Explore the meaning of requirements.  How do you test this requirement? – “When the user presses a button on the touchscreen, the system shall respond within 300 milliseconds”  Possibilities: – Performance analysis instrumentation? – Use a stopwatch? – Estimate it by eye? – Just report whether the product seems annoyingly slow?  Key questions: – What does this requirement really mean? – What risk corresponds to it? – Why is it written that way? – How important is it compared to other requirements?

Explore implications and scenarios.  How do you test this requirement? – “The movie [of x-ray images] shall be played at the frame rate set by the frame rate slider control.”  Possibilities: – Performance analysis instrumentation? – Use a stopwatch? – Estimate it by eye?  Key questions: – No tolerance is given. How close is close enough? – How much variation is acceptable? – Could a doctor come to a false diagnosis because of problems with this feature?

Things I explore when I play with a product.  Composition – Affordances: Ways the product can be used. – Dimensions & Variables: Product space and what changes within it. – Relationships & Interactions: functions that cooperate or interfere. – Navigation: Where things are and how to get to them.  Conformance – Benefits: What the product is good for-- when it has no bugs in it. – Consistencies: Fulfillment of logical, factual, and cultural expectations. – Oracles: Specific mechanisms or principles by which you can spot bugs. – Bugs and Risks: Specific problems and potential problems that matter.  Context (of the Product) – History: Where the product has been, and how it came to be. – Operations: Its users and the conditions under which it will be used.  Conditions (of Testing) – Attitudes: What your clients care about and what they want from you. – Complexities & Challenges: Discover the hardest things to test. – Resources: Discover tools and information that might help you test.

Exploration is a feedback loop  Configure  Operate  Observe  Evaluate execution  Understand Testing Mission  Understand Project Environment  Understand Risk Factors & Areas  Understand Product & Problem Domain  Configure Test Lab planning  Model Product  Identify Risks  Select Coverage  Determine Oracles  Determine Operations design learning Follow Risk Rumors

“How do you organize your work?” Customers Information Developer relations Team Equipment & tools Schedule Test Items Deliverables Structures Functions Data Platforms Operations Capability Reliability Usability Scalability Performance Installability Compatibility Supportability Testability Maintainability Portability Localizability Function testing Domain testing Stress testing Flow testing Scenario testing Claims testing User testing Risk testing Automatic testing

Analyzing Non-Routine Risks is About Learning, not Calculating  This learning… – is a heuristic process, not a rote procedure. – unfolds over time. – involves many different sources and people. – is a model-building process. – involves comparing risks predicted to risks manifested. – is highly social and psychological. – is largely self-correcting.

Develop Your Skills! TESTING SKILL CONSISTS OF THE SKILLS OF  General Systems Thinking  Critical Thinking & Research Methods  Rapid & Exploratory Learning PLUS KNOWLEDGE OF  Cognitive Science & Human Factors  Decision & Utility Theory  Folklore & Heuristics (most testing textbooks are folklore) PLUS  Context-Specific Technology and Social Skills