1. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 1 Black Box Software Testing Fall 2004 PART 12 -- USER TESTING by Cem Kaner, J.D., Ph.D. Professor of Software Engineering Florida Institute of Technology and James Bach Principal, Satisfice Inc. Copyright (c) Cem Kaner & James Bach, 2000-2004 This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

2. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 2 User testing Tag line –Strive for realism –Let’s try this with real humans (for a change). Fundamental question or goal –Identify failures that will arise in the hands of a person, i.e. breakdowns in the overall human/machine/software system. Paradigmatic case(s) –Beta testing –In-house experiments using a stratified sample of target market –Usability testing

3. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 3 User testing Identify categories and roles of users. Determine what each category of user will do, how they will do it, and what they value. Get real user data, or bring real users in to test. Otherwise, systematically simulate a user. Powerful user testing is that which involves a variety of users and user roles, not just one.

4. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 4 Beta testing Primary idea is to involve users with pre-release product Beyond that, what are your objectives for the test? –Equipment compatibility testing? –Usability evaluation? –Expert opinion / advice? –Bug hunting? –Favorable publicity? –Increase adoption confidence of key clients? Different objectives require different release planning / scheduling

5. Black Box Software Testing Copyright © 2003 Cem Kaner & James Bach 5 User Testing Strengths –Design issues are more credibly exposed. –Can demonstrate that some aspects of product are incomprehensible or lead to high error rates in use. –In-house tests can be monitored with flight recorders (capture/replay, video), debuggers, other tools. –In-house tests can focus on areas / tasks that you think are (or should be) controversial. Blind spots –Coverage is not assured (serious misses from beta test, other user tests) –Test cases can be poorly designed, trivial, unlikely to detect subtle errors. –Beta testing is not free, beta testers are not skilled as testers, the technical results are mixed. Distinguish marketing betas from technical betas.