1 Foxtrot, by Bill Amend 10/31/02

Usability Testing Amy Thurston UserWorks, Inc. 20 November 2002

3 What is usability? An effective, efficient, satisfying relationship among: a system or product its users what the user wants to accomplish Considering both initial ease of use (intuitiveness, ease of learning) operational ease of use (productivity, ease of retention)

4

5 What is the goal of usability? To improve aspects of the product with which users come into contact To make user interfaces logical, intuitive and clear to people who use them Includes the entire user experience physical design (dimensions, layout, packaging) look and feel (displays, controls) procedures (the steps to complete a task) environment (physical, geographic) availability/usefulness of support (online help, training, documentation, help line support)

6

7 Types of usability testing Testing with users Heuristic / expert evaluation Online survey Click-path monitoring Concept evaluations Considerations These techniques can be complimentary Ideally, one would use groups of these as you go through an iterative design process

8 Categories of usability testing Formative - Evaluations based on observing representative users completing important and frequent tasks with a prototype of the product. Errors, accuracy, speed of performance, and analyze user comments to recommend design changes Summative / Acceptance - Usability engineering specialists observe and record performance and user comments at the end of a design cycle Benchmarking - Establishes a level of product usability for future testing and measurement

9 ______________ / / Visual DesignUser Testing Takes: 2days-2weeks / Surface / Look and FeelHeuristic Evaluation Lasts: 2weeks - 2mos /_____________/ ______________ / / Interface DesignUser Testing Takes: 2days-2weeks / Skeleton / Heuristic Evaluation Lasts: 2weeks - 2mos /_____________/ Participatory Design ______________ / / Interaction DesignCard sorting Takes: 2days-2weeks / Structure / Prototype Testing Lasts: 1mo - 6mos /_____________/ ______________ / / User ModelingUser Interviews Takes: 2weeks - 2months / Scope / Project ObjectivesPersonas/Scenarios Lasts: 3mos - 1year /_____________/ Mental Models ______________ / / Deep User ResearchEthnography Takes: 6weeks - 6months / Strategy / Business GoalsContextual Research Lasts: 1year - 5years /_____________/ --Peter Merholz, peterme.com, based on a structure by Jesse James Garrett, jjg.net

10 Objectives of usability testing Plan Prepare Collect data Analyze Report the data

11 Whats not on the list? Taking action on the findings Many times, implementation is left to another group Findings need to be balanced with business decisions Whats best for the user might not be best for the product Decision makers often have a broad array of reports which inform their actions; usability is only one of these

12 Planning What are the goals of testing? Is the product in development and is there a need for guidance? Is the product about to be deployed and is there only time for tweaking What if there are show-stoppers? Note that the final results will only be as good as the procedures are appropriate Select the best usability testing technique

13 Planning Who are the users? Are there multiple user groups? How many users should be tested? Do we need to test users from all groups? What type of test is this? (i.e,, do we need quantitative, random sampling based data? Do we need to turn around the results in two days?) What or who determines if the product is usable?

14

15

16 Planning What should be tested? What are the users typically trying to do? Are there some basic functions that have to be right? Are there some functions that seem confusing or problematic? Do anticipated environmental situations affect usability?

17 Planning Who will receive the report? What happens to the information that will be generated? Will changes be made to the product based on the data?

18 What if you dont know the users?

19 Types of user needs and task analysis activities Interviews / focus groups Competitive / comparative analysis Contextual inquiry Walkthrough / talkthrough Link analysis

20 Interviews / focus groups Interview users / conduct focus groups to probe for user needs on specific design features or preferences Example: Interviewer: This product has the capability to do a three-way conference call. Would that be sufficient? User: No; sometimes we have 4 to 5 locations that need to be involved.

21 Competitive / comparative analysis Examine other similar products Determine typical user scenarios and evaluate products List features, noting their design and, if possible, their effectiveness

22 Contextual inquiry Understand procedure Develop training criteria Determine information needs

23 Walkthrough / talkthrough Walkthrough only of use where there are well-established procedures and existing products May pick up more dynamic problems than a static task analysis Enough time to accomplish? Physical distance between needed information

24 Link analysis Task analysis focusing on: Communication links Physical access Emphasis on frequency of contact Often used for: Workplace design Task allocation Determining instrumentation requirements

25 Preparation Materials needed for testing and test plan Screener to qualify participants Video consent form Tasks for participants (enough to fill the time, and a little more) Pre-test and post-test questionnaires Test plan

26 Preparation: Screener Based on a description of user group(s) Determine which characteristics are important for choosing participants Some important characteristics are used to determine group membership (i.e., profession, experience, location) Other characteristics should be balanced within the group (i.e., gender and age are frequently balanced)

27 Preparation: Screener Use the screener to verify participant is willing to be video taped, if appropriate After the potential participant successfully completes the screener, use the opportunity to schedule the participant Participants who are no-shows are costly Give incentives Make reminder calls

28 Preparation: Consent Form Is it legally necessary to receive written permission from a participant to video or audio tape them? Note that testing can still go forward if they agree but that data will be without back-up verification You need the participants agreement to use their image Some clients need written non-disclosure statements from participants

29 Preparation: Tasks Tasks should be written Basic parts of the product (i.e., the critical functions or paths) Parts of particular interest to the client Parts that Are known to cause confusion and problems You anticipate will cause confusion and problems

30 Preparation: Tasks Tasks should be pilot tested with someone unfamiliar with the product To assure that the task wording is clear To determine how long it takes to accomplish the tasks To determine if the tasks are do-able To find if there are alternate correct paths

31 Preparation: Pre-test and Post-test Questionnaires Questionnaires can be of many types Specifically designed for the particular test General questionnaires designed by others (e.g., QUIS from U MD) Validated questionnaires such as SUS (by John Brooke at Redhatch for DEC, free), or SUMI and WAMMI (from Jurek K. at University College Cork, not free) Marketing questionnaires Benchmarking, baseline measures

32 Preparation: Post-task Questionnaires Purpose is to determine from the participants point of view Satisfaction Efficiency Ease of learning Ease of use If feel in control If like system or not

33 Preparation: Test Plan A script for the administrator Gives place for checklist of what to say Can give precise wording if study needs to adhere to more precision Gives the anticipated correct path for ease of the administrator to follow Introduction should be delivered in person even if test is a non-interference test

34 Testing Testing team can be one or more persons Participant fills out preliminary forms Start the tape! Use test guide to direct the production Introduce test Have user do task(s) Note usability issues, comments (by hand or software) Have user answer questions, questionnaire(s) Debrief, thank, and compensate

35 Testing Materials to have participant complete before test Demographic, consent forms Use the test guide during the test Materials to have been completed by end of test Questionnaires, forms for usability Data log and/or other notes

36 Analysis: the Basics Usability heuristics Constantine and Lockwood (see Jakob Nielsen (see tml and Expert opinions (see references) Industry best practices Experience

37 Report Usability tests must be documented To have the appropriate effect To allow the powers-that-be to make appropriate decisions To serve the in the historical context of a project

38 Report Helpful to know these answers before starting the report Who is the report for? How will it be used? How much time after testing is allowed for analysis and reporting? What will be the most effective (or acceptable) format?

39 Report Basic information about participants and test Test based data Analysis of the data--prioritize problem severity High--cannot complete the task or causes data loss Medium--likely to cost user time or difficulty Low--not substantial but worth considering Recommendations (if appropriate for the type of test) including further testing if problems are severe or complicated

40 Report Formats Written reports Top line report (1 p) Brief report (5-7pp) Systematic scientific (academic style) report PowerPoint reports Common Industry Format for software analysis reports (but there is no place for recommendations) Briefings Formal with presentation Telephone based Highlights tape Collection of clips of participants using the product with voice over comments or introductory titles

41 Sample results from a usability test Overall, 4 of 5 users liked the product 3 of 5 users had some difficulty completing Task 3. The average rating for completing this task was 5.3 on a 9 point scale. The three users that had difficulty selected the blue button instead of the red button because the names are ambiguous and confusing. It is recommended that one or both of these names change to reflect their opposing effects. 5 of 5 users completed Task 6. Users commented that they liked this feature and found it easy to complete the task. Average rating was 8.7 on a 9 point scale.

42 Usability resources Online: Usability.gov - National Cancer Institute Useit.com - Jakob Nielsen Usableweb.com - Keith Instone UPAssoc.org - Usability Professionals Association UIE.com - User Interface Engineering

43 Good Luck in Your Own Usability Testing Amy Thurston UserWorks, Inc 1738 Elton Rd., Suite 138 Silver Spring, MD Phone FAX