User Testing and Modeling ICS 205 Chris Wesson Christina Wuerth November 14, 2003

User Testing What is User Testing – Measuring performance of typical users doing typical tasks Goal: – Obtain objective performance data showing how usable a system is in terms of usability goals (ease of use, learnability)

User Testing Applied form of experimentation Tests whether product is usable by intended user group(s) Measures/Records: – Time to complete typical tasks – Number and type of errors – Routes users take through tasks (Web) Typically 6-12 users

User Testing Is part of Usability Testing: – Obsevations – Questionaires – Interviews – User Testing

User Testing and Modeling Doing User Testing Experiments Predictive Models

Keys to Doing User Testing Control testing conditions Careful planning Same conditions for each participant Measurements correspond to test Explicit assumptions DECIDE framework

Q: How Long Should User Testing Process Take? A: As little time as possible while still yielding useful information. On average most companies take more than 8 weeks for user testing Testing of small piece of well developed procedures can be done in less than 1 week if needed

Things to Consider in Planning a User Test What aspects of the product need to be improved? Do the participants in the study represent actual users of the product? What tasks should the participants perform? What information will you observe during the tests? How will you analyze the data you collect? What will you do with this information once it is analyzed?

User Test Planning Steps 1. Define goals and concerns 2. Decide who will participate 3. Recruit participants 4. Select and organize tasks to be tested 5. Create task scenarios 6. Decide how to measure usability 7. Prepare other materials for the tests (questionnaires, etc.) 8. Prepare the testing environment 9. Prepare the test team 10. Conduct pilot tests making changes where needed

The Testing Team The Roles: Usability Specialists The usability specialists know what can be accomplished in a user tests and how a test should be planned. They should also understand the interface and potential problems users may have with it. Designers/Developers The designers and developers understand the product and what the user should be able to do with it. Technical Communicators Technical communicators know what aspects of the documentation should be tested and also see potential problems in how the product communicates to the users.

More User Test Team Roles Trainers The trainers understand the problems that users will have when learning to use the product. Marketing Specialists Marketing specialists know who the users will be. Helpdesk/Customer Service Reps Helpdesk and customer service reps know what problems users have had with earlier versions of the software.

Preparing for the Test 1. Schedule activities 2. Assign roles and responsibilities 3. Train team 4. Write out a test plan 5. Practice by running pilot tests Throughout the entire process remember to: - track progress - document all decisions

DECIDE: User Testing Guideline D etermine goals E xplore questions C hoose paradigm and techniques I dentify practical issues D eal with ethical issues E valuate, analyze, present data

Determine Goals, Explore Questions User testing is best used for testing prototypes and working systems Goals can be broad (how usable?) Specific questions required to focus study (can task X be done in time Y?)

Choose Paradigm and Techniques User testing is part of Usability testing paradigm Data can be recorded using: – Video – Interaction log – User satisfaction questionnaires – Interviews

Identify Practical Issues: Design Typical Tasks Create set of “completion” tasks (finding a website) Choosing which tasks to test is critical Tasks generally last 5-20 minutes Tasks are often straightforward Complex tasks (create design, solve problem) are okay Start simple to build user confidence

Identify Practical Issues: Select Typical Users Must know users’ characteristics Most important: previous experience Use short questionnaire to identify users Equal number of males and females

Identify Practical Issues: Prepare Testing Conditions Control testing environment Minimize outside influences/noises that could distort results

Identify Practical Issues: Plan How to Run Tests Create schedule and scripts for running tests Start with familiarization task Contingency plan for spending too much time on a given task Avoid long tasks and long testing procedure Session < 1 hour

Deal with Ethical Issues Informed consent form Point out any: – One-way mirrors – Video cameras – Interaction logging

Evaluate, Analyze, and Present Data Performance measures recorded from video, interaction logs Only use simple statistics: – MaximumMinimum – MeanStandard Deviation Allows evaluators to compare performance between systems and across tasks

Establishing Goals and Concerns Step 1 of the planning phase is to define goals and concerns. Q: What is a goal? A: A quantitative usability aim that can be stated as a declarative sentence example: A user should be able to locate the print menu item in less than 5 seconds with less than 3 errors during first attempt. Q: What is a concern? A: A concerns are questions raised about the usability of the product – these are often raised while planning for the tests. example: While a user find the new error messages confusing?

Establish Which Goals and Concerns will be tested. Not all goals and concerns can be tested simultaneously. Example of 2 conflicting concerns: 1. Will the user use feature X? 2. Will the user find X easy to use? These two questions can’t be answered at the same time instead you must prioritize.

Using Concerns to Plan the User Test Use General concerns to decide who your participants should be Example: Will new users find this product easy to learn? Make sure you test with some beginning users. Use specific concerns to decide which tasks should be performed. Example: Will users be able to find the print command quickly? In less than 5 seconds? Have the users perform a task of printing.

Sources of Goals and Concerns 1.Task Analysis and Quantitative usability goals Example: General: Menus should be easy to navigate Quantitative Goal: Users should find menu choice in less than 2 minutes with no more than 2 wrong choices when first need it. Task should be done with no errors in less than 1 minute after first attempt.

Sources of Goals and Concerns 2. Timely Issues – Certain concerns arrive at different stages of development and should be resolved at that time. Example: 1) Two different design philosophies that you can’t decide between…plan a user test at start of the project to test prototypes of each 2) There has been previous testing earlier in the project, now you want to test a new feature before continuing development.

Sources of Goals and Concerns 3. Heuristic Analysis and Expert Reviews a) Problems that have been predicted by heuristic analysis and expert reviews should be candidates for user tests. b) Problems predicted by designers, planners, developers, human factor specialists, technical communicators should be tested. c) Concerns found by reflecting on the product itself should be tested.

Sources of Goals and Concerns 4. Previous Tests Example: iterative testing with rapid prototypes run several user tests with the same concerns Example: large scale user tests that raise future concerns retest with same concerns to make sure changes improved the problem

Who Should be Participants? Participants must be like the people who will actually use the product. Develop a user profile then choose participants that fit the profile.

Developing a User Profile Base profile on: General market research Analysis of customers of competitors’ products Focus group sessions Observing and interviewing prospective users

Developing User Profile 1.Think about relevant characteristics a) those that all users share b) those that might make a difference among users example: shared characteristic: users will be undergraduate university students differences: users will have lots of computer experience users will have very little computer experience users will have registered for classes before users will have never registered for a class before

Developing User Profile 2. Decide which characteristics matter most in establishing usability experience and motivation contribute more towards and individual’s understanding than education, income, age, etc. Good Factors to Consider 1.Work experience 2.General computer experience 3.Specific computer experience 4.Experience with this product 5. Experience with similar products

Developing User Profile 3. Think broadly about who the users will be Sample questions to reflect on: 1. Not just those currently working, but what about new hires? What is the rate of turnover that the software will need to handle? 2. Don’t limit yourself to the current market…what about growth, who might want to use this product in the future? 3. What other areas of the corporation may what to adopt this software? 4. Reflect on differences within a category of users. Older users versus younger users? Users who have done the job longer versus users that more adaptable to computer environments?

Developing a User Profile 1.Product Name: Class Registration Software 2.General Characteristics of User Population University Students 3.Characteristics of Users Relevant to the test Year in School Computer Experience Previous Experience Registering for Classes

Developing a User Profile 4. Which Characteristics listed in 3 should all users in the test have in common & how will you define them? All users should be students currently enrolled in a university There will undergraduate and graduate students

Developing a User Profile 5. Which characteristics listed in 3 will vary in the test and how will you define them? Computer Experience will vary. Novice = < 1 year experience Medium = > 1 year experience used < 5 days a week regularly Expert = > 1 year experience used > 5 days a week regularly

Select Subgroups for a User Test Subgroups are people who share specific characteristics important to the user profile Divide groups by one characteristic at a time. Example: University Students Subgroup 1Subgroup 2 UndergraduatesGraduates Subgroup 1aSubgroup 2a UndergraduatesGraduatesLittle computer experience Subgroup 1bSubgroup 2b UndergraduatesGraduatesLots of computer experience

Define Characteristics for Each Subgroup Define what you mean by characteristics Example novice = less than 1 year experience with this product medium = 1 year to 3 years experience with this product expert = more than 3 years experience with this product

Qualify Characteristics of the Subgroups 1. The middle group is often omitted from the subgroups in user studies since most problems found here will be found in novice and expert studies 2. Select a range of participants in each subgroup. Example: Subgroup of novices with < 1 year experience Find users with 1 month, 6 months, and 11 months experience 3. Establish range of each subgroup set min and max for subgroup participants Example: Subgroup of experts with > 3 year experience min would be 3 years but perhaps you want max experience to be 5 years…you don’t want someone whose a hacker even though they may have lots of experience

How Many Participants? Depends on: 1.How many subgroups you need 2.Time and money constraints 3.How important it is to compute statistically significant results * Most of the time all you will care about is inferential statistics

How Many Participants? 6 – 12 participants is the typical choice With a minimum of 3 individuals in each subgroup. * Problems will show up across subgroups

Making the Most of a limited Number of Participants 1.Decide which characteristics are the most important so that your subgroups will be most useful 2.Collect relevant information from participants via interviews or questionnaires to help account for differences that show up in results. Example: one novice user performs faster than all the others, but you find out that he has used a similar product before 3. Select people in subgroups representing the full range of the group

Experiments User testing based on scientific experimentation Aim: test hypothesis that predicts a relationship between variables

Variables Independent: manipulated by researcher Often multiple independent variables: – System version, User experience Dependent: affected by independent variable Common dependent variable: – Time to complete task, Number of errors

Assigning Participants to Conditions Three Experimental Designs: Different participants for all conditions Same participants for all conditions Matched pairs of participants

Matched Participants Participants matched in pairs Based on specific characteristics (expertise, gender, etc.) Used when participants cannot perform in all conditions

Design Comparison DesignAdvantagesDisadvantages Different Participants No order effects.Many participants needed. Individual differences can be a problem. Same Participants Eliminates individual differences. Must counterbalance ordering effects. Matched Participants No ordering effects. Reduces effects of individual differences. No guarantee subjects are match across all variables.

Data Collection Data should measure users’ performance Typical measures: – Response times – Number of errors – Time to complete task

Data Analysis Questions to ask: – Do data sets from two conditions look similar or different? – Any extreme atypical values? – If so, what does that mean? Graph data to display differences

Predictive Models Provide measures of user performance without testing users Useful when it is difficult to do user testing Estimate efficiency of different systems on different tasks Most common models: GOMS family: – GOMS Model, Keystroke Level Model

GOMS Model Models cognitive processes used when interacting with systems G oals: state the user wants to reach O perators: cognitive processes, physical actions necessary to achieve goal M ethods: learned procedures for accomplishing goals S election rules: used to choose between available methods

Keystroke Level Model Variation of GOMS model Provides numerical predictions Uses a standard set of times for main types of operators used during a task Average time to perform certain tasks Compares times for task using different startegies

GOMS Pros and Cons Pros Allows comparative analysis of different systems easily Useful estimates to compare efficiency of different systems Works well for short, well-defined tasks Cons Only models tasks that involve routine tasks Designed to predict experts’ performance Difficult to predict for average users Only predictions about predictable behavior

Fitt’s Law Useful for planning button locations, size and proximity Predicts time to click on objects on a screen T = k log2(D/S +.5), k ~ 100 msec – T: time to move hand to target – D: distance between hand and target – S: size of target Bigger target = easier to reach

Summary User Testing is the core of Usability Testing Controlled laboratory-like conditions Control independent variable(s) to predict dependent variable(s) Experimental designs: different participants, same participants, matched participants GOMS, Keystroke, Fitts’ can be used to predict expert performance