1. Ch 14. Testing & modeling users
Steven Pautz Lauren Sullivan Jessica Herron Chris Moore

2. The aims Describe how to do user testing.
Discuss the differences between user testing, usability testing and research experiments.
Discuss the role of user testing in usability testing.
Discuss how to design simple experiments.
Describe GOMS, the keystroke level model, Fitts’ law and discuss when these techniques are useful.
Describe how to do a keystroke level analysis.

3. User Testing
Developers test whether product is usable by the intended user population
Similar to experimentation, but different from research
Aim is to improve existing design, rather than to discover new knowledge
Not designed to be replicable
Often not published

4. Testing Medlineplus
User testing was done to evaluate changes made after heuristic analysis (chapter 13)
Goal of study was to identity usability problems in the revised interface
Categorization of information
Users’ navigation strategies

5. Medlineplus: Testing Setup
9 participants from washington, DC area
7 were female, all had some web experience
5 tasks were developed, from frequently-asked questions by web visitors
5 scripts were created, one for each stage of the test
Testing was performed in a laboratory setting

6. Medlineplus:testing Every participant worked through the 5 tasks
Their progress through the site and “thinking aloud” comments were recorded
After all tasks were completed, a post-test questionnaire was given, and participants were asked their opinions
Many different data items were collected

7. Medlineplus: Data Analysis
Data was analyzed with a focus on:
Website organization
Browsing efficiency
Search features
Conclusions showed several areas for improvement, which were given to the developers through a presentation and report

8. Medlineplus: Pros and Cons
Appropriate size, qualities of user group
Adequate briefing material was created
Informed consent form
Cons:
Focused heavily on some user qualities (location, etc) at the expense of others
Not much variance among tasks tested

9. Doing User Testing Central idea: controlling the test conditions.
Careful planning is necessary!
Use DECIDE framework for successful study.

10. Determine the goals & Explore the questions
User testing is most suitable for testing prototypes.
Focus the study:
“Can users complete a certain task within a certain time, or find a particular item, etc.”

11. Choose a paradigm Usability testing paradigm Involves:
Recording data by combination of video and interaction logging.
User satisfaction questionnaires.
Interviews.

12. Identifying the practical issues: Design typical tasks
Quantitative measures are obtained.
Types of data produced:
Time to complete a task.
Time to complete a task after a specified time away from the product.
Number and type of errors per task.
Number of errors per unit of time.
Number of navigations to online help or manuals.
Number of users making a particular error.
Number of users completing a task successfully.

13. Usability Engineering Specifications
Current level of performance.
Minimum acceptable level of performance.
Target level of performance.

14. Identifying practical issues: Select typical users
Find the ‘typical user’.
Most important characteristic:
Previous experience with similar systems.
Ex. Hutchworld – targeted towards cancer patients.

15. Identifying practical issues: Prepare the testing conditions
Testing conditions include:
Controlled environment.
Observation room.
Viewing room.
Reception area.

16. Deal with ethical issues
Consent form.
Point out presence of
One way mirrors
Video cameras
Use of interaction logging.

17. Evaluate, analyze, and present the data
Performance measures are recorded from video and transaction logs.
User tests involve a small number of participants. Basic descriptive statistics enable the evaluators to compare performance on different prototypes or systems.

18. Experimental Conditions
In order to test a hypothesis, the experimenter has to set up the experimental conditions and find ways to control other variables that could influence the test result.
Condition 1: read screen of Helvetica
Condition 2: read screen of Times New Roman
Control Condition: read both on printed paper

19. Experimental Designs
Different participants - single group of participants is allocated randomly to the experimental conditions.
Same participants - all participants appear in both conditions.
Matched participants - participants are matched in pairs, e.g., based on expertise, gender.

20. Advantages & disadvantages

21. Other Issues
You need to determine where the experiment will take place as well as how the equipment will be set up.
You also need to determine how the participants will be introduced to the study and what scripts will be needed to standardize the procedure.
Pilot studies are useful to aid in this process

22. Menu Structure
Study was designed to determine whether breadth is preferably to depth in organizing navigable menus.
3 Experimental Conditions: (top level, sub level, content level)
8 x 8 x (slowest)
16 x (fastest)
32 x (middle)
19 experienced web users were given 8 random and unique searches for each condition.
Breadth IS preferable to Depth

23. Predictive Models
Provide a way of evaluating products or designs without directly involving users
Psychological models of users are used to test designs
Less expensive than user testing
Usefulness limited to systems with predictable tasks - e.g., telephone answering systems, mobiles, etc.
Based on expert behavior
GOMS is the most well-known predictive modeling technique in human-computer interaction along with its “daughter”, the keystroke level model

24. GOMS Goals - the state the user wants to achieve e.g., find a website
Operators - the cognitive processes & physical actions performed to attain those goals, e.g., decide which search engine to use
Methods - the procedures for accomplishing the goals, e.g., drag mouse over field, type in keywords, press the go button
Selection rules - determine which method to select when there is more than one available

25. Keystroke Level Model
GOMS has also been developed further into a quantitative model - the keystroke level model.
This model allows predictions to be made about how long it takes an expert user to perform a task.

26. Response times for keystroke level operators

27. Benefits and Limitations
Allows comparative analyses for different interfaces or computer systems
Example: Project Ernestine
Viewed 2 systems to determine which would be better
Findings showed that a new system would slow down production
Limitations
Not good for evaluations
Does not allow for errors to be modeled, only expert performance
Does not reflect multi-tasking (non-sequential procedures)

28. Fitts’ Law (Paul Fitts 1954)
The law predicts that the time to point at an object using a device is a function of the distance from the target object & the object’s size.
The further away & the smaller the object, the longer the time to locate it and point.
Useful for evaluating systems for which the time to locate an object is important such as handheld devices like mobile phones

29. Summary & Key Points
User testing is a central part of usability testing
Testing is done in controlled conditions
User testing is an adapted form of experimentation
Experiments aim to test hypotheses by manipulating certain variables while keeping others constant
The experimenter controls the independent variable(s) but not the dependent variable(s)
There are three types of experimental design: different-participants, same- participants, & matched participants
GOMS, Keystroke level model, & Fitts’ Law predict expert, error-free performance
Predictive models are used to evaluate systems with predictable tasks such as telephones