1. Alfred Kobsa University of California, Irvine
Preparing a User Test
Alfred Kobsa
University of California, Irvine

2. Specifying Global Test Goals
(Intended) purpose of product
Product development status
Test goals (with priorities)
User profiles (personae)
Even if you know and can clearly specify for the purpose of the user test what the intended purpose of the product is, when it’s launched to actual users on a massive scale, there are instances when the purpose of the product is poorly communicated. Can someone give examples?
Facebook -
What are some industry scenarios where it’s difficult to specify the intended purpose of the product?
Understanding where your product is at its developmental stage is very crucial – why? Could you give me examples?
If you don’t understand where your product is at – what does it even mean to know where your product development status is at?
User reach/ penetration/ developmental stage;
Having 2 out of 3 intended features implemented (does it make sense to develop the third) – Chat with Friends
Facebook Live – has commenting

3. Specifying Global Test Goals
(Intended) purpose of product
Product development status
Test goals (with priorities)
User profiles (personae)
Even if you know and can clearly specify for the purpose of the user test what the intended purpose of the product is, when it’s launched to actual users on a massive scale, there are instances when the purpose of the product is poorly communicated. Take a look at Facebook Desktop application

4. Specifying Global Test Goals
Take a look at Facebook here. Insights vs. Explore Feed vs. Pages Feed
Pretty sure when the explore feed team and the insights team each came up with each of their own product, they both had very clear intended purposes in mind when they were doing these usability tests and interviews around how users viewed their respective features. Once it rolled out on to the desktop app – it be came unclear.
Product application is too dense
Competing / overlapping feature purposes
What are some industry scenarios where it’s difficult to specify the intended purpose of the product?

5. Specifying Global Test Goals
(Intended) purpose of product
Product development status
Test goals (with priorities)
User profiles (personae)
Understanding where your product is at its developmental stage is very crucial before you actually perform the user test.
What does it mean to understand where your product is at? Does it mean that you know that out of the 4 features that are supposed to be implemented, it’s knowing that right now you only have 3. Is that what it means to understand what the product development status is?
Where the product is at:
Known UX issues  with what, who, and why
Hacks  How are users overcoming these issues?
User metrics around product use
Regional penetration of the product
What else?
The more you can specify about the product development status, the more you know what to do / options you have with what you can do in designing a user test.

6. Specifying Global Test Goals
(Intended) purpose of product
Product development status
Test goals (with priorities)
User profiles (personae)
Super important – have clear hypotheses and even assumptions around what you are trying to test. Why? It’s the goal around which you designed the user test.
(with priorities) – what does that mean?
Clear test goals should lead to clear results… not always. Why?
Communication with non-researchers and engineers

7. Specifying Global Test Goals
(Intended) purpose of product
Product development status
Test goals (with priorities)
User profiles (personae)
Why important? You can narrow down your participant selection for the user test. Why is this important?

8. Selecting Tasks
Test subjects should not merely “try out the system for n minutes”, but rather carry out selected tasks with the system.
One cannot test every possible user task. Rather, usability tests must focus on representative and/or important tasks.
1. Make the Task Realistic
User goal: Browse product offerings and purchase an item.
Source:
Poor task: Purchase a pair of orange Nike running shoes.
Better task: Buy a pair of shoes for less than $40.

9. Selecting Tasks
Test subjects should not merely “try out the system for n minutes”, but rather carry out selected tasks with the system.
One cannot test every possible user task. Rather, usability tests must focus on representative and/or important tasks.
2. Make the Task Actionable
User goal: Find movie and show times.
Poor task: You want to see a movie Sunday afternoon.
Go to and tell me where you’d click next.
Better task: Use to find a movie you’d be interested in seeing on Sunday afternoon.

10. Selecting Tasks
Test subjects should not merely “try out the system for n minutes”, but rather carry out selected tasks with the system.
One cannot test every possible user task. Rather, usability tests must focus on representative and/or important tasks.
3. Avoid Giving Clues and Describing the Steps
User goal: Look up grades.
Task scenarios that include terms used in the interface also bias the users. If you’re interested in learning if people can sign up for the newsletter and your site has a large button labeled Sign up for newsletter, you should not phrase the task as "Sign up for this company's weekly newsletter." It's better to use a task such as: “Find a way to get information on upcoming events sent to your on a regular basis.” 
Poor task: You want to see the results of your midterm exams.
Go to the website, sign in, and tell me where you would click to get your transcript.
Better task: Look up the results of your midterm exams.

11. Selecting Tasks
Tasks should be selected that
may be fraught with usability problems, as suggested from earlier concerns and usage experience;
will be frequently carried out by users (20% is used 80% of time)
are mission-critical;
are performed under time pressure; or
are new or have been modified in comparison with previous version or competitive program
☞ Brainstorm and then filter tasks using these criteria
☞ Test the comprehensibility of task descriptions
☞ Specify timeout for each task (possibly do not reveal to subjects)

12. Creating Scenarios
Scenarios are created to contextualize user experiments
(which in general yields more representative test results)
Scenario descriptions should be
- short
- formulated in the words of the user / task domain
- unambiguous
- contain enough information for test subjects to carry out tasks
- be directly linked to tasks and concerns
☛ User should read scenario descriptions (and experimenters should possibly read them aloud at the same time)
☛ Scenario descriptions should be tested (in the pilot test or even earlier)
Examples:

13. Deciding how to measure usability
Performance measures
Time needed to carry out a task
Error rate
Task completion rate
Time spent on “unproductive” activities (navigation, looking up help, recovery after an error)
Frequency of “unproductive” activities
Counting keystrokes / mouse clicks
Etc. (see Dumas & Reddish)
Measures of satisfaction
User-provided:
Observed(?): frustration / confusion / surprise / satisfaction
User ratings (e.g., SUS – System Usability Scale)
Comparisons with previous version / competitors’ software / current way of doing it
Behavioral intentions (use, buy(?), recommend to friend)
Free comments (after and during experiment)

14. Preparing the Test Materials
Legal documents
Informed consent form
Non-disclosure form
Waiver of liability form
Permissions form (e.g., for video recordings)
Instruction and training related materials
Software / Powerpoint slides / video to be shown
Summary of software functionality
Write-up of oral instruction
(Guided) training tasks
Task-related materials
Scenario description
Task descriptions (☛ one task on each page, large font for task/page number)
Pre-test and post-test questionnaires
Experiment-related materials
“Screener” with participant election criteria
Experimental time sheet / log book
To-do list for all experimenters

15. Preparing the testing environment
Hardware equipment
☛ cater to users’ normal equipment; remove all potentially distracting programs
Sample data
☛ make it look real
Voice recording
☛ take care of ambient noise (also exceptional), direction of microphone,…
Screen recording
☛ (mind a possible slowdown of tested program)
Video recording
☛ take care of video angle, blocked view, glare, different sunlight over the day,…
Time taking
☛ avoid races (between participants, or participants against stop watch)
Lab layout (see Courage & Baxter)
☛ participants should not influence each other

16. Setting up a test team Typical roles
Greeter: welcomes subjects, makes them relaxed, bridges wait time
Briefer: informs about study, makes them sign forms
Instructor / trainer: instructs them about the software
Test administrator: tells subjects what to do
Note taker
Video operator
Backup technician for emergencies
Many of these roles can be combined in a single person
No role-switching over the duration of an experiment, to ensure comparability
The number of team members also depends on the number of parallel / overlapping subjects and the experimental design
Teams of 2-3 are typical
Tests are typically carried out by UI design team, and/or outside usability specialists
Developers, managers, user representatives should be able to watch (invisible to the subjects, or in the background)

17. You can make assumptions about “where the product is at”
Exercise
Facebook search sucks - we don’t know why. We just know that user metrics are low for search bar related activity.
Design a user test that would inform you around what kind of barriers people have around using the Facebook search product
You can make assumptions about “where the product is at”
What is your user test goals? How would you prioritize them?
Specify user goal (for tasks)
Provide user task description and steps
List what you are going to measure and how that would answer your test goals