1. Chapter 14: Usability testing and field studies
Right – How to observe the real effects of your fishing equipment on the fish! From
Chapter 14: Usability testing and field studies
Question 1 asks about the usability of something very familiar – your favorite development environment!
Question 1

2. Usability Testing Question 2 Emphasizes the property of being usable
Key Components
User Pre-Test
User Test
User Satisfaction Questionnaire
User Action is recorded
The goal is to test whether the product being developed is usable by the intended user population. It involves recording users doing very typical tasks and the environment is controlled. The user test measure human performance on specific tasks, please use the website to find the route between cities A and B.
The user satisfaction questionnaire is used for recording how the users feel about using a particularly product.
Typically during a usability test, keystrokes, mouse movements together with video recordings of the users actions is used for recording the users performance
Question 2

3. Some type of data Question 3 Time to complete a task.
Number and type of errors per task.
Number of errors per unit of time.
Number of navigations to online help or manuals.
Number of users making a particular error.
Number of users completing task successfully.
Try to decide why some of these would be better than others, in relation to your system that your team is developing.
Question 3

4. Qualitative Data Question 4
Likert Scale: A typical question using a Likert Scale might pose a statement and ask the respondent whether they Strongly Agree - Agree - Undecided - Disagree or Strongly Disagree.
Example:
We have a lot of work to do in 371
Strongly Agree - Agree - Undecided - Disagree or Strongly Disagree.
We’ve looked at this type of data before. Now, how to evaluate it!
Question 4

5. Testing conditions Usability lab or other controlled space.
Emphasis on:
selecting representative users;
developing representative tasks.
Tasks usually last no more than 30 minutes.
The test conditions should be the same for every participant.
Informed consent form explains procedures and deals with ethical issues.
We have a controlled space in our Usability Lab. This is roughly how to make use of that in running tests…

6. How many participants is enough for user testing?
The number is a practical issue.
Depends on:
schedule for testing;
availability of participants;
cost of running tests.
Typically 5-10 participants.
Some experts argue that testing should continue until no new insights are gained.
This is a good question. If you decide you need 10 for your own study – that means you need to find 10 good subjects.

7. Usability lab with observers watching a user & assistant
The facilities comprise of a main testing lab, with recording equipment and the product being tested. Typically there are 2 to 3 wall mounted cameras that record the users behavior, facial expression, general body language. A microphone to get their comments since users are mostly asked to think out loud.

8. Portable equipment for use in the field
A less expensive alternative is using a mobile usability lab. Laptop video cameras and other measuring equipment are temporarily set up in some office and used t conduct the study.

9. Many users can be tested and the data is logged automatically

10. How do we do this? Question 5 Goals and Questions
Select the Participants
Age
Gender
Computer Expertise
Domain Expertise
Develop Tasks
Develop Pre/Post Test Questionnaires
Develop a test procedure
Greet, Explain, Exploration task, Direct behavior
Test [Think Aloud Protocol]
Post Test Debriefing
The tasks you develop must be appropriate with respect to the product/software being tested. For example if you are testing a site like Expedia, a sample task could be purchase two one-way tickets from Indianapolis to Jamaica between a specified date range. You will have to evaluate the efficiency of the tasks by doing a pilot study. The participants for the pilot study must fit the same profile. The main reason behind the final study is to ensure that you have everything in place and it also helps to determine when a person has successfully evaluated a task.
This is a script you have in place to help conduct the study. We’ll hand out an example.
Question 5

11. MedlinePlus.com Typical website that might be used in a test –
An experienced user would know automatically what to click on, to do what they wanted to do.
A new user of this site, not so much!

12. Experiments & usability testing
Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables.
Null Hypothesis
Alternate Hypothesis
This may be the main time as a CS or SE major, that you can do real experiments, with people, in a scientific way!

13. Experiments
Predict the relationship between two or more variables.
Independent variable is manipulated by the researcher.
Dependent variable depends on the independent variable.
Typical experimental designs have one or two independent variables.
Efficacy of audio books versus video tutorials
Null Hypothesis: There is no difference in the amount of learning between two.
Two tailed: There is a difference between the two.( Two tailed hypothesis is where you imply that there is a difference but not what the difference will be)
One-tailed: Where you imply the difference – Audio books are inferior in learning ability when compared to video tutorials. Typically, unless there exists a reason or a theory that seems to support the case, two tailed hypothesis is usually chosen.
When you are doing studies like this, it is important to keep the other variable constant. For example time, the type of material being tested must remain the same.
To make this sort of test very efficient, you usually set up a control variable with which you can compare against. For example paper based books.
Question 6

14. Experimental designs
Different participants - single group of participants is allocated randomly to the experimental conditions.
Same participants - all participants appear in both conditions.
Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.
Does the participant mix really make a difference, in testing your own project?

15. Different, same, matched participant design
Here are all the possible ways that such things might make a difference.

16. How do you conclude?
Use statistical tests like t-Tests that compare the difference between the conditions to reveal if they are significant.
If you’ve had the statistics course (MA381) already, then you will know what a t-Test is.

17. Field studies Field studies are done in natural settings.
The aim is to understand what users do naturally and how technology impacts them.
Field studies can be used in product design to: - identify opportunities for new technology; - determine design requirements; - decide how best to introduce new technology; - evaluate technology in use.
Back to the question of “what kind of study are you doing, when you have the user try your system, in the F-217 lab, instead of in the usability lab?”

18. Data collection & analysis
Observation & interviews
Notes, pictures, recordings
Video
Logging
Analyzes
Narrative
Categories can be provided by theory
Grounded theory
Activity theory
Experience sampling method, maintaining journals etc.

19. The alternative to watching users directly – have them report on their experiences.
Question 7