Usability Testing Chapter 6

Reliability Can you repeat the test?

Reliability Huge difference between test subjects –Therefore, only 2 subjects is not a good reliability check Significance levels (e.g., p =.20) Confidence levels –When should an usability problem be fixed? Error rates have the highest variability –Therefore, more test subjects are needed

Validity Does this test reflect what usability issues you want to test? Reliability measured with stats but validity requires methodological understanding Typical problems –Wrong users or wrong tasks –Confounding effects – text based to GUI system

Test Plans What’s the goal of the test? What is it you are trying to prove or show? Pilot tests – show the problem areas of the test. E.g., the instructions

Test Plans The goal of the test? Where and were will the test take place? How long is each test session expected to take? What computer support will be needed for the test? What software needs to be ready for the test? Who will serve as experimenters for the test? How many test users are needed? …

Getting Test Users Test subjects (users) should be representative of who will use the system. Sales (“demoability”) – is it easy to show Asking managers to choose subjects –They will pick the best or the worse

Formative evaluation –Help improve the interface as part of an iterative design process Summative –Assessing the overall quality of an interface

Novice and expert users Most tests should test novice users If possible, test for expert users. But test separately from novice users. What are the effects of training or not training before a test?

Between-Subject test Simplest and most valid Different tests subject using different systems Possible problems: –Individual differences –Assignment to test groups (volunteer early or late)

Within-Subject test All users are tested on all systems Problem: no longer a set of novice users when testing a secondary system

Choosing experimenters It’s better to get an usability person to conduct the test but you, the developer, can also run a test In fact, it’s good to know the system well From Usability Engineering (Nielsen) –“It is possible for computer scientistws to learn user test methods and apply them with good results.” However, designers try to explain away problems.

Ethical aspect of testing human beings Users should feel as comfortable as possible Tell the test subjects: –No information will be revealed –Explain the testing environment. E.g., computer, etc. The tester should not allow observers because they tend to influence results. The tester should not interfere with the user. Let the user find the solution themselves Emphasize that it is the system that is being tested, not the user

Test tasks Based on a task analysis. What is it the user needs to do. Tasks should be small enough to be completed in time that the experiment takes. Test tasks should be in writing. The user may refer to the steps.

Stages of a Test 1. Prep – is everything in working order? 2. Intro - tell the user the purpose of the test. 3. Test 4. Debrief

Stages of a Test Intro - tell the user the purpose of the test. A reminder that the test is confidential and should not be discussed with others A statement that participation is the test is voluntary and the user may stop at any time At FIT a human test subject document needs to be signed by the tester and returned to the university.

Performance Measurement Has the usability goals been met? (see page 194) Test a set of tasks The data collected: time to perform and error rate Goals are abstract, so break them down. Make sure you know the start and finish of a measured test

Thinking aloud “verbalizing their thoughts” Shows how users interpret each interface item Users performed 9 % better using thinking aloud. Constructive interaction (co-discovery learning) and coaching methods.

Usability Labs Is there really a need? Why videotape? Impact analysis – you look after taping for a known problem. Convince the manager and developers Usability kiosks