1. Introduction to Team-based Usability Testing
“As companies design more for usability and understanding, they will discover a competitive edge, for these principles save customers time and money while increasing morale.” - Donald Norman

2. Objectives Usability overview Setting objectives
Preparing the test plan
Conducting the test

3. Usability
Usability “means that the people who use the product can do so quickly and easily to accomplish their own tasks.” - Redish and Dumas, A Practical Guide to Usability Testing

4. Usability Is not functionality Is not beta testing
Functionality is how the system works
Is not beta testing
Beta testing is unfocused and at end of process
Is not market research
Market research is about customer demands

5. When to test Design phase Development phase surveys and focus groups
low fidelity prototype testing
high fidelity prototype testing
heuristic evaluation
Development phase
usability evaluation

6. Advantages of a usability lab
Lab is helpful for tightly controlling the variables
Lab helps capture low-level details
keyboard monitoring
videotaping for later analysis
eye movement tracking
Labs allow less intrusive observations

7. Disadvantages of a usability lab
Labs are intimidating to users
Labs are rarely portable so they cannot go into the users’ environment
Sometimes low-level information is not as important as making high-level design decisions
Labs are expensive

8. How long will the test take?
How much usability testing has already occurred?
Have users been profiled?
Has a task analysis occurred?
Have usability objectives been set?
How complex is the product?
How much of the product are you going to test?

9. How long will the test take?
How many test participants do you need to get the information you want?
How much training or prior experience does the test team have with the product?
How many different usability objectives do you need to test?

10. Who should run the test?
The most successful approach to usability testing is a cross-functional, team-based approach
People bring varying expertise to the team
All groups benefit from experience with users
Interpretation of challenges and design decisions are more creative with teams

11. Who should run the test?
Team-based testing provides a great way to get support involved early
Developers learn to “hear” user questions when they design
Writers and trainers can help design communication into the product
Human factors experts and user interface designers can provide helpful principles about users and design

12. Who should run the test?
More people can be involved in planning the test than in conducting the test
Target 3-5 people for conducting the test
Sometimes best if the developers are not actively involved in running the test, although they should observe if possible

13. How do you ensure success?
You need the buy-in of upper management if the results are to be applied
You need the cooperation of development to ensure they don’t feel attacked
You need the support of marketing, sales, and product management for funding
You need the support of your immediate manager to spend time testing

14. How do you ensure success?
Pick a manageable portion of the product to test
Select a high-profile project
Select an area that will produce significant results
Advertise your successes (become part of the sales story)

15. Stages of a usability test
Setting objectives
Creating the test plan
Creating questionnaires
Selecting test participants
Training the team
Conducting the pilot test
Conducting the tests

16. Usability objectives
Created during user/task analysis and product scoping
Must be measurable
Should indicate:
Type of user
Task to be performed
Specific performance criteria

17. Defining usability objectives
Five attributes of usability (Nielsen, 1994):
Learnability: system is easy to learn so users can get started quickly
Efficiency: system should be easy to use, resulting in high productivity
Memorability: system should be easy to remember
Errors: system should have low error rate and allow error recovery
Satisfaction: system should be pleasant to use

18. Examples
Novice PC users can change the ink cartridge in their color inkjet printer in less than 5 minutes using the user manual
Users will be able to install and configure the default application in less than 15 minutes
Users should rate the product as either “easy to use” or “very easy to use” (4 or higher on a 5 point scale)

19. Considerations
A single usability objective may result in multiple design decisions
Emphasizes importance of having cross-functional team
Focus on objective that are most important to the success of the product

20. Test plan
Explains what you are testing and how you will conduct the test
Do not skip this step
Important to have a plan that everyone can agree to and understand
Helps communicate your intentions to others in the organization
Do not overdo this step

21. Test plan Purpose Problem Statement/Test Objectives User Profile
Why are you conducting the test?
Problem Statement/Test Objectives
What usability objectives are you testing?
User Profile
Who are your users and their defining characteristics?

22. Test plan Method Task List Test Environment/Equipment
Details how you will run the test
Task List
List of the tasks to be completed, projected times for completion, required state of system to test tasks
Test Environment/Equipment
List what you need to conduct the test

23. Test plan Test Team Roles Evaluation Measures Conclusion
Define what each person will do
Evaluation Measures
What data you will collect through the test
Conclusion
What will happen after the test - reports, design meetings, presentations

24. Pre-test questionnaire
Ensure that participant meets user profile
Gather additional detail about the participants’ knowledge and experience

25. Post-task questionnaire
Asks questions about a task as soon as it is completed
Captures feedback when it is fresh
Can help measure change in attitude toward the task
Keep it short

26. Post-test questionnaire
Measure satisfaction and reaction to the product and the overall experience
Ask participants to elaborate on a few critical areas/issues

27. Training the team
Make sure everyone understands the usability objectives and the testing procedure
Define everyone’s roles and discuss how they should behave during the test

28. Conducting a pilot test
Conduct a practice test using the complete team, all prepared materials, and the actual facilities
Make sure at least one person observes the test to make recommendations
Refine any of the materials or facilities as necessary

29. Selecting test participants
Number of participants depends on the number of user groups (differentiating characteristics)
Target 3-5 users from each user group
Best if users come from the real user population rather than internally

30. Observers
Not actively involved in conducting the test, but can benefit from observing
Must not comment or react to the participant’s actions or comments
Should be separated from the actual room where the testing is going on
Should be removed if they cannot avoid feeling defensive or being intrusive

31. Data logger Be familiar with any data logging forms
Establish standard codes ahead of time for recording events and comments
Remain open to unforeseen directions
Record user comments that summarize or emphasize good and bad points
Keep track of time

32. Facilitator Main contact with the participants
Conducts the briefing and debriefing
Only person to talk to the participants during the test

33. Briefing Thank them for participating in the product evaluation
Assure users that they are not being tested; instead, they are helping you evaluate the product
Explain to them what will happen during the test; they will be especially interested in any recording devices and observers

34. Briefing
Ask them to sign a consent form and a non-disclosure agreement
Encourage them to think out loud so you can record their actions and the reasons behind their actions
Make sure you explain what they should do if they have questions or experience problems

35. During the test Be sensitive to excessive frustration
Be ready to handle unplanned situations
Remind participants to talk out loud
Look and listen for the unexpected
Avoid intervening unless necessary; try using questions to redirect rather than take over as the expert

36. Debriefing Get users to elaborate on significant events or comments
Ask for reasons to specific behaviors from the test
Watch the video tape with them and ask them to explain what was going on in their mind

37. Summary Build a cross-functional team to: When you test:
take advantage of different skills
get “buy in” from different areas
build awareness of usability testing benefits
When you test:
Set measurable objectives
Test with the right people
Be prepared for the test
Share your results

38. Questions?
For more information about this presentation and other training seminars, visit This presentation is at
Scott DeLoach
Founding Partner, User First Services, Inc.
Certified RoboHELP Instructor and Consultant