1. CS 544 Discount Usability Engineering
Heuristic Evaluation
Acknowledgement: Some of the material in these lectures is based on material prepared for similar courses by Saul Greenberg (University of Calgary), Ravin Balakrishnan (University of Toronto), James Landay (University of California at Berkeley), monica schraefel (University of Toronto), and Colin Ware (University of New Hampshire). Used with the permission of the respective original authors.

2. Usability Inspection
is the generic name for a set of methods based on having evaluators inspect or examine usability-related aspects of a user interface
usability inspectors can be usability specialists, software development consultants with special expertise, end users with task/content knowledge, or other professionals
4 ways to evaluate an interface:
automatically (compute usability measures using evaluation software)
empirically (usability testing with real users)
formally (using models to calculate usability measures)
informally (rule of thumb and general knowledge of evaluators)
usability inspection is an informal evaluation

3. Discount Usability Engineering
Developed by Nielson, based on using four techniques:
User and task observation
Scenarios
Simplified thinking aloud
Heuristic evaluation

4. Interface Design and Usability Engineering
Articulate:
who users are
their key tasks
Brainstorm designs
Refined designs
Completed designs
Goals:
Task centered system design
Participatory design
User-centered design
Psychology of everyday things
User involvement
Representation & metaphors
Graphical screen design
Interface guidelines
Style guides
Participatory interaction
Task scenario walk- through
Evaluate
Usability testing
Heuristic evaluation
Field testing
Methods:
low fidelity prototyping methods
high fidelity prototyping methods
Products:
User and task descriptions
Throw-away paper prototypes
Testable prototypes
Alpha/beta systems or complete specification

5. Discount usability engineering
Cheap
no special labs or equipment needed
the more careful you are, the better it gets
Fast
on order of 1 day to apply
standard usability testing may take a week
Easy to use
can be taught in 2-4 hours

6. Heuristic Evaluation Developed by Jakob Nielsen
Principles used to “evaluate” a system for usability problems
Becoming very popular
user involvement not required
catches many design flaws
Small set (3-5) of evaluators examine UI
independently check for compliance with usability principles (“heuristics”)
different evaluators will find different problems
evaluators only communicate afterwards
findings are then aggregated
Can perform on working UI or on sketches

7. Why multiple evaluators?
Every evaluator doesn’t find every problem
Good evaluators find both easy & hard ones

8. Heuristic Evaluation Process
Evaluators go through UI several times
inspect various dialogue elements
compare with list of usability principles
consider other principles/results that come to mind
Usability principles
Nielsen’s “heuristics”
supplementary list of category-specific heuristics
competitive analysis & user testing of existing products
Use violations to redesign/fix problems

9. Neilson’s Heuristics (original)
H1-1: Simple & natural dialog
H1-2: Speak the users’ language
H1-3: Minimize users’ memory load
H1-4: Consistency
H1-5: Feedback
H1-6: Clearly marked exits
H1-7: Shortcuts
H1-8: Precise & constructive error messages
H1-9: Prevent errors
H1-10: Help and documentation

10. H2-1: Visibility of system status
keep users informed about what is going on
example: pay attention to response time
0.1 sec: no special indicators needed, why?
1.0 sec: user tends to lose track of data
10 sec: max. duration if user to stay focused on action
for longer delays, use percent-done progress bars
searching database for matches

11. H2-1: Visibility of system status
keep users informed about what is going on
Appropriate visible feedback
What mode am I in now?
What did I select?
How is the system interpreting my actions?

12. H2-2: Match between system & real world
speak the users’ language
follow real world conventions
(bad) example: Mac desktop
Dragging disk to trash
should delete it, not eject it!

13. H2-2: Match between system & real world
speak the users’ language
follow real world conventions
e.g. withdrawing money from a bank machine

14. H2-3: User control & freedom
“exits” for mistaken choices, undo, redo
don’t force down fixed paths
How do I get out of this?

15. H2-3: User control & freedom
“exits” for mistaken choices, undo, redo
don’t force down fixed paths
Strategies:
Cancel button (for dialogs waiting for user input)
Universal Undo (can get back to previous state)
Interrupt (especially for lengthy operations)
Quit (for leaving the program at any time)
Defaults (for restoring a property sheet)

16. H2-4: Consistency & standards
Consistency of effects
same words, commands, actions will always have the same effect in equivalent situations
predictability
Consistency of language and graphics
same info/controls in same location on all screens/dialog boxes
forms follow boiler plate
same visual appearance across the system (e.g. widgets)
e.g. different scroll bars in a single window system!
Consistency of input
consistent syntax across complete system

17. H2-4: Consistency & standards
These are labels with a raised appearance.
Is it any surprise that people try and click on them?

18. H2-5: Error prevention try to make errors impossible
modern widgets: only “legal commands” selected, or “legal data” entered
Provide reasonableness checks on input data
on entering order for office supplies
5000 pencils is an unusually large order. Do you really want to order that many?

19. H2-6: Recognition rather than recall
Computers good at remembering things, people aren’t!
Promote recognition over recall
menus, icons, choice dialog boxes vs command lines, field formats
relies on visibility of objects to the user (but less is more!)

20. H2-7: Flexibility and efficiency of use
Experienced users should be able to perform frequently used operations quickly
Strategies:
keyboard and mouse accelerators
abbreviations
command completion
menu shortcuts
function keys
double clicking vs menu selection
type-ahead (entering input before the system is ready for it)
navigation jumps
e.g., going to window/location directly, and avoiding intermediate nodes
history systems
WWW: ~60% of pages are revisits

21. H2-7: Flexibility and efficiency of use
Keyboard accelerators for menus
Customizable toolbars and palettes for frequent actions
Split menu, with recently used fonts on top
Double-click raises toolbar dialog box
Double-click raises object-specific menu
Scrolling controls for page-sized increments

22. H2-8: Aesthetic and minimalist design
No irrelevant information in dialogues
Bad example:

23. H2-9: Help users recognize, diagnose, and recover from errors
error messages in plain language
precisely indicate the problem
constructively suggest a solution

24. Errors we make Mistakes Slips
arise from conscious deliberations that lead to an error instead of the correct solution
Slips
unconscious behaviour that gets misdirected en route to satisfying goal
e.g. drive to store, end up in the office
shows up frequently in skilled behaviour
usually due to inattention
often arises from similarities of actions

25. I can’t believe I pressed Yes...
Types of slips
Capture error
frequently done activity takes charge instead of one intended
occurs when common and rarer actions have same initial sequence
change clothes for dinner and find oneself in bed (William James, 1890)
confirm saving of a file when you don’t want to delete it
I can’t believe I pressed Yes...

26. Types of slips Description error Loss of activation Mode errors
intended action has much in common with others that are possible
usually occurs when right and wrong objects physically near each other
pour juice into bowl instead of glass
go jogging, come home, throw sweaty shirt in toilet instead of laundry basket
move file to trash instead of to folder
Loss of activation
forgetting what the goal is while undergoing the sequence of actions
start going to room and forget why you are going there
navigating menus/dialogs and can’t remember what you are looking for
but continue action to remember (or go back to beginning)!
Mode errors
people do actions in one mode thinking they are in another
refer to file that’s in a different directory
look for commands / menu options that are not relevant

27. Designing for slips General rules Examples
Prevent slips before they occur
Detect and correct slips when they do occur
User correction through feedback and undo
Examples
mode errors
have as few modes as possible (preferably none)
make modes highly visible
capture errors
instead of confirmation, make actions undoable
allows reconsideration of action by user
e.g. Mac trash can can be opened and “deleted” file taken back out
loss of activation
if system knows goal, make it explicit
if not, allow person to see path taken
description errors
in icon-based interfaces, make sure icons are not too similar,
check for reasonable input, etc.

28. Generic system responses for errors
General idea: Forcing functions
prevent / mitigate continuation of wrongful action
Gag
deals with errors by preventing the user from continuing
eg cannot get past login screen until correct password entered
Warn
warn people that an unusual situation is occurring
when overused, becomes an irritant
e.g.,
audible bell
alert box
Do nothing
illegal action just doesn’t do anything
user must infer what happened
enter letter into a numeric-only field (key clicks ignored)
put a file icon on top of another file icon (returns it to original position)

29. Generic system responses for errors
Self-correct
system guesses legal action and does it instead
but leads to a problem of trust
spelling corrector
Lets talk about it
system initiates dialog with user to come up with solution to the problem
compile error brings up offending line in source code
Teach me
system asks user what the action was supposed to have meant
action then becomes a legal one

30. H2-10: Help and documentation
Help is not a replacement for bad design!
Simple systems:
walk up and use; minimal instructions
Most other systems:
feature rich
some users will want to become “experts” rather than “casual” users
intermediate users need reminding, plus a learning path
Many users do not read manuals
prefer to spend their time pursuing their task
Usually used when users are in some kind of panic, need immediate help
indicates need for online documentation, good search/lookup tools
online help can be specific to current context
paper manuals unavailable in many businesses!
e.g. single copy locked away in system administrator’s office
Sometimes used for quick reference
syntax of actions, possibilities...
list of shortcuts ...

31. Types of help Tutorial and/or getting started manuals
short guides that people are likely to read when first obtaining their systems
encourages exploration and getting to know the system
tries to get conceptual material across and essential syntax
on-line “tours”, exercises, and demos
demonstrates very basic principles through working examples
Reference manuals
used mostly for detailed lookup by experts
rarely introduces concepts
thematically arranged
on-line hypertext
search / find
table of contents
index
cross-index

32. Types of help (cont.) Reminders short reference cards
expert user who just wants to check facts
novice who wants to get overview of system’s capabilities
keyboard templates
shortcuts/syntactic meanings of keys; recognition vs. recall; capabilities
tooltips
text over graphical items indicates their meaning or purpose

33. Types of help (cont.) Context-sensitive help Wizards
system provides help on the interface component the user is currently working with
Macintosh “balloon help”
Microsoft “What’s this” help
brief help explaining whatever the user is pointing at on the screen
Wizards
walks user through typical tasks
but dangerous if user gets stuck
Remember the MS “paperclip”?

34. How to perform evaluation
At least two passes for each evaluator
first to get feel for flow and scope of system
second to focus on specific elements
If system is walk-up-and-use or evaluators are domain experts, no assistance needed
otherwise might supply evaluators with scenarios
Each evaluator produces list of problems
explain why with reference to heuristic or other information
be specific and list each problem separately

35. How to perform evaluation (cont.)
Why separate listings for each violation?
risk of repeating problematic aspect
may not be possible to fix all problems
Where problems may be found
single location in UI
two or more locations that need to be compared
problem with overall structure of UI
something that is missing
hard w/ paper prototypes so work extra hard on those
note: sometimes features are implied by design docs and just haven’t been “implemented” – relax on those

36. Severity ratings Used to allocate resources to fix problems
Estimates of need for more usability efforts
Combination of
frequency
impact
persistence (one time or repeating)
Should be calculated after all evals. are in
Should be done independently by all judges

37. Severity ratings (cont).
0 - don’t agree that this is a usability problem
1 - cosmetic problem
2 - minor usability problem
3 - major usability problem; important to fix
4 - usability catastrophe; imperative to fix

38. Summary Advantages Problems: the “minimalist” approach
a few general guidelines can correct for the majority of usability problems
easily remembered, easily applied with modest effort
discount usability engineering
cheap and fast way to inspect a system
can be done by usability experts, double experts, and end users
Problems:
principles are more or less at the motherhood level
can’t be treated as a simple checklist
subtleties involved in their use
doesn’t necessarily predict user/customers overall satisfaction
may not have the same “credibility” as user test data (one solution to this is to include the design team and developers in the usability evaluation, as in a “pluralistic walkthrough”)

39. Summary Research result: Cost-benefit:
between 4-5 evaluators appear to be sufficient in most cases to identify 80% of total usability problems
user testing and usability inspection have a large degree of non-overlap in the usability problems they find (i.e., it pays to use both methods)
Cost-benefit:
usability engineering activities are often difficult to justify and carry out in a timely way, but many activities can be done quickly and cheaply, and produce useful results
usability inspection turns less on what is “correct” than on what can be done within development constraints
ultimate trade-off may be between doing no usability assessment and doing some kind

40. You now know 10 Heuristics for evaluating designs:
H2-1: Visibility of system status
H2-2: Match between system & real world
H2-3: User control & freedom
H2-4: Consistency & standards
H2-5: Error prevention
H2-6: Recognition rather than recall
H2-7: Flexibility and efficiency of use
H2-8: Aesthetic and minimalist design
H2-9: Help users recognize, diagnose, and recover from errors
H2-10: Help and documentation
Principles can be used to systematically inspect the interface for usability problems

41. Readings and References
BGBG (Evaluating Systems and Their User Interfaces)
BGBG (Usability Inspection Methods)
Optional:
Neilsen, J. (1993). Usability Engineering. Read Chapter 5: Usability Heuristics, 115 – 163.