0. Qualitative Evaluation
Material Sources: Dr. Saul Greenberg, Univ. Calgary John Kelleher, IT Sligo

1. Definition
How to quickly evaluate prototypes by observing people’s use of them
How specific methods can help you discover what a person is thinking about as they are using your system
produces a description, usually in non-numeric terms
may be subjective

2. Methods Methods Introspection Extracting the conceptual model
Observational Evaluation
simple observation
think-aloud
constructive interaction
ethnography
Query via interviews and questionnaires
Continuous evaluation via user feedback and field studies

3. Introspection Method Designer tries the system (or prototype) out
does the system “feel right”?
most common evaluation method
Problems
not reliable as completely subjective
not valid as introspector is a non-typical user
Intuitions and introspection are often wrong

4. Conceptual Model Extraction
Show user the prototype / screen snapshots
Have the user try to explain
what all elements are
what they would do to perform a particular task
Excellent for extracting a novice’s understanding of system
Poor for examining system exploration and learning

5. Observational Evaluation
Advantages
Disadvantages
Quickly highlights difficulties
Verbal protocols valuable source of information
Can be used for rapid iterative development
Rich qualitative data
Observation can affect user activity and performance levels.
Analysis of data can be time-consuming and resource-consuming
As the name suggests, this involves someone observing a small group (usually between 3 and 6 people) of (articulate) users as they work through specified tasks or benchmark tests. The users talk aloud as they work, describing not only what they are doing but also what they are thinking while they are doing it. Their actions and their comments are recorded by the observer using paper notes, video or audio recording.
Advantages of the method are that any difficulties of the system are quickly highlighted and the users comments are a very valuable source of information for the developers, particularly about qualitative aspects of the interface. It can be used at any stage of the development and gives rapid feedback which can be quickly incorporated into the next refinement of the system.
Some disadvantages are that the very fact of being observed can affect one’s performance (up or down) so that the test subjects might not be acting like ‘normal’ users. (This is known as “The Hawthorne Effect.)
The technique can also be quite cumbersome in that many notes, or long recordings, are produced and it is quite a job to analyse these, taking perhaps quite a long time. The analysis might even involve more expensive ‘experts’, such as psychologists to analyse, say, body language observed on video tapes.
In addition, the designer is depending on the test subjects being truthful and not holding anything back for fear of embarrassing either themselves or the designer.

6. Direct Observation
Evaluator observes and records users interacting with design/system
in lab
user asked to complete a set of pre-determined tasks
a specially instrumented usability lab may be available
in field
user goes through normal duties
Excellent at identifying gross interface problems
Validity/reliability depends on how controlled /contrived the situation is
Three general approaches:
simple observation (See “User Testing”)

7. Thinking Aloud Protocol
Asked the user to talk while doing the task
Have them tell what they are doing or trying to do as they perform the task
Have them read a set of instructions aloud before they begin the task
Helps to get them going
Can use the task description for this
Remind them to talk if they stop
But be low-key about it
Spontaneous comments are best – don’t direct user comments with specific questions:
“What do you think those prompts mean?”
“Why did you do that?”

8. Pros and Cons of Thinking Aloud
finds many usability problems
finds why they occur (process data)
small number of test users (3 to 5)
usable early in development process
requires little administrator expertise
generates colourful quotes
Cons
having to think aloud can influence user’s problem-solving behaviour
cannot provide performance data (bottom-line data)
Cannot rely on user explanations (Maier, Nisbett)

9. Constructive Interaction Method
Two people work together on a task
normal conversation between the two users is monitored
removes awkwardness of think-aloud
Variant: Co-discovery learning
use semi-knowledgeable “coach” and naive subject together
make naive subject use the interface
Results in
naive subject asking questions
semi-knowledgeable coach responding
provides insights into thinking process of both beginner and intermediate users

10. Recording Observations
Evaluator may forget, miss, or misinterpret events
Paper and pencil
primitive but cheap
evaluators record events, interpretations, and extraneous observations
hard to get detail (slow writing); coding schemes help…
Audio recording
good for recording talk produced by thinking aloud/constructive interaction
Hard to synch user actions and annotations
Video recording
can see and hear what a user is doing
one camera for screen, another for subject (picture in picture)
can be intrusive during initial period of use

11. Coding Scheme Example
Tracking a person’s activity in the office

12. Ethnography
Ethnographic study methods recognise that the investigator will have to interact directly with the subject, but while taking sufficient care to gain reasonably complete and objective information.
An ethnographic study will attempt to observe subjects in a range of contexts,
over a substantial period of time
making a full record using any possible means
(photography, video and sound recording as well as note-taking) of
both activities and
artefacts that the subject interacts with.

13. Survey Evaluation Advantages Disadvantages
Addresses users’ opinions and understanding of interface.
Can be made to be diagnostic
Can be applied to users and designers
Questions can be tailored to the individual
Rating scales lead to quantitative results
Can be used on a large group of users
User experience is important
Low response rates (especially by post)
Possible interviewer bias
Possible response bias
Analysis can be complicated and lengthy
Interviews very time-consuming
Surveys can be carried out through data collection methods such as interviews or questionnaires. There are many variations on this theme and many of the detailed points will be found in Preece pp
Interviews - structured, flexible, semi-structured, prompted.
Also, consider interviewer/interviewee characteristics, especially things like bias. If, for example, the interviewer is one of the design team and quite proud of the product produced, this attitude can be put across unintentionally. The interviewee senses that the interviewer doesn’t really want criticism of ‘his baby’, and so the interviewee gives him “what he wants to hear” rather than the “truth”.
Questionnaires - One of the main considerations is whether the questions should be of the open or closed type, and whether or not some kind of rating scale is to be used to try to give a quantitative aspect to the responses.
Careful consideration must also be given to who, exactly, will you collect the data from. The test subjects need to have sufficient experience with the system to be able to operate competently and to answer the questions about it.
Perhaps the biggest problems with surveys is that they can be very time-consuming to organise, implement, and then analyse the perhaps quite complex data that results. With questionnaires, particularly postal questionnaires, it is sometimes very difficult to obtain sufficient returns to make the exercise worthwhile (e.g. in terms of statistical significance).

14. Interviews Excellent for pursuing specific issues Problems:
vary questions to suit the context
probe more deeply on interesting issues as they arise
good for exploratory studies via open-ended questioning
often leads to specific constructive suggestions
Problems:
accounts are subjective
time consuming
evaluator can easily bias the interview
prone to rationalization of events/thoughts by user
user’s reconstruction may be wrong
Users recall away from context poor

15. How to Interview? Plan a set of central questions
could be based on results of user observations
gets things started
focuses the interview
ensures a base of consistency
Try not to ask leading questions
Start with individual discussions to discover different perspectives, and continue with group discussions
the larger the group, the more the universality of comments can be ascertained
also encourages discussion between users

16. Questions to elicit task detail
Purpose
1. What do you do?
Obtains the user’s goal
2. Why do you do it?
Obtains method
3. How do you do it?
Obtains the subtask. Used recursively for each subtask
4. What are the preconditions for doing this?
To find out what outside influences there are
5. What are the results of doing this?
To examine the product and see what the purpose is
6. What errors occur?
Error capture
7. How do you correct them?
Error correction

17. Challenges
Users exhibit poor knowledge of exceptions (even regularly occurring ones)
Users exhibit poor recollection of work domain in general
Ethnography a possible solution for widening social and organisation context
Users exhibit little correlation between what they said they would prefer and what they ultimately did prefer
Interviews not touch-stone for system acceptability

18. Retrospective Testing
Post-observation interview to clarify events that occurred during system use
perform an observational test
create a video record of it
have users view the video and comment on what they did
excellent for grounding a post-test interview
avoids erroneous reconstruction
users often offer concrete suggestions

19. Questionnaires & Surveys
Preparation “expensive”, but administration cheap
can reach a wide subject group (e.g. mail)
Can be repeated to establish trends
does not require presence of evaluator
easy to quantify and analyse results
only as good as the questions asked
less flexible than interview
Good for subjective responses

20. Questionnaire Design Establish the purpose of the questionnaire
what information is sought?
how would you analyze the results?
what would you do with your analysis?
Do not ask questions whose answers you will not use! (e.g. how old are you?)
Determine the audience you want to reach
Determine how you will deliver and collect the questionnaire
interview administered questionnaire
on-line for computer users
web site with forms
surface mail (include SAE for better response)
determine the demographics

21. Questionnaire Design Keep the number of questions low
Only questions with answers that you can’t get other ways
Only questions that will have a direct impact on functional requirements
Avoid asking for everything
Avoid questions that users perceive as self-incriminating or inferior
Ask clear questions
precision in framing questions a problem
Ask questions that users can answer validly and reliably
Does the user store information in this way?
Does the user remember such information?
Will the user be inclined to answer your question truthfully?

22. Styles of Questions Open-ended questions asks for unprompted opinions
good for general subjective information
but difficult to analyze rigorously
I found the following aspects particularly easy to use (please list 0–3 aspects): __________________________________ __________________________________ __________________________________

23. Style of Questions Closed questions
restricts the respondent’s responses by supplying alternative answers
makes questionnaires a chore for respondent to fill in
can be easily analyzed
but watch out for hard to interpret responses!
alternative answers should be very specific
Do you use computers at work:
 often  sometimes  rarely vs
In your typical day, do you use computers:
 over 4 hrs a day
 between 2 and 4 hrs daily
 between 1and 2 hrs daily
 less than 1 hr a day

24. Style of Questions Likert Scalar
ask user to judge a specific statement on a numeric scale (5, 6 or 7 point scale best)
scale usually corresponds with agreement or disagreement with a statement
The new drawing package is easy to use:
Strongly agree agree neither agree nor disagree disagree strongly disagree

25. Style of Questions Semantic Differentials
sliding scale between opposing pairs of adjectives (5 or 7 point scale best).
Rate the Design and Go drawing package on the following criteria:
Very
Quite
Neutral
Simple 1 2 3 4 5
Complex
Professional
Unprofessional
Reliable
Unreliable
Attractive
Unattractive

26. Style of Questions Ranked
respondent places an ordering on items in a list
useful to indicate a user’s preferences
forced choice
Rank the usefulness of these methods of issuing a command
(1 most useful, 2 next most useful..., 0 if not used)
__2__ command line
__1__ menu selection
__3__ control key accelerator

27. Style of Questions Combining open-ended and closed questions
gets specific response, but allows room for user’s opinion
It is easy to recover from mistakes:
Disagree…………….Agree
comment: undo facility is really helpful

28. Pros and Cons of Questionnaires
Collect subjective user view of system
Both quantitative and qualitative data
Useful supplementary information
Simple
Cheap
Subjective data often unreliable

29. Continuous Evaluation
Usually done in later stages of development
(i.e. beta releases, delivered system)
Good for monitoring problems of system in actual use
Problems can be fixed in next release
a) User feedback via gripe lines
users can provide feedback to designers while using the system
and web forms
special built-in gripe facility
telephone hot line (help desks)
suggestion box (in-house operations)
best combined with trouble-shooting facility
users always get a response (solution?) to their gripes

30. Continuous Evaluation
b) Case/field studies
careful study of “system usage” at the site
E.g. Quicken by Intuit
good for seeing “real life” use
external observer monitors behaviour or gets feedback via methods described above
Advantages and Dis-advantages
natural environment
context retained (ecological validity)
longitudinal studies possible
cost
distractions
noise

31. Summary
Observing a range of users use your system for specific tasks reveals successes and problems
Qualitative observational tests are quick and easy to do
Several methods reveal what is in a person’s head as they are doing the test
Particular methods include:
Conceptual model extraction
Direct observation
Simple observation
Think-aloud
Constructive interaction
Query via interviews, retrospective testing and questionnaires
Continuous evaluation via user feedback and field studies

32. Summary of Evaluation Methods