Qualitative Evaluation Material Sources: Dr. Saul Greenberg, Univ. Calgary John Kelleher, IT Sligo

Definition How to quickly evaluate prototypes by observing people’s use of them How specific methods can help you discover what a person is thinking about as they are using your system produces a description, usually in non-numeric terms may be subjective

Methods Methods Introspection Extracting the conceptual model Observational Evaluation simple observation think-aloud constructive interaction ethnography Query via interviews and questionnaires Continuous evaluation via user feedback and field studies

Introspection Method Designer tries the system (or prototype) out does the system “feel right”? most common evaluation method Problems not reliable as completely subjective not valid as introspector is a non-typical user Intuitions and introspection are often wrong

Conceptual Model Extraction Show user the prototype / screen snapshots Have the user try to explain what all elements are what they would do to perform a particular task Excellent for extracting a novice’s understanding of system Poor for examining system exploration and learning

Observational Evaluation  Advantages Disadvantages Quickly highlights difficulties Verbal protocols valuable source of information Can be used for rapid iterative development Rich qualitative data Observation can affect user activity and performance levels. Analysis of data can be time-consuming and resource-consuming As the name suggests, this involves someone observing a small group (usually between 3 and 6 people) of (articulate) users as they work through specified tasks or benchmark tests. The users talk aloud as they work, describing not only what they are doing but also what they are thinking while they are doing it. Their actions and their comments are recorded by the observer using paper notes, video or audio recording. Advantages of the method are that any difficulties of the system are quickly highlighted and the users comments are a very valuable source of information for the developers, particularly about qualitative aspects of the interface. It can be used at any stage of the development and gives rapid feedback which can be quickly incorporated into the next refinement of the system. Some disadvantages are that the very fact of being observed can affect one’s performance (up or down) so that the test subjects might not be acting like ‘normal’ users. (This is known as “The Hawthorne Effect.) The technique can also be quite cumbersome in that many notes, or long recordings, are produced and it is quite a job to analyse these, taking perhaps quite a long time. The analysis might even involve more expensive ‘experts’, such as psychologists to analyse, say, body language observed on video tapes. In addition, the designer is depending on the test subjects being truthful and not holding anything back for fear of embarrassing either themselves or the designer.

Direct Observation Evaluator observes and records users interacting with design/system in lab user asked to complete a set of pre-determined tasks a specially instrumented usability lab may be available in field user goes through normal duties Excellent at identifying gross interface problems Validity/reliability depends on how controlled /contrived the situation is Three general approaches: simple observation (See “User Testing”)

Thinking Aloud Protocol Asked the user to talk while doing the task Have them tell what they are doing or trying to do as they perform the task Have them read a set of instructions aloud before they begin the task Helps to get them going Can use the task description for this Remind them to talk if they stop But be low-key about it Spontaneous comments are best – don’t direct user comments with specific questions: “What do you think those prompts mean?” “Why did you do that?”

Pros and Cons of Thinking Aloud finds many usability problems finds why they occur (process data) small number of test users (3 to 5) usable early in development process requires little administrator expertise generates colourful quotes Cons having to think aloud can influence user’s problem-solving behaviour cannot provide performance data (bottom-line data) Cannot rely on user explanations (Maier, Nisbett)

Constructive Interaction Method Two people work together on a task normal conversation between the two users is monitored removes awkwardness of think-aloud Variant: Co-discovery learning use semi-knowledgeable “coach” and naive subject together make naive subject use the interface Results in naive subject asking questions semi-knowledgeable coach responding provides insights into thinking process of both beginner and intermediate users

Recording Observations Evaluator may forget, miss, or misinterpret events Paper and pencil primitive but cheap evaluators record events, interpretations, and extraneous observations hard to get detail (slow writing); coding schemes help… Audio recording good for recording talk produced by thinking aloud/constructive interaction Hard to synch user actions and annotations Video recording can see and hear what a user is doing one camera for screen, another for subject (picture in picture) can be intrusive during initial period of use

Coding Scheme Example Tracking a person’s activity in the office

Ethnography Ethnographic study methods recognise that the investigator will have to interact directly with the subject, but while taking sufficient care to gain reasonably complete and objective information. An ethnographic study will attempt to observe subjects in a range of contexts, over a substantial period of time making a full record using any possible means (photography, video and sound recording as well as note-taking) of both activities and artefacts that the subject interacts with.

Survey Evaluation Advantages Disadvantages Addresses users’ opinions and understanding of interface. Can be made to be diagnostic Can be applied to users and designers Questions can be tailored to the individual Rating scales lead to quantitative results Can be used on a large group of users User experience is important Low response rates (especially by post) Possible interviewer bias Possible response bias Analysis can be complicated and lengthy Interviews very time-consuming Surveys can be carried out through data collection methods such as interviews or questionnaires. There are many variations on this theme and many of the detailed points will be found in Preece pp 628 - 638 Interviews - structured, flexible, semi-structured, prompted. Also, consider interviewer/interviewee characteristics, especially things like bias. If, for example, the interviewer is one of the design team and quite proud of the product produced, this attitude can be put across unintentionally. The interviewee senses that the interviewer doesn’t really want criticism of ‘his baby’, and so the interviewee gives him “what he wants to hear” rather than the “truth”. Questionnaires - One of the main considerations is whether the questions should be of the open or closed type, and whether or not some kind of rating scale is to be used to try to give a quantitative aspect to the responses. Careful consideration must also be given to who, exactly, will you collect the data from. The test subjects need to have sufficient experience with the system to be able to operate competently and to answer the questions about it. Perhaps the biggest problems with surveys is that they can be very time-consuming to organise, implement, and then analyse the perhaps quite complex data that results. With questionnaires, particularly postal questionnaires, it is sometimes very difficult to obtain sufficient returns to make the exercise worthwhile (e.g. in terms of statistical significance).

Interviews Excellent for pursuing specific issues Problems: vary questions to suit the context probe more deeply on interesting issues as they arise good for exploratory studies via open-ended questioning often leads to specific constructive suggestions Problems: accounts are subjective time consuming evaluator can easily bias the interview prone to rationalization of events/thoughts by user user’s reconstruction may be wrong Users recall away from context poor

How to Interview? Plan a set of central questions could be based on results of user observations gets things started focuses the interview ensures a base of consistency Try not to ask leading questions Start with individual discussions to discover different perspectives, and continue with group discussions the larger the group, the more the universality of comments can be ascertained also encourages discussion between users

Questions to elicit task detail Purpose 1. What do you do? Obtains the user’s goal 2. Why do you do it? Obtains method 3. How do you do it? Obtains the subtask. Used recursively for each subtask 4. What are the preconditions for doing this? To find out what outside influences there are 5. What are the results of doing this? To examine the product and see what the purpose is 6. What errors occur? Error capture 7. How do you correct them? Error correction

Challenges Users exhibit poor knowledge of exceptions (even regularly occurring ones) Users exhibit poor recollection of work domain in general Ethnography a possible solution for widening social and organisation context Users exhibit little correlation between what they said they would prefer and what they ultimately did prefer Interviews not touch-stone for system acceptability

Retrospective Testing Post-observation interview to clarify events that occurred during system use perform an observational test create a video record of it have users view the video and comment on what they did excellent for grounding a post-test interview avoids erroneous reconstruction users often offer concrete suggestions

Questionnaires & Surveys Preparation “expensive”, but administration cheap can reach a wide subject group (e.g. mail) Can be repeated to establish trends does not require presence of evaluator easy to quantify and analyse results only as good as the questions asked less flexible than interview Good for subjective responses

Questionnaire Design Establish the purpose of the questionnaire what information is sought? how would you analyze the results? what would you do with your analysis? Do not ask questions whose answers you will not use! (e.g. how old are you?) Determine the audience you want to reach Determine how you will deliver and collect the questionnaire interview administered questionnaire on-line for computer users web site with forms surface mail (include SAE for better response) determine the demographics

Questionnaire Design Keep the number of questions low Only questions with answers that you can’t get other ways Only questions that will have a direct impact on functional requirements Avoid asking for everything Avoid questions that users perceive as self-incriminating or inferior Ask clear questions precision in framing questions a problem Ask questions that users can answer validly and reliably Does the user store information in this way? Does the user remember such information? Will the user be inclined to answer your question truthfully?

Styles of Questions Open-ended questions asks for unprompted opinions good for general subjective information but difficult to analyze rigorously I found the following aspects particularly easy to use (please list 0–3 aspects): __________________________________ __________________________________ __________________________________

Style of Questions Closed questions restricts the respondent’s responses by supplying alternative answers makes questionnaires a chore for respondent to fill in can be easily analyzed but watch out for hard to interpret responses! alternative answers should be very specific Do you use computers at work:  often  sometimes  rarely vs In your typical day, do you use computers:  over 4 hrs a day  between 2 and 4 hrs daily  between 1and 2 hrs daily  less than 1 hr a day

Style of Questions Likert Scalar ask user to judge a specific statement on a numeric scale (5, 6 or 7 point scale best) scale usually corresponds with agreement or disagreement with a statement The new drawing package is easy to use: 1 2 3 4 5 Strongly agree agree neither agree nor disagree disagree strongly disagree

Style of Questions Semantic Differentials sliding scale between opposing pairs of adjectives (5 or 7 point scale best). Rate the Design and Go drawing package on the following criteria: Very Quite Neutral Simple 1 2 3 4 5 Complex Professional Unprofessional Reliable Unreliable Attractive Unattractive

Style of Questions Ranked respondent places an ordering on items in a list useful to indicate a user’s preferences forced choice Rank the usefulness of these methods of issuing a command (1 most useful, 2 next most useful..., 0 if not used) __2__ command line __1__ menu selection __3__ control key accelerator

Style of Questions Combining open-ended and closed questions gets specific response, but allows room for user’s opinion It is easy to recover from mistakes: Disagree…………….Agree 1 2 3 4 5 comment: undo facility is really helpful

Pros and Cons of Questionnaires Collect subjective user view of system Both quantitative and qualitative data Useful supplementary information Simple Cheap Subjective data often unreliable

Continuous Evaluation Usually done in later stages of development (i.e. beta releases, delivered system) Good for monitoring problems of system in actual use Problems can be fixed in next release a) User feedback via gripe lines users can provide feedback to designers while using the system Email and web forms special built-in gripe facility telephone hot line (help desks) suggestion box (in-house operations) best combined with trouble-shooting facility users always get a response (solution?) to their gripes

Continuous Evaluation b) Case/field studies careful study of “system usage” at the site E.g. Quicken by Intuit good for seeing “real life” use external observer monitors behaviour or gets feedback via methods described above Advantages and Dis-advantages natural environment context retained (ecological validity) longitudinal studies possible cost distractions noise

Summary Observing a range of users use your system for specific tasks reveals successes and problems Qualitative observational tests are quick and easy to do Several methods reveal what is in a person’s head as they are doing the test Particular methods include: Conceptual model extraction Direct observation Simple observation Think-aloud Constructive interaction Query via interviews, retrospective testing and questionnaires Continuous evaluation via user feedback and field studies

Summary of Evaluation Methods