James Tam Qualitative Evaluation Techniques Why evaluation is crucial to interface design General approaches and tradeoffs with the different approaches.

Slides:



Advertisements
Similar presentations
Human Computer Interaction
Advertisements

Evaluating interfaces with users Why evaluation is crucial Quickly debug prototypes by observing people use them Methods reveal what a person is thinking.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Qualitative Evaluation Why evaluation is crucial Quickly debug prototypes by observing people use them Methods reveal what a person is thinking about Slide.
6.811 / PPAT: Principles and Practice of Assistive Technology Wednesday, 16 October 2013 Prof. Rob Miller Today: User Testing.
CS305: HCI in SW Development Evaluation (Return to…)
Deciding How to Measure Usability How to conduct successful user requirements activity?
Usability Evaluation with Users CMPT 281. Outline Usability review Observational methods Interview methods Questionnaire methods.
Qualitative Evaluation Techniques
Saul Greenberg User Centered Design Why User Centered Design is important Approaches to User Centered Design.
CPSC 481 Foundations and Principles of Human Computer Interaction
Saul Greenberg CPSC 481 Foundations and Principles of Human Computer Interaction James Tam.
Evaluation 3: Approaches and Data Collection Slides adapted from Saul Greenberg’s “Evaluating Interfaces with Users”.
Qualitative Evaluation Techniques
Data gathering.
Saul Greenberg Qualitative Evaluation Techniques Quickly debug and evaluate prototypes by observing people using them Specific evaluation methods helps.
©N. Hari Narayanan Computer Science & Software Engineering Auburn University 1 COMP 7620 Evaluation Chapter 9.
James Tam User Centered Design Why User Centered Design is important Approaches to User Centered Design.
Saul Greenberg CPSC 481 Foundations and Principles of Human Computer Interaction James Tam.
Useability.
Definition How to quickly evaluate prototypes by observing people’s use of them How specific methods can help you discover what a person is thinking about.
An evaluation framework
Data collection methods Questionnaires Interviews Focus groups Observation –Incl. automatic data collection User journals –Arbitron –Random alarm mechanisms.
James Tam CPSC 481 Foundations and Principles of Human Computer Interaction James Tam.
Qualitative Evaluation Techniques
Human Computer Interaction Design Evaluation and Content Presentation
Saul Greenberg Evaluating Interfaces With Users Why evaluation is crucial to interface design General approaches and tradeoffs in evaluation The role of.
ICS 463, Intro to Human Computer Interaction Design: 8. Evaluation and Data Dan Suthers.
ISE554 The WWW 3.4 Evaluation Methods. Evaluating Interfaces with Users Why evaluation is crucial to interface design General approaches and tradeoffs.
James Tam Evaluating Interfaces With Users Why evaluation is crucial to interface design General approaches and tradeoffs in evaluation The role of ethics.
James Tam Qualitative Evaluation Techniques Quickly debug and evaluate prototypes by observing people using them Specific evaluation methods helps you.
RESEARCH METHODS IN EDUCATIONAL PSYCHOLOGY
류 현 정류 현 정 Human Computer Interaction Introducing evaluation.
Predictive Evaluation
Presentation: Techniques for user involvement ITAPC1.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Evaluation Techniques Material from Authors of Human Computer Interaction Alan Dix, et al.
Research methods in psychology Simple revision points.
Data Collection Methods
Requirements Engineering Requirements Elicitation Process Lecture-8.
Human Computer Interaction
Object-Oriented Software Engineering Practical Software Development using UML and Java Chapter 7: Focusing on Users and Their Tasks.
Usability Testing CS774 Human Computer Interaction Spring 2004.
Usability Testing Chapter 6. Reliability Can you repeat the test?
Midterm Stats Min: 16/38 (42%) Max: 36.5/38 (96%) Average: 29.5/36 (78%)
COMP5047 Pervasive Computing: 2012 Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research.
Ways of Collecting Information Interviews Questionnaires Ethnography Books and leaflets in the organization Joint Application Design Prototyping.
Design and Qualitative Analysis. Why is Usability Important? We are not the users – Even design experts don’t think the same way as all other humans Users.
Marketing Research Approaches. Research Approaches Observational Research Ethnographic Research Survey Research Experimental Research.
JS Mrunalini Lecturer RAKMHSU Data Collection Considerations: Validity, Reliability, Generalizability, and Ethics.
Facilitate Group Learning
Usability Evaluation, part 2. REVIEW: A Test Plan Checklist, 1 Goal of the test? Specific questions you want to answer? Who will be the experimenter?
Fall 2002CS/PSY Empirical Evaluation Data collection: Subjective data Questionnaires, interviews Gathering data, cont’d Subjective Data Quantitative.
EVALUATION PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS ( TEMPUS FR-TEMPUS-JPCR)
AVI/Psych 358/IE 340: Human Factors Data Gathering October 3, 2008.
Ethics in Evaluation Why ethics? What you have to do Slide deck by Saul Greenberg. Permission is granted to use this for non-commercial purposes as long.
Requirements Engineering Processes. Syllabus l Definition of Requirement engineering process (REP) l Phases of Requirements Engineering Process: Requirements.
Usability Evaluation or, “I can’t figure this out...do I still get the donuts?”
How do we know if our UI is good or bad?.
1 ITM 734 Introduction to Human Factors in Information Systems Cindy Corritore Testing the UI – part 2.
School of Engineering and Information and Communication Technology KIT305/607 Mobile Application Development Week 7: Usability (think-alouds) Dr. Rainer.
Imran Hussain University of Management and Technology (UMT)
Lecture3 Data Gathering 1.
Evaluating interfaces with users
Usability Evaluation, part 2
Prototyping.
Evaluating interfaces with users
HCI Evaluation Techniques
Empirical Evaluation Data Collection: Techniques, methods, tricks Objective data IRB Clarification All research done outside the class (i.e., with non-class.
Presentation transcript:

James Tam Qualitative Evaluation Techniques Why evaluation is crucial to interface design General approaches and tradeoffs with the different approaches to evaluation The role of ethics Learning how to quickly debug and evaluate prototypes by observing people using them Specific evaluation methods helps you discover people’s thoughts and motivations as they are using your system

James Tam Why Bother With Evaluation? Tied to all parts of the usability engineering lifecycle Pre-design -Investing in new expensive system requires proof of viability Initial design stages -Develop and evaluate initial design ideas with the user design implementationevaluation

James Tam Why Bother With Evaluation (2)? Iterative design -Does the system match the user’s task requirements? -Are there any specific problems with the design? -Can users provide feedback to modify the design Acceptance testing -Verify that the system meets expected performance criteria: ease of learning, usability, user’s attitude -E.g., A first time user will take between one and three minutes to learn how to withdraw $50 from the ATM. design implementationevaluation

James Tam Approaches: Naturalistic Observation occurs in realistic setting Real life Problems -It may be difficult to arrange and to conduct -It may not always possible to replicate results

James Tam Approaches: Experimental Experimental Classical lab study Study relations by manipulating one or more independent variables -Experimenter controls all environmental factors (nothing else is different) Observe effect on one or more dependent variables

James Tam Tradeoffs: Natural Vs. Experimental Internal validity Do you measure what you set out to measure (correctness) External validity The degree to which results can be generalized to other situations (realism) NaturalisticExperimental Internal validity LowHigh External validity HighLow

James Tam (External) Validity Concerns Does the test measure something of relevance to usability of real products in real use outside of lab? Some typical reliability problems of testing vs. real use -Non-typical users tested -Tasks are not typical tasks -Physical environment different quiet lab vs.. very noisy open offices vs. interruptions -Social influences different motivation towards experimenter vs. motivation towards boss Partial Solution Use real users Use tasks derived from a task-centered approach to system design Test in an environment similar to real situation

James Tam How Many Participants To Test Would the same results be achieved if the test were repeated? Reliability of data Problem: individual differences: The best user 10x faster than slowest The best 25% of users ~2x faster than slowest 25% Partial Solution Get a reasonable number and range of test participants Images from “The Simpsons” © Fox

James Tam Test Roles Test participant Scribe Test administrator Security (optional)

James Tam Test Procedure I) Run a pilot study II) Run the main test “A practice run” of the test Purpose: To debug the test Results: Used to improve the test Running the test “for real” Purpose: To debug the interface Results: Used to improve the interface

James Tam Test Procedure (2) 1.Preparation 2.Introduction 3.Running the system 4.Debriefing

James Tam Ethics... and to think that you want me to test it!!!

James Tam Ethics Testing can be a distressing experience People feel pressure to perform so errors are inevitable This can result in: -Feelings of inadequacy -Competition with other test participants Golden rule Test participants should always be treated with respect

James Tam Managing Participants In An Ethical Manner Before the test Don’t waste the person’s time -Use pilot tests to debug experiments, questionnaires etc -Have everything ready before the participant shows up -Try it out yourself one more time Make participants feel comfortable -Emphasize that it is the system that is being tested, not the person -Acknowledge that the software may have problems -Let participants know they can stop at any time Maintain privacy -Tell the participant that individual test results will be kept completely confidential Inform the participant -Explain any monitoring that is being used -Answer all of the person’s questions (but avoid biasing them) Only use volunteers -Typically the test participant must sign an informed consent form

James Tam Managing Participants In An Ethical Manner During the test Don’t waste the person’s time -Never have the user perform unnecessary tasks Make test participants comfortable -Try to give the person an early success experience -Keep a relaxed atmosphere in the room -Have coffee, breaks, etc -Hand out test tasks one at a time -Never indicate displeasure with the person’s performance -Avoid disruptions -Stop the test if it becomes too unpleasant Maintain privacy -This class: Only show test results to people when it is essential (TA and course instructor) -Actual practice: Do not allow the participant’s management to observe the test

James Tam Managing Participants In An Ethical Manner After the test Make the person feel comfortable -e.g., state that the participant has helped you find areas of improvement Inform the participant -Answer particular questions about the experiment that could have biased the results before Maintain privacy -Never report results in a way that individuals can be identified -Only show videotapes outside the research group with the participant’s permission

James Tam *0# R Pause HOLD CODED DIAL /DIRECTORY V ^ < > PRINTER memory trans delayed trans delayed polling polling confd trans relay broadcareport + D.T. Tone spaceclear ON LINE PRINTER ERROR HS HQ PRINT MODE SHQ PRINTER INTERFACE Canon Fax-B320 Bubble Jet Facsimile

James Tam Discount Usability Evaluation Low cost methods to gather usability problems Approximate: Capture most large and many minor problems How? Quantitative Qualitative

James Tam Quantitative Approach For Usability Evaluation Description of approach: Measure something of interest in user actions Count, log, speed, error rate

James Tam Qualitative Methods For Usability Evaluation Description of approach: Data gathering -Observe the actions of the user -Gather opinions from the user Produces a description, usually in non-numeric terms May be quite subjective Methods Inspection Extracting the conceptual model Direct observation -Simple observation -Think-aloud -Constructive interaction Query via interviews and questionnaires Continuous evaluation via user feedback and field studies Star Trek IV: The Voyage Home © Paramount Pictures

James Tam The Inspection Method Designer tries the system (or prototype) out Does the system “feel right”? Benefits Can probably notice some major problems in early versions during every day use Problems Low reliability rate as completely subjective Low level of validity as inspector is a non-typical user Intuitions and introspections are often wrong

James Tam Extracting The Conceptual Model Show the user static images of: The paper prototype or Screen snapshots or Actual system screens during use Have the user try to explain What all elements are What they would do to perform a particular task How they think that the system works Initial vs.. formative conceptual models Initial: How person perceives a screen the very first time it is viewed Formative: The same, except after the system has been used for a while

James Tam Extracting The Conceptual Model (2) This approach is: Good for eliciting people’s understanding before & after use Requires active intervention by evaluator, which can get in the way

James Tam Direct Observation Evaluator observes and records users interacting with design/system In lab: -User asked to complete a set of pre-determined tasks -A specially built and fully instrumented usability lab may be available In field: -User goes through normal duties This approach is: Validity/reliability depends on how controlled/contrived the situation is Excellent at identifying gross design/interface problems Three general approaches: Simple observation/Silent observer Think-aloud Constructive interaction

James Tam Simple Observation Method Person is given the task, and evaluator silently just watches while employing “The Silent Observer” technique. Problem Does not give insight into the person’s decision process or attitude He must really hate this system!

James Tam The Think Aloud Method Test participants are asked to say what they are thinking/doing Gives insight into what the person is thinking -What they believe is happening -What they are trying to do -Why they took an action Hmm, what does this do? I’ll try it… Ooops, now what happened?

James Tam The Think Aloud Method (2) Problems -Awkward/uncomfortable for person (thinking aloud is not normal!) -Hard to talk when they are concentrating on problem -“Thinking” about it may alter the way people perform their task (could improve or degrade performance) -Certain situations may prohibit the use of this technique Most widely used “formal” evaluation method in industry

James Tam The Constructive Interaction Method Two people work together on a task Normal conversation between the two users is monitored -Removes awkwardness of think-aloud Variant: Co-discovery learning -Use semi-knowledgeable “coach” and novice user together -Only novice uses the interface -Results in: Novice user asking questions Semi-knowledgeable coach responding Provides insights into thinking process of both user groups Now, why did it do that? Oh, I think you clicked on the wrong icon

James Tam Querying People Via Interviews Use a set of pre-created questions Gets things started Focuses the interview Ensures a base of consistency Be sure to follow interesting leads rather than bulldozing through question list Adding additional questions Could be based on results of user observations Don’t forget Balance each question Avoid bias -Try not to ask leading questions

James Tam Issues Associated With Interviews Excellent for pursuing specific issues Flexible -You can vary questions to suit the context Provides a rich depth of data -Probe more deeply on interesting issues as they arise -Often leads to specific constructive suggestions Problems: Time consuming Evaluator can easily bias the interview Requires a skilled and/or experienced interviewer Accounts are subjective Prone to rationalization of events/thoughts by person -Reconstruction may be wrong

James Tam Group Discussions Start with individual discussions to discover different perspectives, and then continue with group discussions Increasing group size may increase the universality of the comments May encourage cross discussions

James Tam Retrospective Testing A special type of interviewing technique that was developed in order to address the weaknesses of traditional interviews. Post-observation interview to clarify events that occurred during system use Approach: 1.Perform an observational test while recording the session on video

James Tam Retrospective Testing (2) 2.Watch the video with the users and encourage them to comment on what they did Do you know why you never tried that option? I didn’t see it. Why don’t you make it look like a button?

James Tam Retrospective Testing (3) Benefits Excellent for grounding a post-test interview Avoids erroneous reconstruction Users often offer more concrete suggestions Unlike the silent observer approach it provides insights into what the person is thinking/feeling and it can be used when thinking aloud is not possible Drawbacks Much like traditional interviews it can be very time consuming

James Tam Querying People Via Questionnaires And Surveys Questionnaires / Surveys Written queries for usability information Benefits Administration cheap -Can reach a wide test group (e.g., mail) Administration requires little training Anonymous Drawbacks Preparation “expensive” – although this may balanced off by the administrative savings Inflexible

James Tam Querying People Via Questionnaires / Surveys (2) Approach for all types Establish the purpose of the questionnaire -What information is sought? -How would you analyze the results? -What would you do with your analysis? Do not ask questions whose answers you will not use! -e.g. How old are you? Determine the audience you want to reach -Typical survey: random sample of between 50 and 1000 users of the system Determine how would you will deliver and collect the questionnaire -On-line for computer users (e.g., web site with fill-in forms) -Surface mail Be sure to include a pre-addressed reply envelope to get a far better response rate

James Tam Querying Users Via Questionnaires / Surveys (3) Determine the demographics -e.g., computer experience

James Tam Style Of Questions Open-ended questions Asks for unprompted opinions Good for general subjective information but difficult to analyze rigorously e.g., Can you suggest any improvements to the interface?

James Tam Style Of Questions Closed-ended questions Restricts the respondent’s responses by supplying alternative answers Data is more narrow (less rich but can be easily analyzed) But watch out for hard to interpret responses - alternative answers should be very specific Types: scalar, multiple choice, ranked Examples: Do you use computers at work: O Often O Sometimes O Rarely vs.. In your typical work day, do you use computers: O Over 4 hrs a day O Between 2 and 4 hrs daily O Between 1and 2 hrs daily O Less than 1 hr a day

James Tam Closed-Ended Questions: Scalar Scalar Ask user to judge a specific statement on a numeric scale Scale usually corresponds with agreement or disagreement with a statement Characters on the computer screen are: Hard to read Easy to read

James Tam Closed-Ended Questions: Multiple Choice Multi-choice Respondent offered a choice of explicit responses How do you most often get help with the system? (Check only one category) O On-line manual O Paper manual O Ask a colleague Which types of software have you used? (Check all that apply) O Word processor O Data base O Spreadsheet O Compiler

James Tam Closed-Ended Questions: Ranked Ranked Respondent places an ordering on items in a list Useful to indicate a user’s preferences Forces a choice Rank the usefulness of these methods of giving a command to the computer (1 = Most useful, 2 = Next most useful..., 0 = Not used __2__ Command line __1__ Menu selection __3__ Control key accelerator

James Tam Mixing Questionnaire Styles Combining open-ended and closed-ended questions Get a specific response, but allows room for user’s opinion It is easy to recover from mistakes: Disagree Agree Comment: The undo facility is really helpful

James Tam Interviews Vs. Questionnaires: Summary Of The Pros And Cons Preparation time Unanticipated/unexpected events Depth of information Analysis time

James Tam Recording Observations How do we record user actions during observation for later analysis? -If no record is kept, evaluator may forget, miss, or mis-interpret events Paper and pencil -Primitive but cheap -Evaluators record events, interpretations, and extraneous observations -Hard to get detail (writing is slow) -Coding schemes or having a second observer may be helpful: Time working on computer person enters room answers telephone initiates telephone working on desk away from desk but in room away from room 9:00 9:02 9:05 9:10 9:13 Interruptions AbsencesDesktop activities s s s s e e e s = start of activity e = end of activity

James Tam Recording Observations (2) Audio recording -Good for recording the dialog produced by thinking aloud/constructive interaction -Hard to tie into user actions (i.e., what they are doing on the screen) Video recording -Can see and hear what a user is doing -One camera for screen, another for test user (picture in picture) -Can be intrusive during initial period of use

James Tam Continuous Evaluation 1) Developers monitor system while it’s actually being used Usually done in later stages of development -i.e., Beta releases, delivered system Good for finding real-world problems Problems can be fixed in the next release Windows is the property of Microsoft Corporation

James Tam Continuous Evaluation (2) 2) Users can provide feedback Special built-in gripe facility (web site, bulletin board) Telephone hot line Help desks Suggestion boxes Best combined with trouble-shooting facility Users always get a response (solution?) to their problem

James Tam Continuous Evaluation (3) 3) Case/field studies Careful study of “system usage” at the site Good for seeing “real life” use Can be informal or more rigorous qualitative approaches can be attempted

James Tam What You Now Know Evaluation is crucial for designing, debugging, and verifying interfaces There is a tradeoff in naturalistic vs. experimental approaches Internal and External validity The number and range of test participants employed will effect the reliability of your results Test participants must be treated with respect The study should be guided by ethical rules of behavior

James Tam What You Now Know (2)? Observing a range of users use your system for specific tasks reveals many successes and problems Qualitative observational tests are quick and easy to do Several methods reveal what is in a person’s head as they are doing the test Particular methods include Inspections Conceptual model extraction Direct observation -Simple observation -Think-aloud -Constructive interaction (Co-discovery learning) Query via interviews, retrospective testing and questionnaires Continuous evaluation via user feedback and field studies

James Tam Articulate: Who (users) What (tasks) User and task descriptions Goals: Methods: Products: Brainstorm designs Task centered system design Participatory design User- centered design Evaluate Psychology of everyday things (psych) User involvement (user) Representation & metaphors low fidelity prototyping methods Throw-away paper prototypes Participatory interaction Task scenario walk- through Refined designs Psych, User, Representations and metaphors Graphical screen design Interface guidelines Style guides high fidelity prototyping methods Testable prototypes Completed designs Alpha/beta systems or complete specification Field testing Interface Design And Usability Engineering Usability testing Heuristic evaluation This diagram is a variation of the one presented by Saul Greenberg