Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011.

Slides:



Advertisements
Similar presentations
©2011 1www.id-book.com Evaluation studies: From controlled to natural settings Chapter 14.
Advertisements

©2011 1www.id-book.com An evaluation framework Chapter 13.
Chapter 13: An evaluation framework
Chapter 14: Usability testing and field studies
Chapter 5 Development and Evolution of User Interface
Imran Hussain University of Management and Technology (UMT)
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
CS305: HCI in SW Development Evaluation (Return to…)
WHAT IS INTERACTION DESIGN?
Chapter 14: Usability testing and field studies. 2 FJK User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept.
Observing Users Paul Bogen, LaShon Johnson, Jehoon Park.
Data gathering.
Chapter 14: Usability testing and field studies. Usability Testing Emphasizes the property of being usable Key Components –User Pre-Test –User Test –User.
An evaluation framework
User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005.
Observing users.
Data-collection techniques. Contents Types of data Observations Event logs Questionnaires Interview.
An evaluation framework
Usability 2004 J T Burns1 Usability & Usability Engineering.
An evaluation framework
Chapter 3 Preparing and Evaluating a Research Plan Gay and Airasian
Evaluation How do we test the interaction design? Several Dimensions
From Controlled to Natural Settings
Formulating the research design
Design in the World of Business
Chapter 14: Usability testing and field studies
Computer Science Department California Polytechnic State University San Luis Obispo, CA, U.S.A. Franz J. Kurfess CPE/CSC 484: User-Centered Design and.
Chapter 11: An Evaluation Framework Group 4: Tony Masi, Sam Esswein, Brian Rood, & Chris Troisi.
Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.
Ch 14. Testing & modeling users
Part 1-Intro; Part 2- Req; Part 3- Design  Chapter 20 Why evaluate the usability of user interface designs?  Chapter 21 Deciding on what you need to.
Human Computer Interaction
Usability testing. Goals & questions focus on how well users perform tasks with the product. – typical users – doing typical tasks. Comparison of products.
Usability Evaluation June 8, Why do we need to do usability evaluation?
Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal.
Chapter 12/13: Evaluation/Decide Framework Question 1.
Testing & modeling users. The aims Describe how to do user testing. Discuss the differences between user testing, usability testing and research experiments.
Level 2 Prepared by: RHR First Prepared on: Nov 23, 2006 Last Modified on: Quality checked by: MOH Copyright 2004 Asia Pacific Institute of Information.
Chapter 13. Reviewing, Evaluating, and Testing © 2010 by Bedford/St. Martin's1 Usability relates to five factors of use: ease of learning efficiency of.
COMP5047 Pervasive Computing: 2012 Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research.
CS3205: HCI in SW Development Evaluation (Return to…) We’ve had an introduction to evaluation. Now for more details on…
© An Evaluation Framework Chapter 13.
Chapter 12/13: Evaluation/Decide Framework. Why Evaluate? Why: to check that users can use the product and that they like it. Designers need to check.
Computer Science Department California Polytechnic State University San Luis Obispo, CA, U.S.A. Franz J. Kurfess CPE/CSC 484: User-Centered Design and.
Observing users. The aims Discuss the benefits & challenges of different types of observation. Describe how to observe as an on-looker, a participant,
Facilitate Group Learning
AVI/Psych 358/IE 340: Human Factors Data Gathering October 3, 2008.
AVI/Psych 358/IE 340: Human Factors Evaluation October 31, 2008.
DECIDE: An evaluation framework. DECIDE: a framework to guide evaluation D D etermine the goals. E E xplore the questions. C C hoose the evaluation approach.
Um ambiente para avaliação. Objetivos Explicar conceitos e termos da avaliação Descrever paradigmas de avaliação e técnicas utilizadas no design de.
Oct 211 The next two weeks Oct 21 & 23: Lectures on user interface evaluation Oct 28: Lecture by Dr. Maurice Masliah No office hours (out of town) Oct.
Data gathering (Chapter 7 Interaction Design Text)
Chapter 13: An evaluation framework. The aims are: To discuss the conceptual, practical and ethical issues involved in evaluation. To introduce and explain.
SIE 515 Design Evaluation Lecture 7.
Lecture3 Data Gathering 1.
ELT 329 ACTION RESEARCH Week 4
CSC 341 Human-Computer Interaction
Introducing Evaluation
From Controlled to Natural Settings
WHAT IS INTERACTION DESIGN?
Evaluation Paradigms & Techniques
From Controlled to Natural Settings
Evaluation.
HCI Evaluation Techniques
Testing & modeling users
Human-Computer Interaction: Overview of User Studies
COMP444 Human Computer Interaction Evaluation
From Controlled to Natural Settings
Presentation transcript:

Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011

The aims Describe the evaluation paradigms & techniques used in interaction design. Discuss the conceptual, practical and ethical issues that must be considered when planning evaluations. Introduce the DECIDE framework.

Four evaluation paradigms ‘quick and dirty’ usability testing field studies predictive evaluation

Quick and dirty ‘quick & dirty’ evaluation describes the common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in-line with users’ needs and are liked. Quick & dirty evaluations are done any time. The emphasis is on fast input to the design process rather than carefully documented findings.

Usability testing Usability testing involves recording typical users’ performance on typical tasks in controlled settings. Field observations may also be used. As the users perform these tasks they are watched & recorded on video & their key presses are logged. This data is used to calculate performance times, identify errors & help explain why the users did what they did. User satisfaction questionnaires & interviews are used to elicit users’ opinions.

Field studies Field studies are done in natural settings The aim is to understand what users do naturally and how technology impacts them. In product design field studies can be used to: - identify opportunities for new technology - determine design requirements - decide how best to introduce new technology - evaluate technology in use.

Predictive evaluation Experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems. Another approach involves theoretically based models. A key feature of predictive evaluation is that users need not be present Relatively quick & inexpensive

Characteristics of different evaluation paradigms Table 11.1, pg.344 The role of users Who controls the process and the relationship between evaluators and users during the evaluation The location of the evaluation When the evaluation is most useful The type of data collected and how it is analyzed How the evaluation findings are fed back into the design process The philosophy and theory that underlies the evaluation paradigms

Overview of techniques observing users, asking users’ their opinions, asking experts’ their opinions, testing users’ performance modeling users’ task performance

DECIDE: A framework to guide evaluation Determine the goals the evaluation addresses. Explore the specific questions to be answered. Choose the evaluation paradigm and techniques to answer the questions. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, interpret and present the data.

Determine the goals What are the high-level goals of the evaluation? The goals influence the paradigm for the study Some examples of goals: Identify the best metaphor on which to base the design. Check to ensure that the final interface is consistent. Investigate how technology affects working practices. Improve the usability of an existing product.

Explore the questions All evaluations need goals & questions to guide them so time is not wasted on ill-defined studies. For example, the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub- questions: - What are customers’ attitudes to these new tickets? - Are they concerned about security? - Is the interface for obtaining them poor?

Example Goals To identify usability problems in the revised interface. More specific questions The evaluators wanted to know if the revised way of categorizing information, suggested by the expert evaluators, worked. They also wanted to check that users could navigate the system to find the information they needed. Navigating around large websites can be a major usability problem, so it is important to check that the design of MEDLINEplus supported users’ navigation strategies.

Choose the evaluation paradigm & techniques The evaluation paradigm strongly influences the techniques used, how data is analyzed and presented. E.g. field studies do not involve testing or modeling

Identify practical issues For example, how to: select users Select facilities and equipment Stay on budget and schedule find evaluators (expertise)

Decide on ethical issues Develop an informed consent form Participants have a right to: - know the goals of the study - what will happen to the findings - privacy of personal information - not to be quoted without their agreement - leave when they wish - be treated politely

Ethical Issues 1.Usability laboratories often have a one-way mirror that allows evaluators to watch users doing their task in the laboratories without the users seeing the evaluation. Should users be told that they are being watched? 2. Studies of users behavior on the internet may involve logging users` interaction and keeping a copy of their conversation with others.Should users be told that this is happening?

Evaluate, interpret & present data How data is analyzed & presented depends on the paradigm and techniques used.

Pilot studies A small trial run of the main study. The aim is to make sure your plan is viable. Pilot studies check: - that you can conduct the procedure - that interview scripts, questionnaires, experiments, etc. work appropriately It’s worth doing several to iron out problems before doing the main study. Ask colleagues if you can’t spare real users.

Key points - Review An evaluation paradigm is an approach that is influenced by particular theories and philosophies. Five categories of techniques were identified: observing users, asking users, asking experts, user testing, modeling users. The DECIDE framework has six parts: - Determine the overall goals - Explore the questions that satisfy the goals - Choose the paradigm and techniques - Identify the practical issues - Decide on the ethical issues - Evaluate ways to analyze & present data Do a pilot study END