We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGabriella Pratt
Modified over 3 years ago
©2011 1www.id-book.com Evaluation studies: From controlled to natural settings Chapter 14
©2011 2www.id-book.com The aims: Explain how to do usability testing Outline the basics of experimental design Describe how to do field studies
©2011 3www.id-book.com Usability testing Involves recording performance of typical users doing typical tasks. Controlled settings. Users are observed and timed. Data is recorded on video & key presses are logged. The data is used to calculate performance times, and to identify & explain errors. User satisfaction is evaluated using questionnaires & interviews. Field observations may be used to provide contextual understanding.
©2011 4www.id-book.com Experiments & usability testing Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables. Usability testing is applied experimentation. Developers check that the system is usable by the intended user population for their tasks. Experiments may also be done in usability testing.
©2011 5www.id-book.com Usability testing & research Usability testing Improve products Few participants Results inform design Usually not completely replicable Conditions controlled as much as possible Procedure planned Results reported to developers Experiments for research Discover knowledge Many participants Results validated statistically Must be replicable Strongly controlled conditions Experimental design Scientific report to scientific community
©2011 6www.id-book.com Usability testing Goals & questions focus on how well users perform tasks with the product. Comparison of products or prototypes common. Focus is on time to complete task & number & type of errors. Data collected by video & interaction logging. Testing is central. User satisfaction questionnaires & interviews provide data about users opinions.
©2011 7www.id-book.com Usability lab with observers watching a user & assistant
©2011 8www.id-book.com Portable equipment for use in the field
©2011 10www.id-book.com Mobile head-mounted eye tracker Picture courtesy of SensoMotoric Instruments (SMI), copyright 2010
©2011 11www.id-book.com Testing conditions Usability lab or other controlled space. Emphasis on: –selecting representative users; –developing representative tasks. 5-10 users typically selected. Tasks usually last no more than 30 minutes. The test conditions should be the same for every participant. Informed consent form explains procedures and deals with ethical issues.
©2011 12www.id-book.com Some type of data Time to complete a task. Time to complete a task after a specified. time away from the product. Number and type of errors per task. Number of errors per unit of time. Number of navigations to online help or manuals. Number of users making a particular error. Number of users completing task successfully.
©2011 13www.id-book.com Usability engineering orientation Aim is improvement with each version. Current level of performance. Minimum acceptable level of performance. Target level of performance.
©2011 14www.id-book.com How many participants is enough for user testing? The number is a practical issue. Depends on: –schedule for testing; –availability of participants; –cost of running tests. Typically 5-10 participants. Some experts argue that testing should continue until no new insights are gained.
©2011 15www.id-book.com Name 3 features for each that can be tested by usability testing iPad
©2011 16www.id-book.com Experiments Predict the relationship between two or more variables. Independent variable is manipulated by the researcher. Dependent variable depends on the independent variable. Typical experimental designs have one or two independent variable. Validated statistically & replicable.
©2011 17www.id-book.com Experimental designs Different participants - single group of participants is allocated randomly to the experimental conditions. Same participants - all participants appear in both conditions. Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.
©2011 18www.id-book.com Different, same, matched participant design
©2011 19www.id-book.com Field studies Field studies are done in natural settings. in the wild is a term for prototypes being used freely in natural settings. Aim to understand what users do naturally and how technology impacts them. Field studies are used in product design to: - identify opportunities for new technology; - determine design requirements; - decide how best to introduce new technology; - evaluate technology in use.
©2011 20www.id-book.com Data collection & analysis Observation & interviews –Notes, pictures, recordings –Video –Logging Analyzes –Categorized –Categories can be provided by theory Grounded theory Activity theory
©2011 21www.id-book.com Data presentation The aim is to show how the products are being appropriated and integrated into their surroundings. Typical presentation forms include: vignettes, excerpts, critical incidents, patterns, and narratives.
©2011 22www.id-book.com UbiFit Garden: An in the wild study
©2011 23www.id-book.com Key points Usability testing is done in controlled conditions. Usability testing is an adapted form of experimentation. Experiments aim to test hypotheses by manipulating certain variables while keeping others constant. The experimenter controls the independent variable(s) but not the dependent variable(s). There are three types of experimental design: different- participants, same- participants, & matched participants. Field studies are done in natural environments. In the wild is a recent term for studies in which a prototype is freely used in a natural setting. Typically observation and interviews are used to collect field studies data. Data is usually presented as anecdotes, excerpts, critical incidents, patterns and narratives.
Chapter 14: Usability testing and field studies. The aims: Explain how to do usability testing through examples. Outline the basics of experimental design.
Chapter 14 Evaluation Studies: From Controlled to Natural Settings.
Computer Science Department California Polytechnic State University San Luis Obispo, CA, U.S.A. Franz J. Kurfess CPE/CSC 484: User-Centered Design and.
Chapter 14: Usability testing and field studies. 2 FJK User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept.
Chapter 14: Usability testing and field studies. Usability Testing Emphasizes the property of being usable Key Components –User Pre-Test –User Test –User.
Chapter 14: Usability testing and field studies Question 1 Right – How to observe the real effects of your fishing equipment on the fish! From
Testing & modeling users. The aims Describe how to do user testing. Discuss the differences between user testing, usability testing and research experiments.
Usability testing. Goals & questions focus on how well users perform tasks with the product. – typical users – doing typical tasks. Comparison of products.
Ch 14. Testing & modeling users Steven Pautz Lauren Sullivan Jessica Herron Chris Moore.
AVI/Psych 358/IE 340: Human Factors Evaluation October 31, 2008.
Evaluation Framework Prevention vs. Intervention CHONG POH WAN 21 JUNE 2011.
©2011 1www.id-book.com Introducing Evaluation Chapter 12 adapted by Wan C. Yoon
An evaluation framework. The aims Explain key evaluation concepts & terms. Describe the evaluation paradigms & techniques used in interaction design.
Chapter 7 Design in the World of Business. Chapter 7 Design in the marketplace Featuritis Timeline for moving from design to product Incremental vs. radical.
1 Copyright © 2010, Elsevier Inc. All rights Reserved Fig 2.1 Chapter 2.
Chapter 11: An Evaluation Framework Group 4: Tony Masi, Sam Esswein, Brian Rood, & Chris Troisi.
©2011 1www.id-book.com An evaluation framework Chapter 13.
User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005.
CS305: HCI in SW Development Evaluation (Return to…) We’ve had an introduction to evaluation. Now for more details on…
©2011 1www.id-book.com Introducing Evaluation Chapter 12.
25 seconds left….. 24 seconds left….. 23 seconds left…..
WEEK 1 You have 10 seconds to name…
We will resume in: 25 Minutes We will resume in: 24 Minutes.
Addition 1’s to
1 Unit 1 Kinematics Chapter 1 Day
Um ambiente para avaliação. Objetivos Explicar conceitos e termos da avaliação Descrever paradigmas de avaliação e técnicas utilizadas no design de.
Business Transaction Management Software for Application Coordination 1 Business Processes and Coordination.
Addition Facts = = =
Usability Evaluation. ©2011 The aims Explain the key concepts used in evaluation. Introduce different evaluation methods. Show.
Jeopardy Topic 1Topic Q 1Q 6Q 11Q 16Q 21 Q 2Q 7Q 12Q 17Q 22 Q 3Q 8Q 13Q 18Q 23 Q 4Q 9Q 14Q 19Q 24 Q 5Q 10Q 15Q 20Q 25 Final Jeopardy.
1 Science as a Process Chapter 1 Section 2. 2 Objectives Explain how science is different from other forms of human endeavor. Identify the steps that.
Copyright © 2011, Elsevier Inc. All rights reserved. Chapter 5 Author: Julia Richards and R. Scott Hawley.
Chapter 13 How Cells Obtain Energy from Food Essential Cell Biology Third Edition Copyright © Garland Science 2010.
DIVIDING INTEGERS 1. IF THE SIGNS ARE THE SAME THE ANSWER IS POSITIVE 2. IF THE SIGNS ARE DIFFERENT THE ANSWER IS NEGATIVE.
+ Controlled User studies HCI /6610 Winter 2013.
Chapter 2 The Process of Experimentation 2.1 Designing and Conducting Agricultural Research.
TWO STEP EQUATIONS 1. SOLVE FOR X 3. DIVIDE BY THE NUMBER IN FRONT OF THE VARIABLE 2. DO THE ADDITION STEP FIRST.
Level 2 Prepared by: RHR First Prepared on: Nov 23, 2006 Last Modified on: Quality checked by: MOH Copyright 2004 Asia Pacific Institute of Information.
Chapter 5 Test Review Sections 5-1 through 5-4. Simplify each expression. 1)2) 3)4) 5) 6)
User Testing & Experiments. Objectives Explain the process of running a user testing or experiment session. Describe evaluation scripts and pilot tests.
1 Human Computer Interaction Week 8 Evaluation. 2 Introduction Evaluation is concerned with gathering data about the usability of a design or product.
BT Wholesale October Creating your own telephone network WHOLESALE CALLS LINE ASSOCIATED.
HOLT, RINEHART AND WINSTON P SYCHOLOGY PRINCIPLES IN PRACTICE 1 Chapter 2 PSYCHOLOGICAL METHODS Section 1: Conducting ResearchConducting Research Section.
Copyright © 2010 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin Statistical Inferences Based on Two Samples Chapter 10.
© 2009 South-Western, Cengage LearningMARKETING 1 Chapter 5 MARKETING INFORMATION AND RESEARCH 5-1Understanding the Need for Market Information 5-2Finding.
© 2017 SlidePlayer.com Inc. All rights reserved.