1. Chapter 14: Usability testing and field studies

2. 2 FJK 2005-2011 User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo

3. 3 FJK 2005-2011 Copyright Notice These slides are a revised version of the originals provided with the book “Interaction Design” by Jennifer Preece, Yvonne Rogers, and Helen Sharp, Wiley, 2002. I added some material, made some minor modifications, and created a custom show to select a subset. Slides added or modified by me are marked with my initials (FJK), unless I forgot it …

4. 4 FJK 2005-2011 484-W09 Quarter The slides I use in class are in the Custom Show “484-W09”. It is a subset of the whole collection in this file. Week 8 contains slides from Chapter 14 of the textbook.

5. 5 FJK 2005-2011 484-W10 Quarter The set of slides I use in class is close to the one in the PowerPoint Custom Show “484-W09”. Since I’m using mostly Keynote now, I use the “Skip” feature to achieve a similar result.

6. 6 FJK 2005-2011 Chapter Overview user testing DECIDE framework for user testing experiments –variables and conditions –data collection and analysis predictive models –GOMs –keystroke level model

7. 7 FJK 2005-2011 Motivation in addition to asking users about their experiences or tasks, it may be necessary to conduct tests where they perform tasks ideally, these tests should be done with the final product –in practice, this is often not feasible –user testing can also be done on prototypes to find out how well the general approach works user modeling tries to predict user performance for tasks performed on a system

8. 8 FJK 2005-2011 Objectives become familiar with the main principles and constraints of user testing be aware of the differences between user testing, usability testing and research experiments be able to design simple experiments become familiar with the GOMS user model, the keystroke level model, and Fitts’ law determine when these techniques can be applied know how to do a keystroke level analysis

9. 9 The aims: Explain how to do usability testing through examples. Outline the basics of experimental design. Discuss the methods used in usability testing. Discuss the role of field studies in evaluation.

10. 10 Usability testing Involves recording performance of typical users doing typical tasks. Controlled environmental settings. Users are observed and timed. Data is recorded on video & key presses are logged. The data is used to calculate performance times, and to identify & explain errors. User satisfaction is evaluated using questionnaires & interviews. Field observations may be used to provide contextual understanding.

11. 11 Experiments & usability testing Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables. Usability testing is applied experimentation. Developers check that the system is usable by the intended user population for their tasks. Experiments may also be done in usability testing.

12. 12 Usability testing & research Usability testing Improve products Few participants Results inform design Usually not completely replicable Conditions controlled as much as possible Procedure planned Results reported to developers Experiments for research Discover knowledge Many participants Results validated statistically Must be replicable Strongly controlled conditions Experimental design Scientific reported to scientific community

13. 13 Usability testing Goals & questions focus on how well users perform tasks with the product. Comparison of products or prototypes common. Focus is on time to complete task & number & type of errors. Data collected by video & interaction logging. Testing is central. User satisfaction questionnaires & interviews provide data about users’ opinions.

14. 14 Usability lab with observers watching a user & assistant

15. 15 Portable equipment for use in the field

16. 16

17. 17 Testing conditions Usability lab or other controlled space. Emphasis on: –selecting representative users; –developing representative tasks. 5-10 users typically selected. Tasks usually last no more than 30 minutes. The test conditions should be the same for every participant. Informed consent form explains procedures and deals with ethical issues.

18. 18 Some type of data Time to complete a task. Time to complete a task after a specified. time away from the product. Number and type of errors per task. Number of errors per unit of time. Number of navigations to online help or manuals. Number of users making a particular error. Number of users completing task successfully.

19. 19 Usability engineering orientation Aim is improvement with each version. Current level of performance. Minimum acceptable level of performance. Target level of performance.

20. 20 How many participants is enough for user testing? The number is a practical issue. Depends on: –schedule for testing; –availability of participants; –cost of running tests. Typically 5-10 participants. Some experts argue that testing should continue until no new insights are gained.

21. 21 MedlinePlus.com Explore Medlineplus. Develop 2 tasks that are not described in the book. Decide what data you will collect. Run a test with two participants. Collect, analyze, present & comment on your data.

22. 22 FJK 2005-2011 User Testing with DECIDE Determine the goals and Explore the questions Choose the paradigm and techniques Identify practical issues Deal with ethical issues Evaluate, analyze, and present data

23. 23 Experiments Predict the relationship between two or more variables. Independent variable is manipulated by the researcher. Dependent variable depends on the independent variable. Typical experimental designs have one or two independent variable.

24. 24 Experimental designs Different participants - single group of participants is allocated randomly to the experimental conditions. Same participants - all participants appear in both conditions. Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.

25. 25 Different, same, matched participant design

26. 26 Field studies Field studies are done in natural settings. The aim is to understand what users do naturally and how technology impacts them. Field studies can be used in product design to: - identify opportunities for new technology; - determine design requirements; - decide how best to introduce new technology; - evaluate technology in use.

27. 27 Data collection & analysis Observation & interviews –Notes, pictures, recordings –Video –Logging Analyzes –Categorized –Categories can be provided by theory Grounded theory Activity theory

28. 28

29. 29 Example of categories from Activity Theory

30. 30 Key points Testing is a central part of usability testing. Usability testing is done in controlled conditions. Usability testing is an adapted form of experimentation. Experiments aim to test hypotheses by manipulating certain variables while keeping others constant. The experimenter controls the independent variable(s) but not the dependent variable(s). There are three types of experimental design: different- participants, same- participants, & matched participants. Field studies are done in natural environments. Typically observation and interviews are used to collect field studies data. Categorization and theory-based techniques are used to analyze the data.