AVI/Psych 358/IE 340: Human Factors Evaluation October 31, 2008

2 Why, what, where and when to evaluate Iterative design & evaluation is a continuous process that examines: –Why: to check that users can use the product and that they like it. –What: a conceptual model, early prototypes of a new system and later, more complete prototypes. –Where: in natural and laboratory settings. –When: throughout design; finished products can be evaluated to collect information to inform new products.

3 Evaluation approaches Usability testing Field studies Analytical evaluation (e.g., think aloud)

4 Characteristics of approaches Usability testing Field studies Analytical Usersdo tasknaturalnot involved Locationcontrollednaturalanywhere Whenprototypeearlyprototype Dataquantitativequalitativeproblems Feed backmeasures & errors descriptionsproblems Typeappliednaturalisticexpert

5 Evaluation approaches and methods MethodUsability testing Field studies Analytical Observing x x Asking users x x Asking experts x x Testing x Modeling x

6 Usability testing of a website MedlinePlus: Provides health information for public, doctors, and other medical professionals Created by National Library of Medicine (NLM)

7 Goals and questions To identify the range of usability problems –Navigation –Information seeking –Etc.

8 Selection of participants Nine subjects from primary health care practices through recruitment posters in Washington, DC Subjects completed brief questionnaire (age, web experience, frequency of seeking health-related information) 7 were women, 2 were men Gender balance?

9 Development of tasks 1.Find information about whether a dark bump on your shoulder might be skin cancer 2.Find information about whether it is safe to use Prozac during pregnancy 3.Find information about whether there is a vaccine for hepatitis C 4.Find recommendations about the treatment of breast cancer, specifically the use of mastectomies 5.Find information about the dangers associated with drinking alcohol during pregnancy

10 Procedure (1/2) Five scripts were prepared for the user test –Introduction to the study –Consent form –List of tasks –Post-experiment questionnaire

11 Procedure (2/2) Same instructions were given to every subject Testing done in laboratory-like conditions –Subjects sat in front of a monitor alone –Were allowed to explore website for 10 minutes prior to tasks –Think aloud protocol used, 20 minutes for each of the five tasks –If task was not finished, they were asked to stop (ensuring equal time for each task)

12 Data collection Video Actions on website were logged Start and completion time Page counts Medical publications accessed Negative comments User satisfaction questionnaire

13 Data analysis Website organization Browsing efficiency Use of search features …

14 Results Reaching external sites was often difficult Users experienced difficult finding health topics devoted to different types of cancer High ease of learning but low ease of use

15 Usability testing (1/2) Involves recording performance of typical users doing typical tasks Controlled environmental settings Users are observed and timed

16 Usability testing (2/2) The data is used to calculate performance times, and to identify & explain errors User satisfaction is evaluated using questionnaires & interviews Field observations may be used to provide contextual understanding

17 Running Experiments Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables.

18 Usability lab with observers watching a user & assistant

19 Portable equipment for use in the field

20

21 Testing conditions Usability lab or other controlled space. Emphasis on: –selecting representative users; –developing representative tasks users typically selected. Tasks usually last no more than 30 minutes. The test conditions should be the same for every participant. Informed consent form explains procedures and deals with ethical issues.

22 Some type of data  Time to complete a task.  Time to complete a task after a specified. time away from the product.  Number and type of errors per task.  Number of errors per unit of time.  Number of navigations to online help or manuals.  Number of users making a particular error.  Number of users completing task successfully.

23 How many participants are enough for user testing? The number is a practical issue. Depends on: –schedule for testing; –availability of participants; –cost of running tests. Typically 5-10 participants. Some experts argue that testing should continue until no new insights are gained.