1. Formal User Studies Marti Hearst (UCB SIMS) SIMS 213, UI Design & Development April 13, 1999

2. Outline l Experiment Design –Factoring Variables –Interactions l Special considerations when involving human participants l Example: Marking Menus –Motivation –Hypotheses –Design –Analysis

3. Formal Usability Studies l Situations in which these are useful –to determine time requirements for task completion –to compare two designs on measurable aspects »time required »number of errors »effectiveness for achieving very specific tasks l Require Experiment Design

4. Experiment Design l Experiment design involves determining how many experiments to run and which attributes to vary in each experiment l Goal: isolate which aspects of the interface really make a difference

5. Experiment Design l Decide on –Response variables »the outcome of the experiment »usually the system performance »aka dependent variable(s) –Factors (aka attributes) »aka independent variables –Levels (aka values for attributes) –Replication »how often to repeat each combination of choices

6. Experiment Design l Studying a system; ignoring users l Say we want to determine how to configure the hardware for a personal workstation Hardware choices »which CPU (three types) »how much memory (four amounts) »how many disk drives (from 1 to 3) –Workload characteristics »administration, management, scientific

7. Experiment Design l We want to isolate the effect of each component for the given workload type. l How do we do this? –WL1CPU1 Mem1Disk1 –WL1CPU1Mem1Disk2 –WL1CPU1Mem1Disk3 –WL1CPU1Mem2Disk1 –WL1CPU1Mem2Disk2 –…–… l There are (3 CPUs)*(4 memory sizes)*(3 disk sizes)*(3 workload types) = 108 combinations!

8. Experiment Design l One strategy to reduce the number of comparisons needed: –pick just one attribute –vary it –hold the rest constant l Problems: –inefficient –might miss effects of interactions

9. Interactions among Attributes A1A2 B135 B268 A1A2 B135 B269 A1 B1 A2 A1 B2 A2 B2 Non-interactingInteracting A2 A1 B1B2B1B2

10. Experiment Design l Another strategy: figure out which attributes are important first l Do this by just comparing a few major attributes at a time –if an attribute has a strong effect, include it in future studies –otherwise assume it is safe to drop it l This strategy also allows you to find interactions between attributes

11. Experiment Design l Common practice: Fractional Factorial Design –Just compare important subsets –Use experiment design to partially vary the combinations of attributes l Blocking –Group factors or levels together –Use a Latin Square design to arrange the blocks

12. Adapted from slide by James Landay Between Groups vs. Within Groups l Do participants see only one design or both? l Between groups –two groups of test users –each group uses only 1 of the systems l Within groups experiment –one group of test users »each person uses both systems »can’t use the same tasks (learning) –Why is this a consideration? –People often learn during the experiment.

13. Special Considerations for Formal Studies with Human Participants l Studies involving human participants vs. measuring automated systems –people get tired –people get bored –people (may) get upset by some tasks –learning effects »people will learn how to do the tasks (or the answers to questions) if repeated »people will (usually) learn how to use the system over time

14. More Special Considerations l High variability among people –especially when involved in reading/comprehension tasks –especially when following hyperlinks! (can go all over the place)

15. Experiment Design Example: Marking Menus Based on Kurtenbach, Sellen, and Buxton, Some Articulartory and Cognitive Aspects of “Marking Menus”, Graphics Interface ‘94, http://reality.sgi.com/gordo_tor/papers

16. Experiment Design Example: Marking Menus l Pie marking menus can reveal –the available options –the relationship between mark and command l 1. User presses down with stylus l 2. Menu appears l 3. User marks the choice, an ink trail follows

17. Why Marking Menus? l Supporting markings with pie menus should help transition between novice and expert l Useful for keyboardless devices l Useful for large screens l Pie menus have been shown to be faster than linear menus in certain situations

18. What do we want to know? l Are marking menus better than pie menus? –Do users have to see the menu? –Does leaving an “ink trail” make a difference? –Do people improve on these new menus as they practice? l Related questions: –What, if any, are the effects of different input devices? –What, if any, are the effects of different size menus?

19. Experiment Factors l Isolate the following factors (independent variables): –Menu condition »exposed, hidden, hidden w/marks (E,H,M) –Input device »mouse, stylus, track ball (M,S,T) –Number of items in menu »4,5,7,8,11,12 (note: both odd and even) l Response variables (dependent variables): –Response Time –Number of Errors

20. Experiment Hypotheses l Note these are stated in terms of the factors (independent variables) –Exposed menus will yield faster response times and lower error rates, but not when menu size is small –Response variables will monotonically increase with menu size for exposed menus –Response time will be sensitive to number of menu choices for hidden menus (familiar ones will be easier, e.g., 8 and 12) –Stylus better than Mouse better than Track ball

21. Experiment Hypotheses –Device performance independent of menu type –Performance on hidden menus (both marking and hidden) will improve steadily across trials. Performance on exposed menus will remain constant.

22. Experiment Design l Participants –36 right-handed people »usually gender distribution is stated –considerable mouse experience –(almost) no trackball, stylus experience l Task –Select target “slices” from a series of different pie menus as quickly and accurately as possible –Menus were simply numbered segments »meaningful items would have longer learning times –Participants saw running scores »lose points for wrong selection

23. Experiment Design l One between-subjects factor –Menu Type »Three levels: E, H, or M l Two within-subjects factors –Device Type »Three levels: M, T, or S –Number of Menu Items »Six levels: 4, 5, 7, 8, 11, 12 l How should we arrange these?

24. Experiment Design EHM 12 Between subjects design How to arrange the devices?

25. Experiment Design M T S T S M S M T EHM 12 A Latin Square No row or column share labels

26. Experiment Design M T S T S M S M T EHM How to arrange the menu sizes? Block by size then randomize the blocks.

27. Experiment Design M T S T S M S M T EHM 511 128 74 Block by size then randomize the blocks.

28. Experiment Design M T S T S M S M T EHM 511 128 74 78 5 411 40 trials per block

29. Experiment Overall Results So exposing menus is faster … or is it? Let’s factor things out more.

30. A Learning Effect When we graph over the number of trials, we find a difference between exposed and hidden menus. This suggests that participants may eventually become faster using marking menus. (hypothesized) A later study verified this.

31. Factoring to Expose Interactions l Increasing menu size increases selection time and number of errors (hypothesized) l No differences across menu groups in terms of response time. l That is, until we factor by menu size AND group –Then we see that menu size has effects on hidden groups not seen on exposed group –This was hypothesized (12 easier than 11)

32. Factoring to Expose Interactions l Stylus and mouse outperformed trackball (hypothesized) l Stylus and mouse the same (not hypothesized) l Initially, effect of input device did not interact with menu type –this is when comparing globally –BUT... l More detailed analysis: –Compare both by menu type and device type –Stylus significantly faster with Marking group –Trackball significantly slower with Exposed group –Not hypothesized!

33. Average response time and errors as a function of device, menu size, and menu type Potential explanations: Markings provide feedback for when stylus is pressed properly. Ink trail is consistent with the metaphor of using a pen.

34. Experiment Design M T S T S M S M T EHM How can we tell if order in which the device appears has an effect on the final outcome? Some evidence: There is no significant difference among devices in the Hidden group. Trackball was slowest and most error prone in all three cases. Still, there may be some hidden interactions, but unlikely to be strong given the previous graph.

35. Statistical Tests l Need to test for statistical significance –This is a big area –Assuming a normal distribution: »Students t-test to compare two variables »ANOVA to compare more than two variables

36. Adapted from slide by James Landay Analyzing the Numbers l Example: trying to get task time <=30 min. –test gives: 20, 15, 40, 90, 10, 5 –mean (average) = 30 –median (middle) = 17.5 –looks good! –wrong answer, not certain of anything l Factors contributing to our uncertainty –small number of test users (n = 6) –results are very variable (standard deviation = 32) »std. dev. measures dispersal from the mean

37. Adapted from slide by James Landay Analyzing the Numbers (cont.) l This is what statistics is for l Crank through the procedures and you find –95% certain that typical value is between 5 & 55 l Usability test data is quite variable –need lots to get good estimates of typical values –4 times as many tests will only narrow range by 2x

38. Followup Work l Hierarchical Markup Menu study

39. Followup Work l Results of use of marking menus over an extended period of time –two person extended study –participants became much faster using gestures without viewing the menus

40. Followup Work l Results of use of marking menus over an extended period of time –participants temporarily returned to “novice” mode when they had been away from the system for a while

41. Summary l Formal studies can reveal detailed information but but take extensive time/effort l Human participants entail special requirements l Experiment design involves –Factors, levels, participants, tasks, hypotheses –Important to consider which factors are likely to have real effects on the results, and isolate these l Analysis –Often need to involve a statistician to do it right –Need to determine statistical significance –Important to make plots and explore the data

42. References l Kurtenbach, Sellen, and Buxton, Some Articulartory and Cognitive Aspects of “Marking Menus”, Graphics Interface ‘94, http://reality.sgi.com/gordo_tor/papers l Kurtenbach and Buxton, User Learning and Performance with Marking Menus, Graphics Interface ‘94, http://reality.sgi.com/gordo_tor/papers l Jain, The art of computer systems performance analysis, Wiley, 1991 l http://www.statsoft.com/textbook/stanman.html l Gonick and Smith, The Cartoon Guide to Statistics, HarperPerennial, 1993 l Dix et al. textbook