1. The Effects of Interface Design on Telephone Dialing Performance Masters thesis in Computer Science Andrew R. Freed 4/30/2003

2. The Effects of Interface Design on Telephone Dialing Performance n Towards automatic interface evaluation n Methods of evaluation n Experiment design n Three analyses n Comparison of analyses n Further work

3. Towards automatic interface evaluation n Why not test with actual users instead? n It takes too much time and money! n Automatic evaluation has been useful in the past (Project Ernestine - Gray et al 1992) to the tune of $2.4M savings/year n Several proposed tools will make this type of evaluation easier

4. Towards automatic interface evaluation n Motivation: –Eye-tracking studies by Byrne (1999, 2001) and Hornof (1997) –Cognitive models as surrogate users (Ritter 2001)

5. Towards automatic interface evaluation n 100 phones to choose from n Selected 10 for analysis

6. Towards automatic interface evaluation n 10 tasks (Ritter 2000) – 1. Call home (*) – 2. Call work (*) – 3. Redial last number (*) – 4. Call directory inquiries – 5. Call mother (*) – 6. Conference call work and home (*) – 7. Conference call work (flash) then home – 8. Forward call to another number (*) – 9. Forward call (flash) to another number –10. Hang up

7. Towards automatic interface evaluation n 10 telephone numbers –814-866-5000215-654-5785 –123-654-7890814-234-9657 –814-863-5000740-611-9273 –412-268-3000101-010-1010 –606-193-3012103-273-1029 n and 3 other tasks –Forward, redial, conference call

8. Methods of evaluation n Possible tools n Cognitive architectures n ACT-R/PM n Generic Simulated Eyes and Hands n Focused analysis methods

9. Possible tools n Ivorys tools to evaluate websites (2001) n Apex (M. Freed 1998) and iGen (Emmerson 2000) model complex tasks n Glean (Kieras et al 1995) evaluates Lisp interfaces n Shortcomings: no learning, no visual search, tied to a specific interface format, no cognitive theory

10. Cognitive architectures n Unified theory of cognition (Newell 1990) n Simulate human behavior n Perceptual and motor capability (simulated eyes and hands) n Can do visual search, click buttons, sometimes learn

11. Cognitive architectures (examples) n EPIC (Kieras and Meyer 1997) - has visual search and perceptual/motor skills… but only evaluates Common Lisp interfaces n Soar (Newell 1990) - also has visual search, perceptual motor skills, plus learning… but only evaluates Tcl/Tk interfaces (or requires a socket connection) n ACT-R/PM (Anderson & Lebiere 1998, Byrne 2001) - nearly identical benefits and limitations as EPIC, plus has learning

12. ACT-R/PM n Why did we choose ACT-R/PM? n Well-accepted cognitive architecture n Used in past to evaluate interfaces n Can overcome the Lisp interface-only problem with generic eyes and hands

13. Generic Simulated Eyes and Hands n Segman (St. Amant & Riedl 2001) can parse a Windows screen capture and determine the interface components n Can use interfaces written in Lisp, Tcl/Tk, HTML, Visual C++,... n Segman can be connected to ACT- R/PM

14. Focus of analysis n A - Analytical model (Fitts Law) n B - Cognitive model (ACT-R/PM) n C - Human data

15. General experiment design n Analytical model, cognitive model, and human users interact with same interfaces n Analytical model dials each number once on each phone, does not do other tasks n Cognitive model: Dialed each phone number 50 times on each phone, performed other phone tasks 50 times on each phone. n Human users (N=9): Dialed each phone number on each phone, performed other phone tasks once on each phone

16. n Experimental software General experiment design

17. n Cognitive model and users –Timing and mouse-click logging –Eye-tracking –Users can control pace of trials, model does not care n Analytical model –Does not need to see telephones –Mathematical formula with pixel-level input yields reaction times

18. A. Fitts Law analysis n What is Fitts Law? n Numerical analysis n Simple conclusions and problems

19. What is Fitts Law? n Fitts Law (two possible forms): –MT = a + b * LOG 2 (2 * D/W) (Fitts 1954) –MT = max(t m, k * LOG 2 [0.5 + D/W]) (Card et al, 1983) n MT is mouse movement time n D is distance to target, W is target width n a, b, k are constants n t m is minimum movement time

20. Numerical analysis n Collected pixel-level input about telephones (size and location of buttons) n Dialing a phone requires 10 movements n Total the times from the 10 movements and a base dialing time is established (with no visual search!)

21. Numerical analysis n Validating our choice of sample telephone numbers (R 2 = 0.96)

22. Simple conclusions and problems n Fitts Law analysis is fast (it is just an equation!) n Does not consider many factors n Not affected by any aspect of interface design other than button sizing and spacing

23. B. ACT-R/PM model analysis n Description of model n Visual search predictions n ACT-R/PM makes different reaction time conclusions

24. Description of ACT-R/PM model n Model has three main components that can operate in parallel: –retrieve a phone digit from memory –visually search for the digit –move the mouse/click on a digit (governed by Fitts Law) n Composed of 71 production rules (mostly for visual search)

25. Description of ACT-R/PM model n Visual search strategy: random or systematic n One production for random search n Find-random-target IFthe goal is to find a phone target THENfind a visual object of type text which has not been attended lately

26. Description of ACT-R/PM model n Sixty productions for systematic search n Systematic-search-from-target IFa digit x is in the visual buffer ANDthe goal is to find a target y ANDy is in direction z from x THENfind a visual object of type text in direction z from target x which is within the bounds of the keypad

27. Visual search predictions n Count fixations and note fixation locations n Search for the keypad is random n Search within the keypad is systematic n The telephones do not generally require a statistically significant different number of fixations to dial (about 16) n (The telephone numbers are significantly different)

28. Visual search predictions n Model trace

29. Visual search predictions Phone 4 Phone 9 Whats wrong with this picture?

30. Visual search predictions n Two phones are predicted to have abnormally long visual searches n These phones require approximately sixty fixations (average on others was sixteen) n Phone 4 has an upside-down keypad -- the systematic search fails! n Phone 9 contains extra information on the buttons… distracts the visual search n We will see the model takes much longer than humans to dial these phones

31. ACT-R/PM makes different reaction time conclusions n This is no surprise - more factors are being considered n Phones 4 and 9 pay a large visual search penalty n Fitts Law still a factor - phones with Fitts Law violations still perform worse

32. ACT-R/PM makes different reaction time conclusions

33. n The phones are often shown to have different dialing times (T-test, p<.05) n The significance level of the differences depends on the telephone number being dialed n On average, approximately 8.7 seconds to dial a telephone. n Never faster than six seconds n No errors!

34. ACT-R/PM makes different reaction time conclusions n Model is able to perform additional tasks (redial, forward, conference) with a random search n Model does not always succeed but never gives up n Will attend the same visual target several times

35. C. User data analysis n Where and how users look (eye- tracking) n Humans make errors n Summary of user reaction times

36. Where and how users look n Fast random search for keypad n Systematic search within keypad

37. Where and how users look n User trace

38. Where and how users look n Users require approximately the same number of fixations per telephone as the model did (also true for telephone numbers) n User able to cope with phones 4 and 9 by changing search strategy –Phone 4: Up is down, down is up –Phone 9: Ignore ABCs on the keypad

39. Where and how users look n Fixation comparison across numbers (R 2 = 0.11)

40. Where and how users look n Fixation comparison across 8 phones (R 2 = 0.34)

41. Humans make errors n Errors not predicted by the automatic analyses n Depend on several factors –Number being dialed –Dialing speed (weak correlation) –Interface being used

42. Errors dependent on interface n Most errors on Fitts Law violators n Least errors when large and adjacent buttons n Users will move mouse while clicking (ACT-R/PM will not), this can cause errors n Possible to estimate number of errors with Fitts index of difficulty?

43. Summary of reaction times n User on average more than one second faster than model n This probably due to efficient pipelining of motor tasks (room for ACT-R/PM improvement) n Users can dial as fast as 3.5 seconds (average is seven seconds)

44. Summary of reaction times n Model (R 2 = 0.41), Fitts (R 2 = 0.85), user dial time across phones

45. Summary of reaction times n Users can do other phone tasks faster than ACT-R/PM n Users can find the target under varied conditions n Users try more strategies to find target n Users will give up if they cant succeed!

46. Summary of reaction times n Model vs user on extra tasks (R 2 = 0.60, 0.26, 0.11)

47. Summary of reaction times n User data also shows that the interfaces are often significantly different (p <.05), though less often than the model says n User time differences also depend on the number being dialed n Theory: users less affected by additional interface objects than ACT- R/PM

48. Comparison of analyses n Analytical model is not enough n Visual search differences between ACT- R/PM and users n ACT-R/PM and Segman need better representation of interfaces n Cognitive models can make more complicated predictions n ACT-R/PM model is generally slower than users

49. Further work n Cellular phones –This analysis does not work out of the box for cellular phones –These phones have different tasks! (Golightly 2003) n Hutchinson 3G UK phone task (Golightly 2003) –Analysis of menu controls for cellular phone menus, included analytical model –Interface became easier to use when more directional controls were provided

50. Further work n Analyzing ten additional designs –Easy if you use existing automatic models! Fifteen minutes for Fitts Law analysis Forty-five minutes for 500 model runs –Hard if you test with actual users! Can take weeks to get scheduled Humans miss appointments

51. Further work n This analysis is generalizable –The same procedures and techniques can be done with other types of interfaces –Automatic models provide fast, easy analysis that mirrors human performance –Must do task analysis first, otherwise you will test for wrong tasks –The hard work (Fitts Law, ACT-R/PM, Segman) has already been done –Cognitive models are available freely as open source

52. Thank you! n Any questions?

53. Why is this Computer Science? n Interfaces affect how computers are used (Project Ernestine) n Cognitive modeling is an inter- disciplinary effort n Automatic analysis similar to SPICE n Analysis of visual search algorithms –Random search: O(10*n) –Systematic search: O(10+n >0,<1 )

54. References n Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, NJ: Lawrence Erlbaum. n Byrne, M. D. (1999). ACT-R Perceptual-Motor (ACT-R/PM) version 1.0b5: A users manual. Houston, TX: Psychology Department, Rice University. n Byrne, M. D. (2001). ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. International Journal of Human-Computer Studies, 55, 41-84. n Card, S., Moran, T., & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. n Emmerson, P. (2000). Review of iGEN software. Ergonomics in Design, 29-31. n Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391. n Freed, M. A. (1998). Simulating performance in complex, dynamic environments. Northwestern, Evanston, IL. n Golightly, D. (2003). Personal communication. n Gray, W. D., John, B. E., & Atwood, M. E. (1992). The precis of Project Ernestine or An overview of a validation of GOMS. Proceedings of the CHI92 Conference on Human Factors in Computer Systems. n Hornof, A. J., & Kieras, D. E. (1997). Cognitive modeling reveals menu search is both random and systematic. Proceedings of the CHI97 Conference on Human Factors in Computer Systems, New York, NY.

55. References n Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470-516. n Kieras, D. E., & Meyer, D. E. (1997). An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human- Computer Interaction, 12, 391-438. n Kieras, D. E., Wood, S. D., Abotel, K., & Hornof, A. (1995). GLEAN: A computer- based tool for rapid GOMS model usability evaluation of user interface designs. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST'95), New York, NY. n Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press. n Ritter, F. E. (2000). A role for cognitive architectures: Guiding user interface design. Seventh Annual ACT-R Workshop, Department of Psychology, Carnegie-Mellon University. n Ritter, F. E., & Young, R. M. (2001). Embodied models as simulated users: Introduction to this special issue on using cognitive models to improve interface design. International Journal of Human-Computer Studies, 55, 1-14. n St. Amant, R., & Riedl, M. O. (2001). A perception/action substrate for cognitive modeling in HCI. International Journal of Human-Computer Studies, 55, 15-39.