1. blindSight: Eyes-free mobile phone interaction Kevin Li, University of California, San Diego Patrick Baudisch, Microsoft Research Ken Hinckley, Microsoft Research

2. calendar preview “Monday 9am” “tic, tic, sssssh” “How about Monday morning?” “Yeah, looks like I’m free after 10” blindSight

3. is an application running on Microsoft Windows Smartphone is launched when user places or receive a call. It then replaces the in-call menu unlike the in-call menu, blindSight shows nothing on the screen, but speaks to the user

4. why?

5. PCs… PC screens have the users’ undivided attention  design for the visual channel

6. eyePhone

7. environment

8. visual impairment

9. screen-less device

10. can’t see screen

13. Lots of information is stored on mobile phones…

14. … the interfaces are visual

15. # of participants “I need to access as part of a phone conversation:” survey

16.  Ok, so let’s just translate all text from visual to auditory –“Menu: Press 1 to search contacts; press 2 to add a contact; press 3 to access your calendar…”  Wait, that sounds familiar

17. Please listen carefully as our options have changed…

18. related work

19.  User’s should be able to “dial ahead” [Perugini et al.,CHI 2007]  Zap and Zoom allows users to jump to locations using shortcuts [Hornstein, UBILAB Rep 1994]  Use visual channel to inform users about options [Yin and Zhai, CHI 2006] interactive voice response

20.  Time compress audio [Dietz and Yerazunis, UIST 2001]  Integrate speech commands into the conversation [Lyons et al., CHI 2004] Interacting mid-conversation

21. blindSight’s auditory feedback

22. audio is heard only by the user, not by the person at the other end

23. rationale  people can recover from audio interruptions as long as interruption is short human-human conversation contains redundancy can we use this redundancy to inject auditory feedback from the device?

24. so how do we make sure device feedback fits into these time windows of low information content?

25. rules 1. feedback only on-demand hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message home

28. rules 2. brevity 2 13 8 delete 4 9 7 6 5 abc def tuv ghi wxyz pqrs mno jkl next play find contact type 6 “200 hits” type 2 “12 hits” type 7 “Marion”

29. rules 3. non-speech previews of composites week day 3 hours ½ hour block ½h preview day + – – – + _ whereAmI go today preview 3 hours ++ calendar (what if the content is a long list, such as appointments for a day?)

32. rules 4. decomposition week day 3 hours ½ hour next preview day + – – – + _ whereAmI go today preview 3 hours (what if the content is a long list, such as appointments for a day?)

33. rules 5. interruptability user interface runs as a separate thread

34. 1 3 2 6 4 5 9 7 8 action deletesave rules 6. minimize modes 1 3 2 6 4 5 9 7 8 action deletesave mon wed tue sat thu fri sun action deletesave pick daystart timeend time … modes

35. rules 6. minimize modes (  avoid wizards) week day 3 hours ½ hour block ½h preview day + – – – + _ whereAmI go today preview 3 hours ++ + +

36. (shows fast use by an experienced user) demo video

39. hardware

43. space

44. epoxy dots enlarged spaces

45. Flip Ear Visual error

47. 1 3 2 6 4 5 9 7 8 # * 0.

48. Flip Ear Visual error

49. blindSight evaluation

50. interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs.

51. task while “driving”idle (1) schedule appointments and (2) add contacts

52. Overall preference results

53. 1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities… driving) lessons

54. environment visual impairment can’t see screen screen-less device next:

55. eyePhone

56. eyesFreePhone ?

57. blindSight: Eyes-free mobile phone interaction Kevin Li, University of California, San Diego Patrick Baudisch, Microsoft Research Ken Hinckley, Microsoft Research

58. 1. built a system 2. a set of eyes-free design rules 3. keypad modifications enabling eyes-free 4. user study comparing with a product (Smartphone 2003) contributions

60. extra slides

62. 1. feedback only on-demand 2. brevity 3. non-speech previews of composites 4. decomposition 5. interruptability 6. minimize modes rules

63. iterator 1 3 2 6 4 5 9 7 8 action deletesave menu patterns

64. hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message home

65. add contact 2 1 3 8 save 4 9 7 6 0 5 delete

66. iterator 1 3 2 6 4 5 9 7 8 action deletesave menu patterns

67. calendar week day 3 hours ½ hour block ½h preview day + – – – + _ whereAmI go today preview 3 hours ++ + +

68. hear voice note mute speaker phone hear task list add contact record voice find contact cal endar hear emails hear text message 2 1 3 8 save 4 9 7 6 0 5 delete week day 3 hours ½ hour block ½h preview day + – – – + _ whereAmI go today preview 3 hours ++ 2 13 8 delete 4 9 7 6 5 abc def tuv ghi wxyz pqrs mno jkl next play add contact find contact calendar home type folder n items item play + – – – + _ + preview + email, tasks, voice, SMS home help hold bottom left for hold bottom right for menu

69. …is a phenomenon in which people who are perceptually blind in a certain area of their visual field demonstrate some visual awareness, without any qualitative experience blindSight... [wikipedia]

70. don’t mode me in blind sight 10 design rules to allow eyes-free use and flow tactile features

71. …are in in a mobile situation If they requires visual attention, users will fail at their current activity interference with social activities drive off the road… phones…

73. vs. baseline interfaces

74. task times