designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

why eyes free?

PCs… PC screens have the users’ undivided attention  design for the visual channel

environment

screen-less device

visual impairment

socially unacceptable

can’t see screen

I propose the way to enable easily learnable eyes-free interaction is to leverage pre-learned associations

touch current eyes-free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

auditory feedback

calendar preview “Monday 9am” “tic, tic, sssssh” “How about Monday morning?” “Yeah, looks like I’m free after 10” blindSight [Li et al. 2008]

Pros – Easily learnable – Can convey lots of information Doesn’t always work – Requires direct interaction with device – Not usable in all scenarios (concerts, meetings, driving) tradeoffs

what about touch?

binary is easy

multidimensional tactons [Brown et al. 2006]

Vibratese Language - Communicate via vibrations Map alphabet and digits to 5 vibrators user with 65 hours training -> 90% accuracy, 38wpm System capable of delivering at 67wpm anything can be learned [Geldard 1956]

vibratese language

human-to-human interaction

Tapping and Rubbing

Can we create tactile stimuli that feels like tapping or rubbing? Will people associate it with what they already associate with tapping and rubbing from human-human communication? research questions

Tapping Rubbing [Li et al. in submission]

soundTouch

what feels like a finger?

applications

messaging

in-car navigation

gaming

questionnaire How would you describe the tactile sensations you just experienced to someone who has not experienced them? Which aspects of the experience felt natural and which aspects did not? If your phone could generate these types of sensations, what would you like to use them for?

Tapping has human quality –13 of 16 used the word ‘tap” in their description –12 of 16 volunteered it had a human quality Fast and slow are perceptually different –12 participants mentioned harder taps don’t feel natural –5 said fast ones don’t feel natural results

Taps have a number of characteristics that make them good for alerts – Quiet – Strong Rubbing is more subtle – Useful for in-the-hand scenarios Number of taps and rubs is key element – Sometimes, this has pre-learned meaning – Limits number of viable distinct icons lessons

mapping music to vibrations

PeopleTones: Buddy Proximity Notification

only two states, nearby and far away when a buddy is near, play their song if phone is in vibrate mode, play a matching vibrotactile sequence PeopleTones [Li et al. 2008]

measuring vibrations

zzz z z z

capturing the essence of music

high level approach just using beat doesn’t always work mapping lyrics doesn’t work well

remove noise isolate 6.6kHz to 17.6kHz components using 8 th order Butterworth Filter use amplitude threshold, to keep only components greater than the average

take running sum take running sum of absolute value, generate 1 value every 20ms this keeps length consistent

exaggerate features compose output from previous step with power function: Ax n,x is sample, A and n are constants, 10<=A<15, 1<=n<=2

field study

participants 3 groups of friends, 2 weeks

did you act on the cue?

could you tell who it was?

lessons cues in the wild should be music higher comprehension rate when users select their own cues obtrusiveness of music cues was not a concern -> longer cues semantic association is key

mapping physical objects to tactile feedback

problem tactile feedback is always the same, but visual and motor has directional aspect information is lost in the conversion

Add state information to tactile feedback – Hover state – Moved to the left/right/up/down Where do we put it? – Under the button – Make the sides move – Tapping? – solenoid solution

mapping speech to touch

research questions what are relevant characteristics of speech when mapping to tactile? how do users naturally perceive these to be mapped?

pilot study 5 common phrases from text messaging literature 4 vibration sequences per phrase 20 sequences Which phrase does this vibration feel like?

5 phrases Hello. Goodbye. Where are you? Are you busy? I miss you.

participants agree…

lessons intonation is important syllables should match number of pulses duration should match (roughly)

potential applications learning to sign augmented sms messages messaging backchannel

thesis summary

touch current eyes-free interaction humanobject hearing musicspeechsounds Tapping and Rubbing Simulating buttons People Tones People Tones Mapping language to tactile stimulus class sense example project People Tones People Tones lessons for application designers

timeline

Jan Apr Jun Sept Dec 2008 Ubicomp Doctoral colloquium Paper Deadlines Projects UIST Doctoral symposium CHI Doctoral consortium CHI Doctoral Symposiums Mapping language to tactile Simulating buttons

2009 Paper Deadlines Projects Jan Apr Jun Sept Dec Mapping language to tactile Part 2: Field Deployment Mapping language to tactile Part 2: Field Deployment CHIUIST Tapping and Rubbing Part 2 Find a job

2010 Paper Deadlines Projects Jan Apr Jun Sept Dec Dissertation Find a job Formalize music mapping

designing easily learnable eyes-free interaction Kevin Li University of California, San Diego

EXTRA SLIDES

Please listen carefully as our options have changed…

blindSight evaluation

interfaces Smartphone 2003 (sighted)BlindSight (eyes-free) vs.

task while “driving”idle (1) schedule appointments and (2) add contacts

Was not missing information Knew position in the menu Knew what day/time I was at Felt in control of the conversation Better for setting meeting times Prefer if driving and talking Prefer Overall blindSightSmartphone Overall preference results

1. brevity is good, but use in moderation clarification of navigation overrides brevity 2. predictable/modeless user interface is key 3. auditory feedback goes a long way even during phone call (disclaimer: need to study how it interferes with activities… driving) lessons

generating vibrations

Task: Identify Given a tap rate its strength on a scale of 1-7. Given a tap rate its speed on a scale of 1- 7.