1. Human Computer Interaction CIS 6930/4930 Section 4188/4186
Interaction Devices
Human Computer Interaction
CIS 6930/4930
Section 4188/4186

2. Interaction Performance
60s vs. Today
Performance
Hz -> GHz
Memory
k -> GB
Storage
k -> TB
Input
punch cards ->
Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web cams
Output
10 character/sec
Megapixel displays, color laser, surround sound, force feedback, VR
Substantial bandwidth increase!

3. Interaction Performance
Future?
Gestural input
Two-handed input
3D I/O
Others: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedback
Engineering research!
Entire companies created around one single technology
Current trend:
Multimodal (using car navigation via buttons or voice)
Helps disabled (esp. those w/ different levels of disability)

4. Keyboard and Keypads QWERTY keyboards been around for a long time
(1870s – Christopher Sholes)
Cons: Not easy to learn
Pros: Familiarity
Stats:
Beginners: 1 keystroke per sec
Average office worker: 5 keystrokes (50 wpm)
Experts: 15 keystrokes per sec (150 wpm)
Is it possible to do better? Suggestions?

5. Keyboard and Keypads Look at the piano for possible inspiration
Court reporter keyboards (one keypress = multiple letters or a word)
300 wpm, requires extensive training and use
Keyboard properties that matter
Size
large - imposing for novices, appears more complex
mobile devices
Adjustable
Reduces RSI, better performance and comfort
Mobile phone keyboards, blackberry devices, etc.

6. Keyboard Layouts QWERTY DVOARK (1920s) ABCDE – style Number pads
Frequently used pairs far apart
Fewer typewriter jams
Electronic approaches don’t jam.. why use it?
DVOARK (1920s)
150 wpm->200 wpm
Reducing errors
Takes about one week to switch
Stops most from trying
ABCDE – style
Easier for non-typists
Studies show no improvement vs. QWERTY
Number pads
What’s in the top row?
Look at phones (slight faster), then look at calculators, keypads
Those for disabled
Split keyboards
KeyBowl’s orbiTouch (screenshot)
Eyetrackers, mice
Dasher - 2d motion with word prediction

7. Keys Current keyboards have been extensively tested
Size
Shape
Required force
Spacing
Speed vs. error rates for majority of users
Distinctive click gives audio feedback
Why membrane keyboards are slow (Atari 400?)
Environment hazards might necessitate
Usually speed is not a factor

8. Keys Guidelines Special keys should be denoted
State keys (such as caps, etc.) should have easily noted states
Special curves or dots for home keys for touch typists
Inverted T Cursor movement keys are important (though cross is easier for novices)
Auto-repeat feature
Improves performance, but only if repeat is customizable (motor impaired, young, old)
Two thinking points:
Why are home keys fastest to type?
Why are certain keys larger? (Enter, Shift, Space bar)
This is called Fitt’s Law

9. Keypads for small devices
PDAs, Cellphones, Game consoles
Fold out keyboards
Virtual keyboard
Cloth keyboards (ElekSen)
Haptic feedback?
Mobile phones
Combine static keys with dynamic soft keys
Multi-tap a key to get to a character
Study: Predictive techniques greatly improve performance
Ex. LetterWise = 20 wpm vs 15 wpm multitap
Draw keyboard on screen and tap w/ pen
Speed: 20 to 30 wpm (Sears ’93)
Handwriting recognition (still hard)
Subset: Graffiti2 (uses unistrokes)

10. Pointing Devices Direct manipulation needs some pointing device
Factors:
Size of device
Accuracy
Dimensionality
Interaction Tasks:
Select – menu selection, from a list
Position – 1D, 2D, 3D (ex. paint)
Orientation – Control orientation or provide direct 3D orientation input
Path – Multiple poses are recorded
ex. to draw a line
Quantify – control widgets that affect variables
Text – move text
Faster w/ less error than keyboard
Two types (Box 9.1)
Direct control – device is on the screen surface (touchscreen, stylus)
Indirect control – mouse, trackball, joystick, touchpad

11. Direct-control pointing
First device – lightpen
Point to a place on screen and press a button
Pros:
Easy to understand and use
Very fast for some operations (e.g. drawing)
Cons:
Hand gets tired fast!
Hand and pen blocks view of screen
Fragile
Evolved into the touchscreen
Pros: Very robust, no moving parts
Cons: Depending on app, accuracy could be an issue
1600x1600 res with acoustic wave
Must be careful about software design for selection (land-on strategy).
If you don’t show a cursor of where you are selecting, users get confused
User confidence is improved with a good lift-off strategy

12. Direct-control pointing
Primarily for novice users or large user base
Case study: Disney World
Need to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

13. Indirect-Control Pointing
Pros:
Reduces hand-fatigue
Reduces obscuration problems
Cons:
Increases cognitive load
Spatial ability comes more into play
Mouse
Familiarity
Wide availability
Low cost
Easy to use
Accurate
Time to grab mouse
Desk space
Encumbrance (wire), dirt
Long motions aren’t easy or obvious (pick up and replace)
Consider, weight, size, style, # of buttons, force feedback

14. Indirect-Control Pointing
Trackball
Pros:
Small physical footprint
Good for kiosks
Joystick
Easy to use, lots of buttons
Good for tracking (guide or follow an on screen object)
Does it map well to your app?
Touchpoint
Pressure-sensitive ‘nubbin’ on laptops
Keep fingers on the home position

15. Indirect-Control Pointing
Touchpad
Laptop mouse device
Lack of moving parts, and low profile
Accuracy, esp. those w/ motor disabilities
Graphics Tablet
Screen shot
comfort
good for cad, artists
Limited data entry

16. Comparing pointing devices
Direct pointing
Study: Faster but less accurate than indirect (Haller ’84)
Lots of studies confirm mouse is best for most tasks for speed and accuracy
Trackpoint < Trackballs & Touchpads < Mouse
Short distances – cursor keys are better
Disabled prefer joysticks and trackballs
If force application is a problem, then touch sensitive is preferred
Vision impaired have problems with most pointing devices
Use multimodal approach or customizable cursors
Read Vanderheiden ’04 for a case study
Designers should smooth out trajectories
Large targets reduce time and frustration

17. Example
Five fastest places to click on for a right-handed user?

18. Example
What affects time?

19. Fitts’s Law MT = a + b log2(D/W + 1)
Paul Fitts (1954) developed a model of human hand movement
Used to predict time to point at an object
What are the factors to determine the time to point to an object?
D – distance to target
W – size of target
Just from your own experience, is this function linear?
No, since if Target A is D distance and Target B is 2D distance, it doesn’t take twice as long
What about target size? Not linear there either
MT = a + b log2(D/W + 1)
a = time to start/stop in seconds (empirically measured per device)
b = inherent speed of the device (empirically measured per device)
Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm
Ans: log2(14/2 + 1) = 900 ms
Really a slope-intercept model

20. Fitts’s Law MT = a + b log2(D/W + 1)
a = time to start/stop in seconds (empirically measured per device)
b = inherent speed of the device (empirically measured per device)
Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm
Ans: log2(14/2 + 1) = 900 ms
Question: If I wanted to half the pointing time (on average), how much do I change the size?
Proven to provide good timings for most age groups
Newer versions taken into account
Direction (we are faster horizontally than vertically)
Device weight
Target shape
Arm position (resting or midair)
2D and 3D (Zhai ’96)

21. Very Successfully Studied
Applies to
Feet, eye gaze, head mounted sights
Many types of input devices
Physical environments (underwater!)
User populations (even retarded and drugged)
Drag & Drop and Point & Click
Limitations
Dimensionality
Software accelerated pointer motion
Training
Trajectory Tasks (Accot-Zhai Steering Law)
Decision Making (Hick’s Law)
Results (what does it say about)
Buttons and widget size?
Edges?
Popup vs. pull-down menus
Pie vs. Linear menus
iPhone/web pages (real borders) vs. monitor+mouse (virtual borders)
Interesting readings:

22. Precision Pointing Movement Time
Study: Sears and Shneiderman ’91
Broke down task into gross and fine components for small targets
PPMT = a + b log2(D/W+1) + c log2(d/W)
c – speed for short distance movement
d – minor distance
Notice how the overall time changes with a smaller target.
Other factors
Age (Pg. 369)
Research: How can we design devices that produce smaller constants for the predictive equation
Two handed
Zooming

23. Novel Devices Themes: Foot controls Make device more diverse
Users
Task
Improve match between task and device
Improve affordance
Refine input
Feedback strategies
Foot controls
Already used in music where hands might be busy
Cars
Foot mouse was twice as slow as hand mouse
Could specify ‘modes’

24. Novel Devices Eye-tracking Multiple degree of freedom devices
Accuracy 1-2 degrees
selections are by constant stare for ms
How do you distinguish w/ a selection and a gaze?
Combine w/ manual input
Multiple degree of freedom devices
Logitech Spaceball and SpaceMouse
Ascension Bird
Polhemus Liberty and IsoTrack

25. Novel Devices Boom Chameleon DataGlove
Pros: Natural, good spatial understanding
Cons: limited applications, hard to interact (very passive)
DataGlove
Pinch glove
Gesture recognition
American Sign Language, musical director
Pros: Natural
Cons: Size, hygiene, accuracy, durability

26. Novel Devices Haptic Feedback Two-Handed input
Why is resistance useful?
SensAble Technology’s Phantom
Cons: limited applications
Sound and vibration are easier and can be a good approximation
Rumble pack
Two-Handed input
Different hands have different precision
Non-dominant hand selects fill, the other selects objects
Ubiquitous Computing and Tangible User Interface
Active Badges allows you to move about the house w/ your profile
Which sensors could you use?
Elderly, disabled
Research: Smart House
Myron Kruger – novel user participation in art (Lots of exhibit art at siggraph)

27. Novel Devices Paper/Whiteboards Handheld Devices
Video capture of annotations
Record notes (special tracked pens Logitech digital pen)
Handheld Devices
PDA
Universal remote
Help disabled
Read LCD screens
Rooms in building
Maps
Interesting body-context-sensitive.
Ex. hold PDA by ear = phone call answer.

28. Novel Devices Miscellaneous
Shapetape – reports 3D shape.
Tracks limbs
Engineer for specific app (like a gun trigger connected to serial port)
Pros: good affordance
Cons: Limited general use, time

29. Speech and Auditory Interfaces
There’s the dream
Then there’s reality
Practical apps don’t really require freeform discussions with a computer
Goals:
Low cognitive load
Low error rates
Smaller goals:
Speech Store and Forward (voice mail)
Speech Generation
Currently not too bad, low cost, available

30. Speech and Auditory Interfaces
Bandwidth is much lower than visual displays
Ephemeral nature of speech (tone, etc.)
Difficulty in parsing/searching (Box 9.2)
Types
Discrete-word recognition
Continuous speech
Voice information
Speech generation
Non-speech auditory
If you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand

31. Discrete-Word Recognition
Individual words spoken by a specific person
Command and control
90-98% for word vocabularies
Training
Speaker speaks the vocabulary
Speaker-independent
Still requires
Low noise operating environment
Microphones
Vocabulary choice
Clear voice (language disabled are hampered, stressed)
Reduce most questions to very distinct answers (yes/no)

32. Discrete-Word Recognition
Helps:
Disabled
Elderly
Cognitive challenged
User is visually distracted
Mobility or space restrictions
Apps:
Telephone-based info
Study: much slower for cursor movement than mouse or keyboard (Christian ’00)
Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl ’93)
However acoustic memory requires high cognitive load (> than hand/eye)
Toys are successful (dolls, robots). Accuracy isn’t as important
Feedback is difficult

33. Continuous Speech Recognition
Dictation
Error rates and error repair are still poor
Higher cognitive load, could lower overall quality
Why is it hard?
Recognize boundaries (normal speech blurs them)
Context sensitivity
“How to wreck a nice beach”
Much training
Specialized vocabularies (like medical or legal)
Apps:
Dictate reports, notes, letters
Communication skills practice (virtual patient)
Automatic retrieval/transcription of audio content (like radio, CC)
Security/user ID

34. Voice Information Systems
Use human voice as a source of info
Apps:
Tourist info
Museum audio tours
Voice menus (Interactive Voice Response IVR systems)
Use speech recognition to also cut through menus
If menus are too long, users get frustrated
Cheaper than hiring 24 hr/day reps
Voice mail systems
Interface isn’t the best
Get in your car
Also helps with non-tech savvy like the elderly
Potentially aides with
Learning (engage more senses)
Cognitive load (hypothesize each sense has a limited ‘bandwidth’)
Think ER, or fighter jets

35. Speech Generation Play back speech (games)
Combine text (navigation systems)
Careful evaluation!
Speech isn’t always great
Door is ajar – now just a tone
Use flash
Supermarket scanners
Often times a simple tone is better
Why? Cognitive load
Thus cockpits and control rooms need speech
Competes w/ human-human communication

36. Speech Generation Ex: Text-to-Speech (TTS)
Latest TTS uses multiple syllabi to make generated speech sound better
Robotic speech could be desirable to get attention
All depends on app
Thus don’t assume one way is the best, you should user test
Apps: TTS for blind, JAWS
Web-based voice apps: VoiceXML and SALT (tagged web pages).
Good for disabled, and also for mobile devices
Use if
Message is short
Requires dynamic responses
Events in time
Good when visual displays aren’t that useful. When?
Bad lighting, vibrations (say liftoff)

37. Non-speech Auditory Interface
Audio tones that provide information
Major Research Area
Sonification – converting information into audio
Audiolization
Auditory Interfaces
Browsers produced a click when you clicked on a link
Increases confidence
Can do tasks without visual cognitive load
Helps figure out when things are wrong
Greatly helps visually impaired

38. Non-speech Auditory Interface
Terms:
Auditory icons – familiar sounds (record real world sound and play it in your app)
Earcons – new learned sounds (door ajar)
Role in video games is huge
Emotions, Tension, set mood
To create 3D sound
Need to do more than stereo
Take into account Head-related transfer function (HRTF)
Ear and head shape
New musical instruments
Theremin
New ways to arrange music

39. Displays Primary Source of feedback Properties: Physical Dimension
Resolution
Color Depth and correctness
Brightness, contrast, glare
Power
Refresh rate
Cost
Reliability
# of users

40. Display Technology Monochrome displays (single color) Color Low cost
Greater intensity range (medical)
Color
Raster Scan CRT
LCD – thin, bright
Plasma – very bright, thin
LED – large public displays
Electronic Ink – new product w/ tiny capsules of negative black particles and positive white
Braille – refreshable cells with dots that rise up

41. Large Displays Wall displays Informational Interactive
Control rooms, military, flight control rooms, emergency response
Provides
System overview
Increases situational awareness
Effective team review
Old: Array of CRTs
Interactive
Require new interaction methods (freehand sketch, PDAs)
Local and remote collaboration
Art, engineering

42. Large Displays Multiple Desktop Displays HMD
Multiple CRTs or Flat panels for large desktops
Cheap
Familiar
Spatial divide up tasks
Comparison tasks are easier
Too much info?
HMD
Eventually -> Every surface a pixel

43. Mobile device displays
Applications
Personal
Reprogrammable picture frames
Digital family portrait (GaTech)
Business
PDAs, cellphones
Medical
Monitor patients
Research: Modality Translation Services (Trace Center – University of Wisconsin)
As you move about it auto converts data, info, etc. for you

44. Mobile device displays
Actions on mobile devices
Monitor information and alert (calendar)
Gather then spread out information (phone)
Participate in groups and relate to individual (networked devices)
Locate services and identify objects (GPS car system)
Capture and then share info (phone)

45. Mobile device displays
Guidelines for design
Bergman ’00, Weiss, ’02
Industry led research and design case studies (Lindholm ’03)
Typically short in time usage (except handheld games)
Optimize for repetitive tasks (rank functions by frequency)
Research: new ways to organize large amounts of info on a small screen
Study: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist ’03)
Searching and web browsing still very poor performance
Promising: Hierarchical representation (show full document and allow user to select where to zoom into)

46. Animation, Image, and Video
Content quality has also greatly increased
3D rendering is near life-like
Digital Photography is common
Scanned documents
Video compression
Multimedia considerations for the disabled
Printers
3D Printers create custom objects from 3D models