1. HUMAN COMPUTER INTERACTION CIS 6930/4930 INTERACTION DEVICES

2. INTERACTION PERFORMANCE 60s vs. Today Performance Hz -> GHz Memory k -> GB Storage k -> TB

3. INTERACTION PERFORMANCE 60s vs. Today Input punch cards -> Keyboards, Pens, tablets, mobile phones, mice, cameras, web cams Output 10 character/sec -> Megapixel displays, HD capture and display, color laser, surround sound, force feedback, VR Substantial bandwidth increase!

4. INTERACTION FUTURE Gestural input Two-handed input 3D/6D I/O Glass (1:52) Glass Others: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedback Engineering research! Entire companies created around one single technology Magic Leap http://virginradiodubai.blob.core.windows.net/images/2014/05/googleglass_660.jpg

5. INTERACTION CURRENT TRENDS Multimodal (using car navigation via buttons or voice) Helps disabled (especially those with different levels of disability)

6. KEYBOARD AND KEYPADS QWERTY keyboards been around for a long time (1870s – Christopher Sholes) Cons: Not easy to learn Pros: Familiarity Stats: Beginners: 1 keystroke per sec Average office worker: 5 keystrokes (50 wpm) Experts: 15 keystrokes per sec (150 wpm) Is it possible to do better?

7. KEYBOARD AND KEYPADS How important is: Accuracy Training Keyboard properties that matter Size Adjustability Reduces RSI, better performance and comfort Mobile phone keyboards, blackberry devices, etc.

8. KEYBOARD AND KEYPADS How important is: Accuracy Training Keyboard properties that matter Size Adjustability Reduces RSI, better performance and comfort Mobile phone keyboards, blackberry devices, etc.

9. KEYBOARD LAYOUTS QWERTY Frequently used pairs far apart Fewer typewriter jams Electronic approaches don’t jam.. why use it? DVOARK (1920s) 150 wpm->200 wpm Reducing errors Takes about one week to switch Stops most from trying

10. KEYBOARD LAYOUTS ABCDE – style Easier for non-typists Studies show no improvement vs. QWERTY Number pads What’s in the top row? Look at phones (slight faster), then look at calculators, keypads Those for disabled Split keyboards KeyBowl’s orbiTouch Eyetrackers, mice Dasher - 2d motion with word prediction Dasher

11. KEYS Current keyboards have been extensively tested Size Shape Required force Spacing Speed vs. error rates for majority of users Distinctive click gives audio feedback Why membrane keyboards are slow (Atari 400?) Environment hazards might necessitate Usually speed is not a factor

12. KEYS GUIDELINES Special keys should be denoted State keys (such as caps, etc.) should have easily noted states Special curves or dots for home keys for touch typists Inverted T Cursor movement keys are important (though cross is easier for novices)

13. KEYS GUIDELINES Auto-repeat feature Improves performance But only if repeat is customizable (motor impaired, young, old) Two thinking points: Why are home keys fastest to type? Why are certain keys larger? (Enter, Shift, Space bar) Another example of Fitt’s Law

14. KEYPADS FOR SMALL DEVICES PDAs, Cellphones, Game consoles Fold out keyboards Virtual keyboard Cloth keyboards (ElekSen) Most lack haptic feedback?

15. KEYPADS FOR SMALL DEVICES Mobile phones Combine static keys with dynamic soft keys Multi-tap a key to get to a character Study: Predictive techniques greatly improve performance Ex. LetterWise = 20 wpm vs 15 wpm multitap Draw keyboard on screen and tap w/ pen Speed: 20 to 30 wpm (Sears ’93) Swipe Handwriting recognition (still hard) Subset: Graffiti2 (uses unistrokes)

16. POINTING DEVICES Direct manipulation needs some pointing device Factors: Size of device Accuracy Dimensionality Interaction Tasks: Select – menu selection, from a list Position – 1D, 2D, 3D (ex. paint) Orientation – Control orientation or provide direct 3D orientation input Path – Multiple poses are recorded ex. to draw a line Quantify – control widgets that affect variables Text – move text

17. POINTING DEVICES Faster w/ less error than keyboard Two types (Box 9.1) Direct control – device is on the screen surface (touchscreen, stylus) Indirect control – mouse, trackball, joystick, touchpad

18. DIRECT-CONTROL POINTING First device – lightpenlightpen Point to a place on screen and press a button Pros: Easy to understand and use Very fast for some operations (e.g. drawing) Cons: Hand gets tired fast! Hand and pen blocks view of screen Fragile

19. DIRECT-CONTROL POINTING Evolved into the touchscreen Pros: Very robust, no moving parts Cons: Depending on app, accuracy could be an issue 1600x1600 res with acoustic wave Must be careful about software design for selection (land-on strategy). If you don’t show a cursor of where you are selecting, users get confused User confidence is improved with a good lift-off strategy Now combination Nintendo DS Samsung Note

20. DIRECT-CONTROL POINTING Primarily for novice users or large user base Case study: Disney World Need to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

21. INDIRECT-CONTROL POINTING Pros: Reduces hand-fatigue Reduces obscuration problems Cons: Increases cognitive load Spatial ability comes more into play

22. INDIRECT-CONTROL POINTING - MOUSE Pros: Familiarity Wide availability Low cost Easy to use Accurate Cons: Time to grab mouse Desk space Encumbrance (wire), dirt Long motions aren’t easy or obvious (pick up and replace) Consider, weight, size, style, # of buttons, force feedback

23. INDIRECT-CONTROL POINTING Trackball Pros: Small physical footprint Good for kiosks Joystick Easy to use, lots of buttons Good for tracking (guide or follow an on screen object) Does it map well to your app? Touchpoint Pressure-sensitive ‘nubbin’ on laptops Keep fingers on the home position

24. INDIRECT-CONTROL POINTING Touchpad Laptop mouse device Lack of moving parts, and low profile Accuracy potentially low for those with motor disabilities Graphics Tablet Comfort Good for CAD, artists Limited data entry

25. COMPARING POINTING DEVICES Direct pointing Study: Faster but less accurate than indirect (Haller ’84) Lots of studies confirm mouse is best for most tasks for speed and accuracy Trackpoint < Trackballs & Touchpads < Mouse Short distances – cursor keys are better (experts use keyboard for movement more) Disabled prefer joysticks and trackballs If force application is a problem, then touch sensitive is preferred Vision impaired have problems with most pointing devices Use multimodal approach or customizable cursors Read Vanderheiden ’04 for a case study Designers should smooth out trajectories Large targets reduce time and frustration

26. EXAMPLE Five fastest places to click on for a right-handed user?

27. EXAMPLE What affects time?

28. Fitt’s Law Recreation.5” 1” 2”

29. FITTS’S LAW Paul Fitts (1954) developed a model of human hand movement Used to predict time to point at an object What are the factors to determine the time to point to an object? D – distance to target W – size of target Just from your own experience, is this function linear? No, since if Target A is D distance and Target B is 2D distance, it doesn’t take twice as long What about target size? Not linear there either T = a + b log 2 (D/W + 1) http://www.lynda.com/Web-User-Experience- tutorials/Understanding-Fittss-Law/103677/119792- 4.html http://www.lynda.com/Web-User-Experience- tutorials/Understanding-Fittss-Law/103677/119792- 4.html

30. FITTS’S LAW T = a + b log 2 (D/W + 1) T = mean time a = time to start/stop in seconds (empirically measured per device) b = inherent speed of the device (empirically measured per device) [time/bit or ms/bit] Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm Ans: 300 + 200 log 2 (14/2 + 1) = 900 ms Question: If I wanted to half the pointing time (on average), how much do I change the size? Proven to provide good timings for most age groups Newer versions taken into account Direction (we are faster horizontally than vertically) Device weight Target shape Arm position (resting or midair) 2D and 3D (Zhai ’96)

31. EXAMPLES T = a + b log 2 (D/W + 1)

32. EXAMPLES T = a + b log 2 (D/W + 1)

33. EXAMPLES

34. FITTS’S LAW T = a + b log 2 (D/W + 1) T = mean time a = time to start/stop in seconds (empirically measured per device) b = inherent speed of the device (empirically measured per device) [time/bit or ms/bit] First part is device characteristics Second part is target difficulty

35. VERY SUCCESSFULLY STUDIED Applies to Feet, eye gaze, head mounted sights Many types of input devices Physical environments (underwater!) User populations (even mentally handicapped and drugged) Drag & Drop and Point & Click Limitations Dimensionality Software accelerated pointer motion Training Trajectory Tasks (Accot-Zhai Steering Law is a good predictor and joins Fitt’s Law) Decision Making (Hick’s Law)

36. VERY SUCCESSFULLY STUDIED Results (what does it say about) Buttons and widget size? Edges? Popup vs. pull-down menus Pie vs. Linear menus iPhone/web pages (real borders) vs. monitor+mouse (virtual borders) Interesting readings: http://particletree.com/features/vis ualizing-fittss-law/ http://particletree.com/features/vis ualizing-fittss-law/ http://www.asktog.com/columns/0 22DesignedToGiveFitts.html http://www.asktog.com/columns/0 22DesignedToGiveFitts.html http://www.yorku.ca/mack/GI92.ht ml http://www.yorku.ca/mack/GI92.ht ml Using Fitt’s Law to slow people down Using Fitt’s Law to slow people down

37. PRECISION POINTING MOVEMENT TIME Study: Sears and Shneiderman ’91 Broke down task into gross and fine components for small targets Precision Point Mean Time = a + b log 2 (D/W+1) + c log 2 (d/W) c – speed for short distance movement d – minor distance Notice how the overall time changes with a smaller target. Other factors Age (Pg. 369) Research: How can we design devices that produce smaller constants for the predictive equation Two handed Zooming

38. AFFORDANCE Quality of an object, or an environment, that allows an individual to perform an action. Gibson (’77) – perceived action possibilities Norman – The Design of Everyday Things

39. AFFORDANCE EXAMPLES

41. https://jbs2010.wordpress.com/2010/06/23/affordances-making-things-visible/

42. FOR YOUR PROJECT Look at the interface What will people assume they can do with it? Write it down.

43. TRADEOFF FOR NEW INTERFACES Consider a military training simulator How would you allow a user to user a gun in the simulator? Engineered Device High Affordance High Cost Low Reusability Standard Device Low Affordance Low Cost High Reusability

44. NOVEL DEVICES Themes: Make device more diverse Users Task Improve match between task and device Improve affordance Refine input Feedback strategies Foot controls Already used in music where hands might be busy Cars Foot mouse was twice as slow as hand mouse Could specify ‘modes’ Xk-a-75-r pedal switch

45. NOVEL DEVICES Eye-tracking Either worn by user or in the environment (e.g. Tobii) Accuracy.4 degrees Selections are by constant stare for 200-600 ms How do you distinguish w/ a selection and a gaze? video games, studying user behavior (1:41), design evaluation (pause 0:24) video gamesstudying user behaviordesign evaluation Multiple degree of freedom devices Logitech Spaceball and SpaceMouse Ascension Bird Polhemus Liberty and IsoTrack Polhemus

46. NOVEL DEVICES Boom Chameleon Pros: Natural, good spatial understanding Cons: limited applications, hard to interact (very passive). Not in production Large simulators DataGlove Pinch glove Gesture recognition American Sign Language Music Pros: Natural Cons: Size, hygiene, accuracy, durability

47. NOVEL DEVICES Haptic Feedback Why is resistance useful? SensAble Technology’s Phantom, Novint Falcon Cons: limited applications, computational complex (1 kHz update rate) Sound and vibration can be a good approximation Rumble pack Two-Handed input Different hands have different precision Myron Kruger – novel user participation in art (Lots of exhibit art at siggraph) Myron Kruger

48. UBIQUITOUS COMPUTING AND TANGIBLE USER INTERFACES Interacting with physical objects https://www.youtube.com/w atch?v=Rik8Z_TaxDw https://www.youtube.com/w atch?v=Rik8Z_TaxDw Which sensors could you use? Elderly, disabled Research: Smart House http:// www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/030/3047/3047f2.png

49. NOVEL DEVICES Paper/Whiteboards Video capture of annotations Record notes (special tracked pens Logitech digital pen) Handheld Devices Smartphones/PDA Universal remote Help disabled Read LCD screens Rooms in building Maps Interesting body-context-sensitive. Ex. hold phone by ear = phone call answer.

50. NOVEL DEVICES Miscellaneous Shapetape – reports 3D shape. Shapetape Tracks limbs

51. SPEECH AND AUDITORY INTERFACES There’s the dream There’s the dream Then there’s reality Then there’s reality Practical apps don’t really require freeform discussions with a computer Goals: Low cognitive load Low error rates Smaller goals: Speech Store and Forward (voice mail) Speech Generation Currently not too bad, low cost, available

52. SPEECH AND AUDITORY INTERFACES Ray Kurzweil (’87) – first commercial speech recognition software Bandwidth is much lower than visual displays Ephemeral nature of speech (tone, etc.) Difficulty in parsing/searching (Box 9.2) http://www.kurzweiltech.com/raybio.html

53. SPEECH AND AUDITORY INTERFACES Types Discrete-word recognition Continuous speech Voice information Speech generation Non-speech auditory If you want to do research here, review research in: Audio Audio psychology Digital signal processing http://www.kurzweiltech.com/raybio.html

54. DISCRETE-WORD RECOGNITION Individual words spoken by a specific person Command and control 90-98% for 100-10000 word vocabularies Training Speaker speaks the vocabulary Speaker-independent Still requires Low noise operating environment Microphones Vocabulary choice Clear voice (language disabled are hampered, stressed) Reduce most questions to very distinct answers (yes/no)

55. DISCRETE-WORD RECOGNITION Helps: Disabled Elderly Cognitive challenged User is visually distracted Mobility or space restrictions Apps: Telephone-based info Study: much slower for cursor movement than mouse or keyboard (Christian ’00) Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl ’93) However acoustic memory requires high cognitive load (> than hand/eye) Toys are successful (dolls, robots). Accuracy isn’t as important Feedback is difficult

56. CONTINUOUS SPEECH RECOGNITION Dictation Error rates and error repair are still poor Higher cognitive load, could lower overall quality Why is it hard? Recognize boundaries (normal speech blurs them) Context sensitivity “How to wreck a nice beach” Much training Specialized vocabularies (like medical or legal) Apps: Dictate reports, notes, letters Communication skills practice (virtual patient) Automatic retrieval/transcription of audio content (like radio, CC) Security/user ID

57. VOICE INFORMATION SYSTEMS Use human voice as a source of info Apps: Tourist info Museum audio tours Voice menus (Interactive Voice Response IVR systems) Use speech recognition to also cut through menus If menus are too long, users get frustrated Cheaper than hiring 24 hr/day reps Voice mail systems Interface isn’t the best Get email in your car Also helps with non-tech savvy like the elderly Potentially aides with Learning (engage more senses) Cognitive load (hypothesize each sense has a limited ‘bandwidth’) Think ER, or fighter jets

58. SPEECH GENERATION Play back speech (games) Combine text (navigation systems) Careful evaluation! Speech isn’t always great Door is ajar – now just a tone Supermarket scanners Often times a simple tone is better Why? Speech involves cognitive load Competes w/ human-human communication Thus cockpits and control rooms need speech

59. SPEECH GENERATION Ex: Text-to-Speech (TTS) Latest TTS uses multiple syllabi to make generated speech sound better Latest TTS Robotic speech could be desirable to get attention All depends on app Thus don’t assume one way is the best, you should user test Apps: TTS for blind, JAWS Web-based voice apps: VoiceXML and SALT (tagged web pages). Good for disabled, and also for mobile devices Use if Message is short Requires dynamic responses Events in time Good when visual displays aren’t that useful. When? Bad lighting, vibrations (say liftoff)

60. NON-SPEECH AUDITORY INTERFACE Audio tones that provide information Major Research Area Sonification – converting information into audio Audiolization Auditory Interfaces Browsers produced a click when you clicked on a link Increases confidence Can do tasks without visual cognitive load Helps figure out when things are wrong Greatly helps visually impaired

61. NON-SPEECH AUDITORY INTERFACE Terms: Auditory icons – familiar sounds (record real world sound and play it in your app) Earcons – new learned sounds (door ajar) Earcons Role in video games is huge Emotions, Tension, set mood To create 3D sound Need to do more than stereo Take into account Head- related transfer function (HRTF)HRTF Ear and head shape New musical instruments Theremin New ways to arrange music

62. DISPLAY TECHNOLOGY Monochrome displays (single color) Low cost Greater intensity range (medical) Color Raster Scan CRT LCD – thin, bright Plasma – very bright, thin LED – large public displays Electronic Ink – new product w/ tiny capsules of negative black particles and positive white Braille – refreshable cells with dots that rise up

63. WALL DISPLAYS 1024*768 = 786k, 1080p = 2m, large displays = 20m+ Visual images of gigapixels Informational Control rooms, military, flight control rooms, emergency response Provides System overview Increases situational awareness Effective team review Interactive Require new interaction methods (freehand sketch, handheld) Local and remote collaboration Art, engineering

64. LARGE DISPLAYS Multiple Desktop Displays Multiple CRTs or Flat panels for large desktops Cheap Familiar Spatial divide up tasks Comparison tasks are easier Too much info? Eventually -> Every surface a pixel

65. MOBILE DEVICE DISPLAYS Personal Reprogrammable picture frames Digital family portrait (GaTech) Medical Monitor patients Research: Modality Translation Services (Trace Center – University of Wisconsin) As you move about it auto converts data, info, etc. for you

66. MOBILE DEVICE DISPLAYS GUIDLINES Bergman ’00, Weiss, ’02 Industry led research and design case studies (Lindholm ’03) Typically short in time usage (except handheld games) Optimize for repetitive tasks (rank functions by frequency) Research: new ways to organize large amounts of info on a small screen Study: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist ’03) Searching and web browsing still very poor performance Promising: Hierarchical representation (show full document and allow user to select where to zoom into)

67. 3D PRINTING Create custom objects from 3D models Create custom objects from 3D models Create physical models for Design review Construction