2Interaction Performance 60s vs. TodayPerformanceHz -> GHzMemoryk -> GBStoragek -> TBInputpunch cards ->Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web camsOutput10 character/secMegapixel displays, color laser, surround sound, force feedback, VRSubstantial bandwidth increase!
3Interaction Performance Future?Gestural inputTwo-handed input3D I/OOthers: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedbackEngineering research!Entire companies created around one single technologyCurrent trend:Multimodal (using car navigation via buttons or voice)Helps disabled (esp. those w/ different levels of disability)
4Keyboard and Keypads QWERTY keyboards been around for a long time (1870s – Christopher Sholes)Cons: Not easy to learnPros: FamiliarityStats:Beginners: 1 keystroke per secAverage office worker: 5 keystrokes (50 wpm)Experts: 15 keystrokes per sec (150 wpm)Is it possible to do better? Suggestions?
5Keyboard and Keypads Look at the piano for possible inspiration Court reporter keyboards (one keypress = multiple letters or a word)300 wpm, requires extensive training and useKeyboard properties that matterSizelarge - imposing for novices, appears more complexmobile devicesAdjustableReduces RSI, better performance and comfortMobile phone keyboards, blackberry devices, etc.
6Keyboard Layouts QWERTY DVOARK (1920s) ABCDE – style Number pads Frequently used pairs far apartFewer typewriter jamsElectronic approaches don’t jam.. why use it?DVOARK (1920s)150 wpm->200 wpmReducing errorsTakes about one week to switchStops most from tryingABCDE – styleEasier for non-typistsStudies show no improvement vs. QWERTYNumber padsWhat’s in the top row?Look at phones (slight faster), then look at calculators, keypadsThose for disabledSplit keyboardsKeyBowl’s orbiTouch (screenshot)Eyetrackers, miceDasher - 2d motion with word prediction
7Keys Current keyboards have been extensively tested SizeShapeRequired forceSpacingSpeed vs. error rates for majority of usersDistinctive click gives audio feedbackWhy membrane keyboards are slow (Atari 400?)Environment hazards might necessitateUsually speed is not a factor
8Keys Guidelines Special keys should be denoted State keys (such as caps, etc.) should have easily noted statesSpecial curves or dots for home keys for touch typistsInverted T Cursor movement keys are important (though cross is easier for novices)Auto-repeat featureImproves performance, but only if repeat is customizable (motor impaired, young, old)Two thinking points:Why are home keys fastest to type?Why are certain keys larger? (Enter, Shift, Space bar)This is called Fitt’s Law
9Keypads for small devices PDAs, Cellphones, Game consolesFold out keyboardsVirtual keyboardCloth keyboards (ElekSen)Haptic feedback?Mobile phonesCombine static keys with dynamic soft keysMulti-tap a key to get to a characterStudy: Predictive techniques greatly improve performanceEx. LetterWise = 20 wpm vs 15 wpm multitapDraw keyboard on screen and tap w/ penSpeed: 20 to 30 wpm (Sears ’93)Handwriting recognition (still hard)Subset: Graffiti2 (uses unistrokes)
10Pointing Devices Direct manipulation needs some pointing device Factors:Size of deviceAccuracyDimensionalityInteraction Tasks:Select – menu selection, from a listPosition – 1D, 2D, 3D (ex. paint)Orientation – Control orientation or provide direct 3D orientation inputPath – Multiple poses are recordedex. to draw a lineQuantify – control widgets that affect variablesText – move textFaster w/ less error than keyboardTwo types (Box 9.1)Direct control – device is on the screen surface (touchscreen, stylus)Indirect control – mouse, trackball, joystick, touchpad
11Direct-control pointing First device – lightpenPoint to a place on screen and press a buttonPros:Easy to understand and useVery fast for some operations (e.g. drawing)Cons:Hand gets tired fast!Hand and pen blocks view of screenFragileEvolved into the touchscreenPros: Very robust, no moving partsCons: Depending on app, accuracy could be an issue1600x1600 res with acoustic waveMust be careful about software design for selection (land-on strategy).If you don’t show a cursor of where you are selecting, users get confusedUser confidence is improved with a good lift-off strategy
12Direct-control pointing Primarily for novice users or large user baseCase study: Disney WorldNeed to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.
13Indirect-Control Pointing Pros:Reduces hand-fatigueReduces obscuration problemsCons:Increases cognitive loadSpatial ability comes more into playMouseFamiliarityWide availabilityLow costEasy to useAccurateTime to grab mouseDesk spaceEncumbrance (wire), dirtLong motions aren’t easy or obvious (pick up and replace)Consider, weight, size, style, # of buttons, force feedback
14Indirect-Control Pointing TrackballPros:Small physical footprintGood for kiosksJoystickEasy to use, lots of buttonsGood for tracking (guide or follow an on screen object)Does it map well to your app?TouchpointPressure-sensitive ‘nubbin’ on laptopsKeep fingers on the home position
15Indirect-Control Pointing TouchpadLaptop mouse deviceLack of moving parts, and low profileAccuracy, esp. those w/ motor disabilitiesGraphics TabletScreen shotcomfortgood for cad, artistsLimited data entry
16Comparing pointing devices Direct pointingStudy: Faster but less accurate than indirect (Haller ’84)Lots of studies confirm mouse is best for most tasks for speed and accuracyTrackpoint < Trackballs & Touchpads < MouseShort distances – cursor keys are betterDisabled prefer joysticks and trackballsIf force application is a problem, then touch sensitive is preferredVision impaired have problems with most pointing devicesUse multimodal approach or customizable cursorsRead Vanderheiden ’04 for a case studyDesigners should smooth out trajectoriesLarge targets reduce time and frustration
17ExampleFive fastest places to click on for a right-handed user?
19Fitts’s Law MT = a + b log2(D/W + 1) Paul Fitts (1954) developed a model of human hand movementUsed to predict time to point at an objectWhat are the factors to determine the time to point to an object?D – distance to targetW – size of targetJust from your own experience, is this function linear?No, since if Target A is D distance and Target B is 2D distance, it doesn’t take twice as longWhat about target size? Not linear there eitherMT = a + b log2(D/W + 1)a = time to start/stop in seconds (empirically measured per device)b = inherent speed of the device (empirically measured per device)Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmAns: log2(14/2 + 1) = 900 msReally a slope-intercept model
20Fitts’s Law MT = a + b log2(D/W + 1) a = time to start/stop in seconds (empirically measured per device)b = inherent speed of the device (empirically measured per device)Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmAns: log2(14/2 + 1) = 900 msQuestion: If I wanted to half the pointing time (on average), how much do I change the size?Proven to provide good timings for most age groupsNewer versions taken into accountDirection (we are faster horizontally than vertically)Device weightTarget shapeArm position (resting or midair)2D and 3D (Zhai ’96)
21Very Successfully Studied Applies toFeet, eye gaze, head mounted sightsMany types of input devicesPhysical environments (underwater!)User populations (even retarded and drugged)Drag & Drop and Point & ClickLimitationsDimensionalitySoftware accelerated pointer motionTrainingTrajectory Tasks (Accot-Zhai Steering Law)Decision Making (Hick’s Law)Results (what does it say about)Buttons and widget size?Edges?Popup vs. pull-down menusPie vs. Linear menusiPhone/web pages (real borders) vs. monitor+mouse (virtual borders)Interesting readings:
22Precision Pointing Movement Time Study: Sears and Shneiderman ’91Broke down task into gross and fine components for small targetsPPMT = a + b log2(D/W+1) + c log2(d/W)c – speed for short distance movementd – minor distanceNotice how the overall time changes with a smaller target.Other factorsAge (Pg. 369)Research: How can we design devices that produce smaller constants for the predictive equationTwo handedZooming
23Novel Devices Themes: Foot controls Make device more diverse UsersTaskImprove match between task and deviceImprove affordanceRefine inputFeedback strategiesFoot controlsAlready used in music where hands might be busyCarsFoot mouse was twice as slow as hand mouseCould specify ‘modes’
24Novel Devices Eye-tracking Multiple degree of freedom devices Accuracy 1-2 degreesselections are by constant stare for msHow do you distinguish w/ a selection and a gaze?Combine w/ manual inputMultiple degree of freedom devicesLogitech Spaceball and SpaceMouseAscension BirdPolhemus Liberty and IsoTrack
25Novel Devices Boom Chameleon DataGlove Pros: Natural, good spatial understandingCons: limited applications, hard to interact (very passive)DataGlovePinch gloveGesture recognitionAmerican Sign Language, musical directorPros: NaturalCons: Size, hygiene, accuracy, durability
26Novel Devices Haptic Feedback Two-Handed input Why is resistance useful?SensAble Technology’s PhantomCons: limited applicationsSound and vibration are easier and can be a good approximationRumble packTwo-Handed inputDifferent hands have different precisionNon-dominant hand selects fill, the other selects objectsUbiquitous Computing and Tangible User InterfaceActive Badges allows you to move about the house w/ your profileWhich sensors could you use?Elderly, disabledResearch: Smart HouseMyron Kruger – novel user participation in art (Lots of exhibit art at siggraph)
27Novel Devices Paper/Whiteboards Handheld Devices Video capture of annotationsRecord notes (special tracked pens Logitech digital pen)Handheld DevicesPDAUniversal remoteHelp disabledRead LCD screensRooms in buildingMapsInteresting body-context-sensitive.Ex. hold PDA by ear = phone call answer.
28Novel Devices Miscellaneous Shapetape – reports 3D shape.Tracks limbsEngineer for specific app (like a gun trigger connected to serial port)Pros: good affordanceCons: Limited general use, time
29Speech and Auditory Interfaces There’s the dreamThen there’s realityPractical apps don’t really require freeform discussions with a computerGoals:Low cognitive loadLow error ratesSmaller goals:Speech Store and Forward (voice mail)Speech GenerationCurrently not too bad, low cost, available
30Speech and Auditory Interfaces Bandwidth is much lower than visual displaysEphemeral nature of speech (tone, etc.)Difficulty in parsing/searching (Box 9.2)TypesDiscrete-word recognitionContinuous speechVoice informationSpeech generationNon-speech auditoryIf you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand
31Discrete-Word Recognition Individual words spoken by a specific personCommand and control90-98% for word vocabulariesTrainingSpeaker speaks the vocabularySpeaker-independentStill requiresLow noise operating environmentMicrophonesVocabulary choiceClear voice (language disabled are hampered, stressed)Reduce most questions to very distinct answers (yes/no)
32Discrete-Word Recognition Helps:DisabledElderlyCognitive challengedUser is visually distractedMobility or space restrictionsApps:Telephone-based infoStudy: much slower for cursor movement than mouse or keyboard (Christian ’00)Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl ’93)However acoustic memory requires high cognitive load (> than hand/eye)Toys are successful (dolls, robots). Accuracy isn’t as importantFeedback is difficult
33Continuous Speech Recognition DictationError rates and error repair are still poorHigher cognitive load, could lower overall qualityWhy is it hard?Recognize boundaries (normal speech blurs them)Context sensitivity“How to wreck a nice beach”Much trainingSpecialized vocabularies (like medical or legal)Apps:Dictate reports, notes, lettersCommunication skills practice (virtual patient)Automatic retrieval/transcription of audio content (like radio, CC)Security/user ID
34Voice Information Systems Use human voice as a source of infoApps:Tourist infoMuseum audio toursVoice menus (Interactive Voice Response IVR systems)Use speech recognition to also cut through menusIf menus are too long, users get frustratedCheaper than hiring 24 hr/day repsVoice mail systemsInterface isn’t the bestGet in your carAlso helps with non-tech savvy like the elderlyPotentially aides withLearning (engage more senses)Cognitive load (hypothesize each sense has a limited ‘bandwidth’)Think ER, or fighter jets
35Speech Generation Play back speech (games) Combine text (navigation systems)Careful evaluation!Speech isn’t always greatDoor is ajar – now just a toneUse flashSupermarket scannersOften times a simple tone is betterWhy? Cognitive loadThus cockpits and control rooms need speechCompetes w/ human-human communication
36Speech Generation Ex: Text-to-Speech (TTS) Latest TTS uses multiple syllabi to make generated speech sound betterRobotic speech could be desirable to get attentionAll depends on appThus don’t assume one way is the best, you should user testApps: TTS for blind, JAWSWeb-based voice apps: VoiceXML and SALT (tagged web pages).Good for disabled, and also for mobile devicesUse ifMessage is shortRequires dynamic responsesEvents in timeGood when visual displays aren’t that useful. When?Bad lighting, vibrations (say liftoff)
37Non-speech Auditory Interface Audio tones that provide informationMajor Research AreaSonification – converting information into audioAudiolizationAuditory InterfacesBrowsers produced a click when you clicked on a linkIncreases confidenceCan do tasks without visual cognitive loadHelps figure out when things are wrongGreatly helps visually impaired
38Non-speech Auditory Interface Terms:Auditory icons – familiar sounds (record real world sound and play it in your app)Earcons – new learned sounds (door ajar)Role in video games is hugeEmotions, Tension, set moodTo create 3D soundNeed to do more than stereoTake into account Head-related transfer function (HRTF)Ear and head shapeNew musical instrumentsThereminNew ways to arrange music
39Displays Primary Source of feedback Properties: Physical Dimension ResolutionColor Depth and correctnessBrightness, contrast, glarePowerRefresh rateCostReliability# of users
40Display Technology Monochrome displays (single color) Color Low cost Greater intensity range (medical)ColorRaster Scan CRTLCD – thin, brightPlasma – very bright, thinLED – large public displaysElectronic Ink – new product w/ tiny capsules of negative black particles and positive whiteBraille – refreshable cells with dots that rise up
41Large Displays Wall displays Informational Interactive Control rooms, military, flight control rooms, emergency responseProvidesSystem overviewIncreases situational awarenessEffective team reviewOld: Array of CRTsInteractiveRequire new interaction methods (freehand sketch, PDAs)Local and remote collaborationArt, engineering
42Large Displays Multiple Desktop Displays HMD Multiple CRTs or Flat panels for large desktopsCheapFamiliarSpatial divide up tasksComparison tasks are easierToo much info?HMDEventually -> Every surface a pixel
43Mobile device displays ApplicationsPersonalReprogrammable picture framesDigital family portrait (GaTech)BusinessPDAs, cellphonesMedicalMonitor patientsResearch: Modality Translation Services (Trace Center – University of Wisconsin)As you move about it auto converts data, info, etc. for you
44Mobile device displays Actions on mobile devicesMonitor information and alert (calendar)Gather then spread out information (phone)Participate in groups and relate to individual (networked devices)Locate services and identify objects (GPS car system)Capture and then share info (phone)
45Mobile device displays Guidelines for designBergman ’00, Weiss, ’02Industry led research and design case studies (Lindholm ’03)Typically short in time usage (except handheld games)Optimize for repetitive tasks (rank functions by frequency)Research: new ways to organize large amounts of info on a small screenStudy: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist ’03)Searching and web browsing still very poor performancePromising: Hierarchical representation (show full document and allow user to select where to zoom into)
46Animation, Image, and Video Content quality has also greatly increased3D rendering is near life-likeDigital Photography is commonScanned documentsVideo compressionMultimedia considerations for the disabledPrinters3D Printers create custom objects from 3D models