stanford hci group / cs376 u Jeffrey Heer · 19 May 2009 Speech & Multimodal Interfaces

Project Progress Meetings Monday May 25, 30 minute meetings Open times: 9am – 6pm (except 12:30-2pm) To-do ASAP:  cs376 with any times you can not make, plus any time preferences Submit online by 7am Friday:  Any materials you want to discuss 2

Final Project Presentations Tuesday June 9, 3:30-6:30pm, 104 Gates 8 minute presentations  6 minutes to present your research  2 minutes for questions  More details on course website 3

4 Some hci definitions  Multimodal generally refers to an interface that can accept input from two or more combined modes  Multimedia generally refers to an interface that produces output in two or more modes  The vast majority of multimodal systems have been speech + pointing (pen or mouse) input, with graphical (and sometimes voice) output

5 Canonical App: Maps Why are maps so well-suited? A visual artifact for computation (Hutchins)

6 What is an interface?  Is it an interface if there’s no method for a user to tell if they’ve done something?  What might an example be?  Is it an interface if there’s no method for explicit user input?  example: health monitoring apps

7 Multimodal vs. Sensor Fusion multimodal = multiple human channels sensor fusion = multiple sensor channels  Example: Tracking people (1 human channel)  might use: RFID + vision + keyboard activity + …  Disagree with the Oviatt paper?  Speech + lips: multimodality or sensor fusion

8 What constitutes a modality?  To some extent, it’s a matter of semantics  Is pen a different modality than a mouse?  Is a captured modality the same as an input modality?  How does an audio notebook fit into this?

9 Input modalities mouse pen: recognized or unrecognized speech non-speech audio tangible object manipulation gaze, posture, body-tracking  Each of these experiences has different implementing technologies

10 Output modalities Visual displays  Raster graphics, oscilloscope, paper printer, … Haptics, e.g. force feedback Audio Smell Taste

11 Why multimodal?  Hands busy / eyes busy  Mutual disambiguation  Faster / higher bandwidth communication  “More natural”

Example Systems  Dual-Purpose Speech [Georgia Tech]  Active Capture [UC Berkeley] 12

27 April System Direction of Human Action Applications: “Directive” systems Now SCREAM!!! MEDIA TRAINING SAFETY

14 Multimodal Software Architectures  OAA: Open Agent Architecture  AAA: Adaptive Agent Architecture  OOPS: Organized Option Pruning System

How to handle error & ambiguity? Avoidance  Adopt strategies that reduce probability of error or simplify recognition tasks Repetition  Elicit new, less ambiguous input  How to vary prompt to improve recognition? Choice  Let user choose from a ranked list of alternatives 15

Anticipation  Anticipate common errors before they happen. Actively seek out and address problems before they disrupt interaction.

External Aids  Use physical props or other external aids to guide actions and provide feedback.

Confirmation  Explicitly query the user to ensure they are in the expected state.

Progressive Assistance  Provide “successively more informative error messages which consider the probable context of misunderstanding” [Yankelovich95].

Modality Shifts  When a particular direction approach repeatedly fails, switch or augment the modalities of communication, e.g., use visual rather than auditory cues.

Level of Discourse  Simplify vocabulary and language when people have difficulty understanding.

Backtracking  When grounding is lost, backtrack to the last state of mutual understanding.

Graceful Failure  When all else fails, provide natural exits from the interaction. JUMP!

Project Progress Meetings Monday May 25, 30 minute meetings Open times: 9am – 6pm (except 12:30- 2pm) To-do ASAP:  cs376 with any times you can not make, plus any time preferences Submit online by 7am Friday:  Any materials you want to discuss 24