1. Multimodal Human-Computer Interaction New Interaction Techniques 22. 1
Multimodal Human-Computer Interaction New Interaction Techniques Roope Raisamo Department of Computer and Information Sciences University of Tampere, Finland

2. Multimodal human-computer interaction
A definition [Raisamo, 1999e, p. 2]:
”Multimodal interfaces combine many simultaneous input modalities and may present the information using synergistic representation of many different output modalities”

3. Multimodal interaction techniques
Our definition of an interaction technique [Raisamo, 2000]:
An interaction technique is a way to carry out an interactive task. It is defined in the binding, sequencing, and functional levels, and is based on using a set of input and output devices or technologies.
In a multimodal interaction technique there are more than one inputs or outputs used for the same task.

4. Two views A Human-Centered View A System-Centered View
common in psychology
often considers human input channels, i.e., computer output modalities, and most often vision and hearing
applications: a talking head, audio-visual speech recognition, ...
A System-Centered View
common in computer science
a way to make computer systems more adaptable

5. Multimodal human- computer interaction

6. Senses and modalities
[Silbernagel, 1979]

7. Design space for multimodal user interfaces
[Nigay and Coutaz, 1993]

8. An architecture for multimodal user interfaces
Adapted from [Maybury and Wahlster, 1998]

9. Modeling
[Nigay and Coutaz, 1993]

10. Put – That – There
[Bolt, 1980]

11. Potential benefits A list by Maybury and Wahlster [1998, p. 15]:
Efficiency
Redundancy
Perceptability
Naturalness
Accuracy
Synergy
Mutual disambiguation of recognition errors [Oviatt, 1999a]

12. Common misconceptions
A list by Oviatt [1999b]:
If you build a multimodal system, user will interact multimodally.
Speech and pointing is the dominant multimodal integration pattern.
Multimodal input involves simultaneous signals.
Speech is the primary input mode in any multimodal system that includes it.
Multimodal language does not differ linguistically from unimodal language.

13. Common misconceptions
Multimodal integration involves redundancy of content between modes.
Individual error-prone recognition technologies combine multimodally to produce even greater unreliability.
All users’ multimodal commands are integrated in a uniform way.
Different input modes are capable of transmitting comparable content.
Enhanced efficiency is the main advantage of multimodal systems.

14. Two paradigms for multimodal user interfaces
Computer as a tool
multiple input modalities are used to enhance direct manipulation behavior of the system
the machine is a passive tool and tries to understand the user through all different input modalities that the system recognizes
the user is always responsible for initiating the operations
follows the principles of direct manipulation [Shneiderman, 1982; 1983]

15. Two paradigms for multimodal user interfaces
Computer as a dialogue partner
the multiple modalities are used to increase the anthropomorphism in the user interface
multimodal output is important: talking heads and other human-like modalities
speech recognition is a common input modality in these systems
can often be described as an agent-based conversational user interface

16. Two hypotheses on combining modalities
The combination of human output channels effectively increases the bandwidth of the humanmachine channel.
This has been discovered in many empirical studies of multimodal human-computer interaction [Oviatt, 1999b].

17. Two hypotheses on combining modalities
Adding extra output modality requires more neurocomputational resources and will lead to deteriorated output quality, resulting in reduced effective bandwidth.
Two types of effects are usually observed:
a slow-down of all output processes, and
interference errors due to the fact that selective attention cannot be divided between the increased number of output channels.
Two examples of this: writing when speaking, and speaking when driving a car.

18. Call for research
A summary in [Raisamo, 1999e] pointed out that more research is needed to understand the following:
How the brain works and which modalities can best be used to gain the synergy advantages that are possible with multimodal interaction?
When a multimodal system is preferred to a unimodal system?
Which modalities make up the best combination for a given interaction task?
Which interaction devices to assign to these modalities in a given computing system?
How to use these interaction devices, that is, which interaction techniques to select or develop for a given task?

19. Touch’n’Speak [Raisamo, 1998]
Touch’n’Speak is a multimodal user interface framework that makes use of combined touch and speech input and different output modalities
Input: touch buttons, touch lists, touch gestures in area selection (time, location, pressure), speech commands
Output: graphical, textual, and auditory (non-speech) output, speech feedback
The framework was used to implement a restaurant information system that provides information on restaurants in Cambridge, MA, USA.

20. A snapshot of Touch’n’Speak

21. Examples
CHI2000 Video Proceedings: The Efficiency of Multimodal Interaction for a Map-Based Task (8:18)
SIGGRAPH Video Review 76, CHI’92 Technical Video Program: Multi-Modal Natural Dialogue (10:25)
SIGGRAPH Video Review 77, CHI’92 Technical Video Program: Combining Gestures and Direct Manipulation (9:56)
CHI’99 Video Proceedings: Embodiment in Conversational Interfaces: Rea (2:08)

22. Homework Read Chapter 2 (Multimodal interaction) in [Raisamo, 1999e].
[Raisamo, 1999e] is available online at
A printable version is available online at