U1, Speech in the interface: 1. Introduction1 Module u1: Speech in the Interface 1: Introduction Jacques Terken HG room 2:40 tel. (247) 5254

U1, Speech in the interface: 1. Introduction2 contents n 1. Aims and overview of course n 2. Speech interfaces n 3. Usability issues: introduction n 4. Project

U1, Speech in the interface: 1. Introduction3 Aims n Acquire insight into usability issues and obtain an overview of state of the art for speech in the interface n Obtain hands-on experience with design of speech-centric interface n Exercise project skills (organisation, collaboration, report, presentation)

U1, Speech in the interface: 1. Introduction4 Overview of Module n Introduction n Dialog management n Speech input technologies n Speech output technologies n Multimodal interaction n Evaluation n Human Communication n Exercises and project

U1, Speech in the interface: 1. Introduction5 contents n 1. Aims and overview of course n 2. Speech interfaces n 3. Usability issues: introduction n 4. Project

U1, Speech in the interface: 1. Introduction6 Speech in the interface Non- Interactive OnlineMonitoring speech communications, Live speech processing Dialogue systems OfflineSpeech data- mining X

U1, Speech in the interface: 1. Introduction7 Markets and applications  R. Moore 2005

U1, Speech in the interface: 1. Introduction8 Speech interfaces n Conversational interfaces: natural language interaction with machines (Star Trek syndrome) n Command & Control applications: voice-based equivalent of command-line interfaces and button interfaces (utterances need to adhere to strict grammar)

U1, Speech in the interface: 1. Introduction9 Components of conversational interfaces Speech recognition Natural Language Analysis Dialogue Manager Speech Synthesis Language Generation Application

U1, Speech in the interface: 1. Introduction10 Spin-offs Speech recognition Natural Language Analysis Dialogue Manager Speech Synthesis Language Generation Application (e.g. MS-Word) n 1. Dictation systems: what you say

U1, Speech in the interface: 1. Introduction11 n 2. Command-control: what you mean Speech recognition (Natural Language) Analysis Dialogue Manager Speech Synthesis Language Generation Application (e.g. stereo)

U1, Speech in the interface: 1. Introduction12 n 3. Text-to-speech conversion Speech recognition Natural Language Analysis Dialogue Manager Speech Synthesis LangGeneration: prosody Application (e.g. )

U1, Speech in the interface: 1. Introduction13 contents n 1. Aims and overview of course n 2. Speech interfaces n 3. Usability issues: introduction n 4. Project

U1, Speech in the interface: 1. Introduction14 Speech in HCI: “yes please” n Among others Zue (MIT): Speech will be key technology of the 21 st century

U1, Speech in the interface: 1. Introduction15 Background n Zue c.s.: –Aim: developing the conversational interface –Motivation: natural language interaction is the most natural form of communication (learned at a very early age); among other things very efficient error handling

U1, Speech in the interface: 1. Introduction16 Advantages of speech n direct access to functionality n supports mobility n suited for hands busy/dirty - eyes busy situations n no special motor abilities needed, optimal compatibility with communicative abilities of users n compatible with trend towards miniaturisation of equipment

U1, Speech in the interface: 1. Introduction17 Maturity hypothesis n Speech interfaces not yet mature because of complexity of technology: –R.K. Moore: “Spoken language interaction is the most sophisticated behaviour of the most complex organism in the known universe”

U1, Speech in the interface: 1. Introduction18 Phylogenetic argumentation n First: direct manipulation (“you do what i want”) n Later: symbolic manipulation (cf. management, commercials) n Physical manipulation and violence considered primitive

U1, Speech in the interface: 1. Introduction19 Ontogenetic argumentation n Russian educational psychology (Galperin): –knowledge acquisition starts with direct manipulation –later-on symbolic manipulation n ”stay off” warning to children: “look with your eyes not with your hands”

U1, Speech in the interface: 1. Introduction20 Therefore n Direct manipulation phylogenetically and ontogenetically more primitive and less complex n Maturity hypothesis: same trajectory for HCI: first direct manipulation then symbolic manipulation (speech)

U1, Speech in the interface: 1. Introduction21 However n UI design principles (Schneiderman ‘86): –transparency: continuous representation of objects and actions –fast, incremental and reversible operations with immediate effect –physical actions or labelled buttons, avoid complex syntax/natural language as much as possible n Design principles difficult to realise in speech interfaces

U1, Speech in the interface: 1. Introduction22 n In addition, language and speech technology is not (yet) very robust, and development costs are high n Getting towards the application semantics is more complicated for (natural) language than for direct manipulation n Finally: HCI is domain in its own right, so there is no a priori reason to model HCI after HHI n SO: avoid natural language

U1, Speech in the interface: 1. Introduction23 Speech interfaces: yes or no n Speech not suited for all kinds of information or situations (e.g. “a picture is worth a thousand words”) n Nevertheless, speech is useful under certain conditions, e.g. –hands busy - eyes busy –mobility, miniaturisation –disabilities (CTS/RSI!)

U1, Speech in the interface: 1. Introduction24 n use interface design guidelines for design of speech interfaces e.g. n and in return: offer human communication theory as model for HCI

U1, Speech in the interface: 1. Introduction25 Speech interfaces (SI) and Direct- manipulation interfaces n Main problems with speech interfaces: –no external support for functionality –unreliability of input technology

U1, Speech in the interface: 1. Introduction26 Dealing with unreliability n Constrain domain –restricted vocabulary –restricted application / task domain –restricted number of users: speaker-dependent speech recognition n Extensive verification (in connection with error cost)

U1, Speech in the interface: 1. Introduction27 Dealing with functionality problem n Quick reference card n Training n System-driven dialogue  experience  need for adaptive systems (e.g. barge-in)

U1, Speech in the interface: 1. Introduction28 contents n 1. Aims and overview of course n 2. Speech interfaces n 3. Usability issues: introduction n 4. Project

U1, Speech in the interface: 1. Introduction29 Aim n Provide hands-on experience with design and implementation of a speech-centric interface, involving (at least) voice-based control and speech output. n The topic: speech/multimodal interface for in-car information and entertainment systems.

U1, Speech in the interface: 1. Introduction30 Tools n Download CSLU toolkit from (requires registering)

U1, Speech in the interface: 1. Introduction31 Project stages n Task analysis (requirements gathering) n Design on paper (V0.1) n Wizard of Oz n Redesign, implementation of V1.0 n Validation n Evaluation n Report

U1, Speech in the interface: 1. Introduction32 Exercise for today n CSLU Exercises: McTear ch. 7, pizza application n Extend the pizza application: –Goto –Click “online bestellen” –Extend the dialogue system to include all the topping options, the side dishes and the drinks (see “menukaart”) –Test the system and discuss your experiences

U1, Speech in the interface: 1. Introduction33 Composition of project teams