1. Spoken Dialogue Technology Achievements and Challenges
Michael McTear
University of Ulster

2. Overview Introduction - What is a spoken dialogue system?
Examples of spoken dialogue systems
Technical issues and challenges
Future Prospects

3. What is a spoken dialogue system?
A spoken dialogue system is an automated system that engages in a dialogue with a human user using spoken language as the medium of interaction.

4. Types of dialogue system
Two main types of spoken dialogue system
Task-oriented: involves the use of dialogues to accomplish a task, e.g. making a hotel booking, or planning a family holiday
Non-task-oriented: engaging in conversational interaction, but without necessarily being involved in a task that needs to be accomplished e.g conversational companion for the elderly

5. Application Domains for SDS
Telephone-based services and transactions
Call-routing, Directory assistance, Travel enquiries, Bank balance, Bank transactions, Flight / hotel / car rental reservations
In-car interactive and entertainment systems
Automated trouble-shooting
Smart homes applications
Health-care systems e.g. patient monitoring
Educational e,g. Intelligent Tutoring Systems, Foreign Language Learning
Computer games

6. Three generations of task-oriented spoken dialogue system
Informational – to retrieve information e.g. flight times, football scores, …
Transactional – to assist the user to perform a transaction e.g. book a flight, pay a bill
Problem-solving – to support the user in solving a problem e.g. to troubleshoot a PC that is not working

7. Why is dialogue interesting?
Fundamental aspect of human behaviour
Model human conversational competence
Simulate human conversational behaviour
Provide tool for interacting with data, services, resources on computers
Research challenges
Applications in assistive and educational environments
Commercial opportunities

8. Commercial Systems Focus on
Business opportunities, return on investment (ROI)
Benefits for end users
Benefits for providers
Human factors: performance, usability
Tools and languages for design and maintainability
Application areas: call centre, enquiries, transactions, healthcare, …

9. Academic Systems Focus on
Technologies: speech recognition, spoken language understanding, dialogue management
AI inspired: planning, reasoning, machine learning
Statistical v symbolic approaches
Advanced dialogue control, error handling, adaptivity, context representation

10. Overview Introduction - What is a spoken dialogue system?
Examples of spoken dialogue systems
Technical issues and challenges
Future Prospects

11. Example 1: Voice Menu System: Hello and welcome ….
Main menu. For customer service, say ‘service’.
To enquire about an existing order, say ‘order’ …
User: Service
System: Customer service. Would you like to report a fault or enquire about an extended warranty?
User: Fault
System: Do you have a PC or a laptop?
User: Laptop
System: And the name of the manufacturer?
User: Sony
System: Thank you. Please hold while I transfer you to the Sony …

12. Example 2: Research System (Mercury: MIT)
Open ended prompt
How may I help you?
Disfluencies in input
August twenty-first no August twelfth
I'd like to fly from Boston to Minneapolis on Tuesday no Wednesday November 21st
Inexact response
Prompt: Can you provide the approximate departure time or airline preference
User: Yeah I'd like to fly United and I'd like to leave in the afternoon

13. Example 2: continued Response generation
There are more than 3 flights.
The earliest departure leaves at 1.45 pm.
Mixed initiative: user asks question
Do you have something leaving around 4.45?
Relative date reference
I’d like to return the following Tuesday

14. Example 3: Voice Search GOOG411
GOOG-411 (or Google Voice Local Search) is Google's
new 411 service.
With GOOG-411, you can find local business information
completely free, directly from your phone.
You can access GOOG-411 from any phone,
anywhere, at anytime.

15. GOOG411: Prompts What city and state? What business name or category?
(Lists services) Number one, …..
Connects to requested service

16. GOOG411: What can you say? At any point in the call:
To go back say "go back"
To start over say "start over" or press *All phones
When asked for a city and state: 
Say the full names for example, "Palo Alto California“
To enter a zip code say it or enter with keypad
When asked for business name or category: 
Say the full names for example, "Joe's Pizzaria" or "Pizza“
When given results: 
To navigate between results say or press the listing number
To receive an SMS say "text message"
To receive a map say "map it"
To get more details say "details"

17. Overview Introduction - What is a spoken dialogue system?
Examples of spoken dialogue systems
Technical issues and challenges
Future Prospects

18. Architecture of a spoken dialogue system
Speech
Recognition
(ASR)
Back
end
Response
Generation
Text to Speech
Synthesis
(TTS)
a --> xu
Spoken
Language
Understanding
(SLU)
yu, c
ã, c
Concepts
Words
Audio
HMM
Acoustic
Model
N-Gram
Dialogue Manager (DM)
Dialogue
Control
Context Model
a user dialogue act (intended ) c confidence
ã user dialogue act (interpreted)
xu user acoustic signal
yu speech recognition hypothesis (words)

19. Component Technologies
Automatic Speech Recognition (ASR)
Spoken Language Understanding (SLU)
Response Generation (RG)
Text to speech synthesis (TTS)
Dialogue Management (DM)

20. Issues in ASR for Dialogue
recognising spontaneous speech in noisy environments
word accuracy does not have to be 100%
use of confidence scores in combination with other information to determine DM actions
use of additional information (ASR and parse probabilities, semantic and contextual features) to re-score recognition hypotheses

21. Issues in SLU for Dialogue
grammars and parsers for spontaneous speech (disfluencies, errors)
robust understanding
problems with hand-crafted approaches
use of statistical/ data-driven methods
combined approaches e.g TINA (MIT)
hand-crafted rules with trained probabilities
robust strategy – if full sentence cannot be parsed, parse and combine fragments, else use word spotting

22. Issues in Response Generation for Dialogue
Content selection
Determining what to say, selecting and ranking options
Discourse planning
discourse relations e.g. comparison, contrast
user-adapted information
Presentation ordering
Referring expression generation
Aggregation – grouping propositions into clauses and sentences
Use of discourse cues (e.g. firstly, finally, however, moreover, …)

23. Issues in Dialogue Management
Dialogue Control
Scripts, frames, intelligent agents
Representations
Information State Theory
Error handling
Dialogue design
Traditional approaches
Statistical approaches
Reinforcement learning
Corpus / example based approaches

24. Overview Introduction - What is a spoken dialogue system?
Examples of spoken dialogue systems
Technical issues and challenges
Future Prospects

25. A vision for the future
Develop systems that can interact intelligently and co-operatively across a range of environments using a range of appropriate modalities to support people in the activities of their daily lives.

26. Fundamental research topics
Modelling human conversational competence
Dialogue-related issues for ASR, SLU, NLG, TTS
Comparison of methods for dialogue management: rule-based v stochastic
Representation and use of contextual information
Integration and usage of modalities to complement and supplement speech
Incremental processing in dialogue

27. Areas of application Voice search Dialogue in vehicles
Mobile speech applications
Multimodal embodied and situated systems
Troubleshooting applications
Dialogue systems for ambient intelligence and as assistive technologies

28. Concluding remarks Spoken Dialogue Technology
embraces a range of speech and language technologies
poses lots of theoretical as well as practical challenges
is interesting for commercial developers as well as academic researchers
has a wide range of potential applications

29. Recommended reading
McTear, M. (2004) Spoken Dialogue Technology. Springer.
Lopez Cozar, R. & Araki, M. (2005) Spoken, multilingual and multimodal dialogue systems. John Wiley & Sons.
Aghajan, H., Augusto, J.C., Lopez Cozar, R. (2009) Human-Centric Interfaces for Ambient Intelligence. Elsevier.
Jokinen, K. & McTear, M. (2010) Spoken Dialogue Systems. Morgan Claypool Publishers.
Wilks, Y. (ed.) (2010) Close Engagements with Artificial Companions: Key social, psychological, ethical and design issues. John Benjamins Publishing Company.

30. Thank you
Questions?