1. Forschungszentrum Telekommunikation Wien [Telecommunications Research Center Vienna] Interfaces between Speech and Non-Speech Audio Technology Michael Pucher (FTW Vienna, ICSI Berkeley)

2. © ftw. 2005 Contents  Text-to-Speech Synthesis (TTS)  Automatic Speech Recognition (ASR, STT)  Dialog Systems  Multimodal Mobile Applications  Resources

3. © ftw. 2005 Auditory representations Affective states and attitudes Speaker characteristics Structural prosodic elements Pragmatics and discourse Sound signals Music Perspectival, spatial cues Non-linguistic Paralinguistic Linguistic Lexical semantics and syntax TTS ASR Dialog Systems

4. © ftw. 2005 TTS Examples  16kHz natural voice  16kHz unit selection synthesis (server-based)  8kHz diphone-based synthesis with lexicon (embedded or distributed)  8kHz diphone-based synthesis without lexicon (embedded)  Application specific lexicon tSE-r6ld a:R fo:rd -Gerald R. Ford  tSE-r6ld a:R fo:rd

5. © ftw. 2005 TTS Evaluation

6. © ftw. 2005 TTS and Non-Speech Audio TTSFEATURESTATUS COMBINATION WITH NON- SPEECH AUDIO Comprehensible TTS Low word- error-rate Solved Diphone based TTS TTS provides lexical information Add structural prosodic elements Natural TTS Single style prosody Solved Unit selection TTS provides structural prosodic elements Add affective states and attitudes Expressive TTS Various prosodic styles Not solved ? Add pragmatic information, dialog turns

7. © ftw. 2005 Limited Expressiveness of Speech 1  Limited expressiveness of Expressive TTS = Limited expressiveness of speech  Limited expressiveness of speech because of unlimited expressiveness 1 of speech -Because everything is expressible in language, the messages are less useful for certain purposes (too complex) -Simpler, less expressive codes (sounds, icons) may be used in context and lead to shorter messages  Disadvantages of speech -Seriality -Non-universality

8. © ftw. 2005 Types of ASR and Applications  Isolated word recogniton  Large vocabulary Speech recognition  Conversational Speech recognition  Speech Recognition in noisy environments Car navigation Meeting transcription Command & control Broadcast news transcription Speaker dependent or speaker independent

9. © ftw. 2005 Other Related Technologies  Speech -Speaker verification  NLP -Dialog act detection -Topic detection

10. © ftw. 2005 Music Information Retrieval (MIR)  Query By Humming (Fraunhofer) -Non-speech sound as an input pattern to search for other non-speech sounds -http://www.musicline.de/de/melodiesuche/inputhttp://www.musicline.de/de/melodiesuche/input  Performer Style Identification  Melody and Rhythm Extraction  Music Similarity  Genre Classification

11. © ftw. 2005 Dialog Systems - ASR  3 Types of Recognition in state- of-the-art Dialog Systems -Isolated word -Recognition grammar -Statistical Language Model (SLM) + grammar for more robustness move forward backward exit quit „um ah to san francisco from new york“ 1. Apply SLM 2. Apply grammar on results of SLM

12. © ftw. 2005 Dialog Systems – TTS and Audio  Loquendo TTS Mixer -Play and mix TTS and audio files -Fadein, fadeout -Pause and resume -Record Paolo Massimino : Loquendo S.p.A. From Marked Text to Mixed Speech and Sound

13. © ftw. 2005 Dialog Management 1  Usages of non-speech audio -Replace prompts -Indicate dialog turns and dialog states -Indicate menu structure (3Daudio) -Create listen & feel of the application -System response time  Questions -Bargein, Streaming and Standardization

14. © ftw. 2005 Dialog Management 2  A good bad example -Uses only speech -Audio enhancement for transitions -Audio enhancement for states Bob Cooper : Avaya Corporation A Case Study on the planned and actual Use of Auditory Feedback and Audio Cues in the Realization of a Personal Virtual Assistant

15. © ftw. 2005 Dialog Managment 3  VoiceXML Version 2.0 -W3C (Word Wide Web Consortium) standard for voice dialog design -Form filling paradigm similar to web forms  Synthesis Markup Language (SSML) Version 1.0 good morning Any female voice here. A female child voice here.

16. © ftw. 2005 Limited Expressiveness of Speech 2  Limited expressiveness of human-machine voice dialog compared to a natural dialog -Natural dialog is probable multimodal -Role of non-speech sound in human communication

17. © ftw. 2005 The Importance of Multimodality for Mobile Applications  Multimodal communication is perceived as natural  Disadvantages of unimodal interfaces for mobile devices -Small displays -No comfortable alphanumeric keyboards -Visual access to the display is not always possible  Disadvantages cannot be overcome by increasing processor and memory capabilities

18. © ftw. 2005 Multimodal Dialog Managment  Speech Application Language Tags (http://www.saltforum.org)  Possible combination with non-speech audio at all states and transitions  Similar to (unimodal) dialog systems Minhua Ma : University of Ulster Paul Mc Kevitt : University of Ulster Lexical Semantics and Auditory Display in Virtual Storytelling

19. © ftw. 2005 Asymmetric Multimodality  For Multiparty applications -Users select preferred modalities (e.g. speech, visual, music?) -System is able to translate content from one modality to another  MONA – Mobile Multimodal Next Generation Applications -Multiuser quiz application InputOutput Preference=SpeechOutput Preference=Visual Speech Speech-To-Text TextText-To-SpeechText

20. © ftw. 2005 Resources  TTS -Festival 2.0, to build unit selection voices -Festival Lite, for embedded TTS -FreeTTS, a Java speech synthesizer -The Mbrola project, many synthetic voices available  ASR -Sphinx -Htk  Multimodal Systems -SALT implementations

21. © ftw. 2005 Thank you for your attention Contact: pucher@ftw.at http://userver.ftw.at/~pucher