1. Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI GmbH Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302-5252/4162 fax: (+49 681) 302-5341 e-mail: wahlster@dfki.de WWW:http://www.dfki.de/~wahlster Verbmobil Multilingual Processing of Spontaneous Speech

2. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Mobile Speech-to-Speech Translation of Spontaneous Dialogs As the name Verbmobil suggests, the system supports verbal communication with foreign dialog partners in mobile situations. 1 2 face-to-face conversations telecommunication

3. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Mobile Speech-to-Speech Translation of Spontaneous Dialogs Verbmobil Speech Translation Server Solution: Conference Call: The Verbmobil Speech Translation Server is accessed by GSM mobile phones.

4. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil is a Multilingual System German English (American) German Japanese It supports bidirectional translation between: German Chinese (Mandarine) Siemens, Philips, FH Konstanz, 2 Chinese Universities Final industrial demo at the end of 2000

5. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Input Conditions Naturalness Adaptability Dialog Capabilities Increasing Complexity Close-Speaking Microphone/Headset Push-to-talk Telephone, Pause-based Segmentation Isolated Words Read Continuous Speech Speaker Independent Speaker Dependent Monolog Dictation Information- seeking Dialog Open Microphone, GSM Quality Spontaneous Speech Speaker adaptive Multiparty Negotiation Verbmobil Challenges for Language Engineering

6. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Wann fährt der nächste Zug nach Hamburg ab? When does the next train to Hamburg depart? Wo befindet sich das nächste Hotel? Where is the nearest hotel? Final Verbmobil Demos: CeBIT-2000 (Hannover) COLING-2000 (Saarbrücken) ECAI-2000 (Berlin) Context-Sensitive Speech-to-Speech Translation Verbmobil Server

7. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing)

8. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911

9. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911 Verbmobil: Willkommen beim Verbmobil-Sprachserver. Bitte sprechen sie nach dem Piepton.

10. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911 Verbmobil: Willkommen beim Verbmobil-Sprachserver. Bitte sprechen sie nach dem Piepton. German Speaker:Verbmobil neuer Teilnehmer hinzufügen. (Speech command to initiate a conference call)

11. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911 Verbmobil: Willkommen beim Verbmobil-Sprachserver. Bitte sprechen sie nach dem Piepton. German Speaker:Verbmobil neuer Teilnehmer hinzufügen. (Speech command to initiate a conference call) Verbmobil: Bitte sprechen Sie jetzt die Telephonnummer Ihres Gesprächspartners.

12. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911 Verbmobil: Willkommen beim Verbmobil-Sprachserver. Bitte sprechen sie nach dem Piepton. German Speaker:Verbmobil neuer Teilnehmer hinzufügen. (Speech command to initiate a conference call) Verbmobil: Bitte sprechen Sie jetzt die Telephonnummer Ihres Gesprächspartners. German Speaker: 0681/302 5253

13. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil: The First Speech-Only Dialog Translation System Mobile GSM Phone Mobile DECT Phone German Speaker: Verbmobil (Voice Dialing) Connect to the Verbmobil Speech-to-Speech Translation Server +49 631 3111911 Verbmobil: Willkommen beim Verbmobil-Sprachserver. Bitte sprechen sie nach dem Piepton German Speaker:Verbmobil neuer Teilnehmer hinzufügen (Speech command to initiate a conference call) Verbmobil: Bitte sprechen Sie jetzt die Telephonnummer Ihres Gesprächspartners. German Speaker: 0681/302 5253 Foreign Participant is placed into the Conference Call To German Participant Verbmobil: Verbmobil hat eine neue Verbindung aufgebaut. Bitte sprechen Sie jetzt. To American Participant Verbmobil: Welcome to the Verbmobil server. Please start your input after the beep.

14. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Scenario 1 Appointment Scheduling When? When? Where? How? What? When? Where? How? Focus on temporal expressions Focus on temporal and spatial expressions Integration of special sublanguage lexica Vocabulary Size: 2500/6000 Vocabulary Size: 7000/10000 Vocabulary Size: 15000/30000 Verbmobil II: Three Domains of Discourse Scenario 2 Travel Planning Scenario 3 Remote PC Maintenance

15. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH The Control Panel of Verbmobil

16. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH M1 M2M3 M5 M6 M4 BB 2BB 1 BB 3 M1 Verbmobil I Verbmobil II Multi-Agent Architecture Multi-Blackboard Architecture Each module must know, which module produces what data Direct communication between modules Each module has only one instance Heavy data traffic for moving copies around Multiparty and telecooperation applications are impossible Software: ICE and ICE Master Basic Platform: PVM All modules can register for each blackboard dynamically No direct communication between modules Each module can have several instances No copies of representation structures (word lattice, VIT chart) Multiparty and Telecooperation applications are possible Software: PCA and Module Manager Basic Platform: PVM From a Multi-Agent Architecture to a Multi- Blackboard Architecture Blackboards M2 M3 M6 M4 M5

17. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Audio Data Command Recognizer Spontaneous Speech Recognizer Channel/Speaker Adaptation Prosodic Analysis A Multi-Blackboard Architecture for the Combination of Results from Deep and Shallow Processing Modules

18. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Audio Data Word Hypotheses Graph with Prosodic Labels Command Recognizer Spontaneous Speech Recognizer Channel/Speaker Adaptation Prosodic Analysis Statistical Parser Dialog Act Recognition Chunk Parser HPSG Parser A Multi-Blackboard Architecture for the Combination of Results from Deep and Shallow Processing Modules

19. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Audio Data Word Hypotheses Graph with Prosodic Labels VITs Underspecified Discourse Representations Command Recognizer Spontaneous Speech Recognizer Channel/Speaker Adaptation Prosodic Analysis Statistical Parser Dialog Act Recognition Chunk Parser HPSG Parser Semantic Construction Robust Dialog Semantics Semantic Transfer Generation A Multi-Blackboard Architecture for the Combination of Results from Deep and Shallow Processing Modules

20. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil as the First Dialog Translation System that Uses Prosodic Information Systematically at All Processing Stages Speech SignalWord Hypotheses Graph Multilingual Prosody Module Prosodic features: duration pitch energy pause Search Space Restriction Parsing Dialog Act Segmentation and Recognition Dialog Understanding Constraints for Transfer Translation Lexical Choice Generation Speech Synthesis Speaker Adaptation Boundary Information Boundary Information Boundary Information Boundary Information Sentence Mood Sentence Mood Accented Words Accented Words Prosodic Feature Vector

21. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Augmented Word Hypotheses Graph Augmented Word Hypotheses Graph Chunk Parser Statistical Parser HPSG Parser Integrating Shallow and Deep Analysis Components in a Multi-Blackboard Architecture

22. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH partial VITs Chart with a combination of partial VITs Chart with a combination of partial VITs Integrating Shallow and Deep Analysis Components in a Multi-Blackboard Architecture Augmented Word Hypotheses Graph Augmented Word Hypotheses Graph Chunk Parser Statistical Parser HPSG Parser

23. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Robust Dialog Semantics Combination and knowledge- based reconstruction of complete VITs Robust Dialog Semantics Combination and knowledge- based reconstruction of complete VITs Complete and Spanning VITs Complete and Spanning VITs Integrating Shallow and Deep Analysis Components in a Multi-Blackboard Architecture Chunk Parser Statistical Parser HPSG Parser partial VITs Chart with a combination of partial VITs Chart with a combination of partial VITs Augmented Word Hypotheses Graph Augmented Word Hypotheses Graph

24. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobils Massive Data Collection Effort Transliteration Variant 1 Transliteration Variant 2 Lexical Orthography Canonical Pronounciation Manual Phonological Segmentation Automatic Phonological Segmentation Word Segmentation Prosodic Segmentation Dialog Acts Noises Superimposed Speech Syntactic Category Word Category Syntactic Function Prosodic Boundaries The so-called Partitur (German word for musical score) orchestrates fifteen strata of annotations 3,200 dialogs (182 hours) with 1,658 speakers 79,562 turns distributed on 56 CDs, 21.5 GB

25. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Machine Learning for the Integration of Statistical Properties into Symbolic Models for Speech Recognition, Parsing, Dialog Processing, Translation Transcribed Speech Data Segmented Speech with Prosodic Labels Annotated Dialogs with Dialog Acts Treebanks & Predicate- Argument Structures Aligned Bilingual Corpora Hidden Markov Models Neural Nets, Multilayered Perceptrons Probabilistic Automata Probabilistic Grammars Probabilistic Transfer Rules Extracting Statistical Properties from Large Corpora

26. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

27. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Statistical LR parser trained on treebank (Block, Ruland) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

28. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Statistical LR parser trained on treebank (Block, Ruland) Very fast HPSG parser (see two papers at ACL99, Kiefer, Krieger et al.) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

29. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Statistical LR parser trained on treebank (Block, Ruland) Very fast HPSG parser (see two papers at ACL99, Kiefer, Krieger et al.) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

30. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Rule-based combination and transformation of partial UDRS coded as VITs Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Statistical LR parser trained on treebank (Block, Ruland) Very fast HPSG parser (see two papers at ACL99, Kiefer, Krieger et al.) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

31. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Incremental chart construction and anytime processing Rule-based combination and transformation of partial UDRS coded as VITs Selection of a spanning analysis using a bigram model for VITs (trained on a tree bank of 24 k VITs) Chart Parser using cascaded finite-state transducers (Abney, Hinrichs) Statistical LR parser trained on treebank (Block, Ruland) Very fast HPSG parser (see two papers at ACL99, Kiefer, Krieger et al.) Semantic Construction VHG: A Packed Chart Representation of Partial Semantic Representations

32. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Goals of robust semantic processing (Pinkal, Worm, Rupp) Combination of unrelated analysis fragments Completion of incomplete analysis results Skipping of irrelevant fragments Method:Transformation rules on VIT Hypothesis Graph: Conditions on VIT structures Operations on VIT structures The rules are based on various knowledge sources: lattice of semantic types domain ontology sortal restrictions semantic constraints Results: 20% analysis is improved, 0.6% analysis gets worse Robust Dialog Semantics: Deep Processing of Shallow Structures

33. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH The preposition in is missing in all paths through the word hypothesis graph. A temporal NP is transformed into a temporal modifier using a underspecified temporal relation: [temporal_np(V1)] [typeraise_to_mod (V1, V2)] & V2 The modifier is applied to a proposition: [type (V1, prop), type (V2, mod)] [apply (V2, V1, V3)] & V3 Let us meet the late afternoon to catch the train to Frankfurt Let us meet (in) the late afternoon to catch the train to Frankfurt Robust Dialog Semantics: Combining and Completing Partial Representations

34. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH I need a car next Tuesdayoops Monday Original Utterance Editing PhaseRepair Phase Reparandum Hesitation Reparans Recognition of Substitutions Transformation of the Word Hypothesis Graph I need a car next Monday Verbmobil Technology:Understands Speech Repairs and extracts the intended meaning Dictation Systems like: ViaVoice, VoiceXpress, FreeSpeech, Naturally Speaking cannot deal with spontaneous speech and transcribe the corrupted utterances. The Understanding of Spontaneous Speech Repairs

35. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Integrating a Deep HPSG-based Analysis with Probabilistic Dialog Act Recognition for Semantic Transfer Probabilistic Analysis of Dialog Acts (HMM) Probabilistic Analysis of Dialog Acts (HMM) Recognition of Dialog Plans (Plan Operators) Recognition of Dialog Plans (Plan Operators) Dialog Act Type HPSG Analysis Robust Dialog Semantics Robust Dialog Semantics VIT Semantic Transfer Semantic Transfer Dialog Act Type

36. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Probabilistic Analysis of Dialog Acts (HMM) Probabilistic Analysis of Dialog Acts (HMM) Recognition of Dialog Plans (Plan Operators) Recognition of Dialog Plans (Plan Operators) Dialog Act Type Dialog Phase Dialog Act Type Integrating a Deep HPSG-based Analysis with Probabilistic Dialog Act Recognition for Semantic Transfer HPSG Analysis Robust Dialog Semantics Robust Dialog Semantics VIT Semantic Transfer Semantic Transfer

37. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Statistical Prediction Statistical Prediction Context Evaluation Dialog Module Dialog-Act based Translation Plan Recognition Plan Recognition Dialog Memory Dialog Memory Main Proprositional Content Dialog Act Context Evaluation Dialog-Act based Translation Transfer by Rules Generation of Minutes Dialog Act Predictions Dialog Act Dialog Phase Focus Combining Statistical and Symbolic Processing for Dialog Processing

38. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Using Context and World Knowledge for Semantic Transfer All other dialog translation systems translate word-by-word or sentence-by-sentence. 1 Nehmen wir dieses Hotel, ja. Let us take this hotel. Ich reserviere einen Platz. I will reserve a room. 2 Machen wir das Abendessen dort. Let us have dinner there. Ich reserviere einen Platz. I will reserve a table. 3 Gehen wir ins Theater. Let us go to the theater. Ich möchte Plätze reservieren. I would like to reserve seats. Example: Platz room / table / seat

39. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Dialog Translation by Verbmobil Multilingual Generation of Protocols HTML-Document in English Transferred by Internet or Fax HTML-Document in German Transferred by Internet or Fax German Dialog Partner American Dialog Partner Automatic Generation of Multilingual Protocols of Telephone Conversations

40. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Segment 1 If you prefer another hotel, Segment 1 If you prefer another hotel, Segment 2 please let me know. Segment 2 please let me know. Integrating Deep and Shallow Processing: Combining Results from Concurrent Translation Threads

41. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Alternative Translations with Confidence Values Statistical Translation Statistical Translation Dialog-Act Based Translation Dialog-Act Based Translation Semantic Transfer Semantic Transfer Case-Based Translation Case-Based Translation Integrating Deep and Shallow Processing: Combining Results from Concurrent Translation Threads Segment 1 If you prefer another hotel, Segment 1 If you prefer another hotel, Segment 2 please let me know. Segment 2 please let me know.

42. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Integrating Deep and Shallow Processing: Combining Results from Concurrent Translation Threads Segment 1 Translated by Semantic Transfer Segment 1 Translated by Semantic Transfer Segment 2 Translated by Case-Based Translation Segment 2 Translated by Case-Based Translation Alternative Translations with Confidence Values Statistical Translation Statistical Translation Dialog-Act Based Translation Dialog-Act Based Translation Semantic Transfer Semantic Transfer Case-Based Translation Case-Based Translation Segment 1 If you prefer another hotel, Segment 1 If you prefer another hotel, Segment 2 please let me know. Segment 2 please let me know. Selection Module

43. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Funding by the German Ministry for Education and Research BMBF Phase I (1993-1996)$ 33 M Phase II (1997-2000)$ 28 M 60% Industrial funding according to shared cost model$ 17 M Additional R&D investments of industrial partners$ 11 M Total$ 89 M > 800 Publications (>600 refereed) >Many Patents > 17 Commercial Spin-off Products >6 Spin-off Companies > 900 trained Researchers for >Product Announcement German Language Industry for GSM version in 2001 Philips, DaimlerChrysler and Siemens are leaders in Spoken Dialog Applications Verbmobil: Long-Term, Large-Scale Funding and Its Impact

44. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH More than 80% of Verbmobils Translations are Approximately Correct - Large-Scale Web-based Evaluation: 25 345 Translations, 65 Evaluators - Sentence Length 1 - 60 Words Translation Thread Case-based Translation Statistical Translation Dialog-Act based Translation Semantic Transfer Substring-based Translation Automatic Selection Manual Selection 37% 69% 40% 65% 57% / 78% * 88% 44% 79% 45% 47% 75% 66% / 83% * 95% 46% 81% 46% 49% 79% 68% / 85% * 97% Word Accuracy 50% 5069 Turns Word Accuracy 75% 3267 Turns Word Accuracy 80% 2723 Turns * After Training with Instance-based Learning Algorithm Percentage of Approximately Correct Translation

45. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Three Domains: Appointment Scheduling, Travel Planning, PC Hotline Bi-directional and speaker-independent translation in the domains: appointment scheduling and travel planning Translation pairs: German English, German Japanese Vocabulary Size: 10 000 for German, Equivalent English Lexicon, 2500 for Japanese Operational Success Criteria: Word recognition rate (16 kHz): German: spontaneous: 75% (cooperative: 85%) English: spontaneous: 72% (cooperative: 82%) Japanese: spontaneous: 75% (cooperative: 85%) (8kHz) spontaneous: 70% (cooperative: 80%) 80% of the translations are approximately correct and the dialog task success rate should be around 90%. The average end-to-end processing time should be four times real time (length of the input signal) Checklist for Final Verbmobil System I

46. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH The system can work in the open microphone mode and cope with speech over GSM mobile phones. Verbmobil can be controlled by speech commands. A spelling mode is integrated into the speech recognizer. The speech recognizers can cope with simple non-speech input (like coughing). Spontaneous speech phenomena like repairs, hesitations and agreement failures can be handled. The language identification and speech recognition components are implemented as separate components. A three-party conference call with Verbmobil and a foreign partner can be initiated by one speaker. A high-quality speech synthesis for German and American English is realized. Checklist for Final Verbmobil System II

47. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Prosodic information is used for input segmentation. Unknown words can be identified and processed. Robust semantic processing integrates partial analysis results of the competing parsing approaches. The selection of the translation result is based on a dynamic choice function based on confidence values computed by competing translation threads. Some translation ambiguities can be resolved by the exploitation of world and context knowledge, so that the translation quality is improved. Verbmobil can generate various forms of dialog protocols in German and English. Checklist for Final Verbmobil System III

48. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Results of the Verbmobil Project have been used in 17 Spin-Off Products by the Industrial Partners DaimlerChrysler, Philips and Siemens Verbmobil Dictation Systems 3 Spoken Dialog Systems 4 Dialog Engines 2 Command & Control Systems 5 Text Classification Systems 1 Translation Systems 2

49. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil CLT Sprachtechnologie GmbH Language Technology for Text Processing www.clt-st.de Saarbrücken RETIVOX GbR Speech Synthesis Systems www.retivox.de Bonn XTRAMIND Technologies Language Technology for Customer Interaction Services www.xtramind.com Saarbrücken SYMPALOG GmbH Spoken Dialog Systems www.sympalog.de Nürnberg GSDC GmbH Multilingual Documentation www.ic-portal.gsdc.de Nürnberg SCHEMA GmbH Document Engineering www.schema.de Nürnberg Successful Technology Transfer: 6 High-Tec Spin-Off Companies in the Area of Language Technology have been founded by Verbmobil Researchers

50. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil Internships 18 Master Students 238 PhD Students 164 Student Research Assistants 483 Habilitations 16 Total 919 Verbmobil was the Key Resource for the Education and Training of Researchers and Engineers Needed to Build Up Language Industry in Germany

51. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Main Contractor Project Management Testbed Software Integration DFKI Saarbrücken The SmartKom Consortium: Project Budget: $ 34 M Project Duration: 4 years SmartKom: Intuitive Multimodal Interaction MediaInterface European Media Lab IMS Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart Ludwig-Maximilians- Universität München

52. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Smartcard/ Credit Card for authentication and billing Docking station for PDA/Notebook/ Camcorder high speed and broad bandwidth Internet connectivity Loudspeaker Room microphone Face-tracking camera Virtual touchscreen protected against vandalism Multipoint video conferencing High-resolution scanner SmartKom-Public: A Multimodal Communication Booth

53. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Camera GPS Microphone Loudspeaker Stylus-Activated Sketch Pad Wearable Compute Server Docking Station for Car PC Biosensor for Authentication & Emotional Feedback GSM for Telephone, Fax, Internet Connectivity SmartKom-Mobile: A Handheld Communication Assistant

54. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Verbmobil is a Very Large Dialog System 69 modules communicate via 224 blackboards HPSG for German uses a hierarchy of 2,400 types 15,385 entries in the semantic database 22,783 transfer rules and 13,640 microplanning rules 30,000 templates for case-based translation 691,583 alignment templates 334 finite state-transducers

55. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Additional Information about Verbmobil during COLING 2 Tutorials: Klüter/Reithinger: Verbmobil Development and Integration Müller: HPSG 11 Presentations at main conference (regular papers and project notes) - Probabilistic Parsing - Tense Translation - Selection of Translation Results - Statistical Translation (4) - HPSG Parsing - Semantic Construction - Self Corrections Verbmobil Demos at the COLING exhibition 1 2 3

56. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Real-world problems in language technology like the understanding of spoken dialogs, speech-to-speech translation and multimodal dialog systems can only be cracked by the combined muscle of deep and shallow processing approaches. In a multi-blackboard architecture based on packed representations on all processing levels (speech recognition, parsing, semantic processing, translation, generation) using charts with underspecified representations (eg. UDRS) the results of concurrent processing threads can be combined in an incremental fashion. Conclusion I

57. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH All results of concurrent processing modules should come with a confidence value, so that a selection module can choose the most promising result at a each processing stage. Packed representations together with formalisms for underspecification capture the uncertainties in a each processing phase, so that the uncertainties can be reduced by linguistic, discourse and domain constraints as soon as they become applicable. Conclusion II

58. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH Deep Processing can be used for merging, completing and repairing the results of shallow processing strategies. Shallow methods can be used to guide the search in deep processing. Statistical methods must be augmented by symbolic models (eg. Class-based language modelling, word order normalization as part of statistical translation). Statistical methods can be used to learn operators or selection strategies for symbolic processes. It is much more than a balancing act... (see Klavans and Resnik 1996) Conclusion III

59. Verbmobil Final Symposium, 30 July 2000 © Wolfgang Wahlster, DFKI GmbH English speech recognition for telephone input (DaimlerChrysler) Two additional translation engines: case-based (ALI, DFKI) and substring-based translation (LTrans, Siemens) An additional protocol mode (baseline protocol, DFKI) Open Problems: Integrating top-down knowledge into basic speech recognition processes Exploiting more knowledge about human interpretation strategies More robust translation of turns with very low word accuracy rates More systematic use of expert knowledge about the domain of discourse Additional Results (not promised in the project proposal)

60. URL of this Presentation: www.dfki.de/~wahlster/vm-final