1. Spoken Dialogue Systems and the GALAXY Architecture 29 October 2000 Advanced Technology Laboratories 1 Federal Street A&E Building 2W Camden, New Jersey 08102 Jerry Franke Senior Member, Engineering Staff jfranke@atl.lmco.com 856.338.3341

2. September 99 Talk Outline Spoken Language DevelopmentSpoken Language Development GALAXY II SystemGALAXY II System SUMMIT (speech recognition)SUMMIT (speech recognition) TINA (natural language parsing)TINA (natural language parsing) GENESIS (natural language generation)GENESIS (natural language generation) Turn ManagementTurn Management GALAXY II DomainsGALAXY II Domains Overview/DemonstrationOverview/Demonstration

3. September 99 Spoken Language Development UniversitiesUniversities –MIT, CMU, Colorado Basic research labsBasic research labs –ATT, SRI Software developersSoftware developers –Nuance, SpeechWorks Domain developersDomain developers –Lockheed Martin - ATL

4. September 99 GALAXY II System Developed by MIT Spoken Language Systems groupDeveloped by MIT Spoken Language Systems group Multiple servers performing parts of the dialogue processMultiple servers performing parts of the dialogue process Speech Recognition Language Understanding Turn Management Language Generation Speech Synthesis SoundSound SoundSound SUMMITTINAGENESIS

5. September 99 SUMMIT (speech recognition) Three elements: vocabulary, language models, acoustic modelsThree elements: vocabulary, language models, acoustic models Pause words stripped outPause words stripped out AM: segment-based models and boundary-based diphone modelsAM: segment-based models and boundary-based diphone models LM: Forward Viterbi search with a class bigram model, followed by a backward A* search with a class trigram modelLM: Forward Viterbi search with a class bigram model, followed by a backward A* search with a class trigram model Produces N-best list or word graph of possible utterancesProduces N-best list or word graph of possible utterances Models trained on domain corpusModels trained on domain corpus Models achieve speaker-independenceModels achieve speaker-independence

6. September 99 TINA (natural language parsing) Selects from N-best list depending on grammatical parseSelects from N-best list depending on grammatical parse Grammars reflect both syntactic and semantic structureGrammars reflect both syntactic and semantic structure Result is a semantic frameResult is a semantic frame Example:Example: “Where is the library in Swain Hall?” Clause: LOCATE Topic: PUBLIC-BUILDING Quantifier: DEF Name: library Predicate: IN Topic: HALL Name: Swain

7. September 99 GENESIS (natural language generation) Processes semantic framesProcesses semantic frames Embeds semantic frame components into context-dependent message templatesEmbeds semantic frame components into context-dependent message templates Two types of output:Two types of output: –natural language messages –messages are sent to some speech synthesis module –possibility of output in multiple languages –keyword-value pairs –useful structure for the turn management backend Can be used to map between (translate) languagesCan be used to map between (translate) languages

8. September 99 Turn Management Manages the system’s part of the dialogueManages the system’s part of the dialogue Fuses current utterance with dialogue history for full contextFuses current utterance with dialogue history for full context Five main tasks:Five main tasks: –Answer user’s requests (information retrieval) –Initiate sub-dialogues to clarify the user’s request –Track progress through the dialogue –Control response to the user –Provide assistance in using the system when needed

9. September 99 GALAXY II Domains From MIT: –Jupiter - weather forecasts –Pegasus - airline scheduling –Voyager - Cambridge, Massachusetts city guide –Dinex - Boston restaurant guide –Wheels - automobile classified ads –Mercury - airline flight booking From Lockheed Martin - ATL: –DARPA Communicator - airline flight, hotel, car rental booking –DARPA LCS-Marine/Marine Small Unit Logistics - supply request –DARPA LCS-Army - data collection during equipment tests

10. September 99 Overview/Demonstration Booking airline flights (round trip and one way)Booking airline flights (round trip and one way) Hotel, car rentalHotel, car rental Uses user profile to fill in some information about reservationUses user profile to fill in some information about reservation Uses some real-world knowledgeUses some real-world knowledge Information retrieval via mobile agentsInformation retrieval via mobile agents

11. September 99 Try It Yourself CMU 1-412-268-1084CMU 1-412-268-1084 MIT 1-877-527-8255MIT 1-877-527-8255 Colorado http://communicator.colorado.edu/Colorado http://communicator.colorado.edu/