CoGenTex, Inc. Ontology-based Multimodal User Interface in MOQA AQUAINT 18-Month Workshop San Diego, California Tanya Korelsky Benoit Lavoie Ted Caldwell

CoGenTex, Inc. Goals of MOQA Interface To enable the analyst in ergonomic browsing of an information-rich, ontology based repository To facilitate justification of search results and production of session reports

CoGenTex, Inc. Interface Features Support for both natural language (NL) queries and structured queries used for validation of NL queries Intuitive web-based multimodal interface for answers (search results) Tables, text, maps, timeline, and social network graphs interconnected by hyperlinks Natural language generation used in answers, answer justification, and for query validation Implemented using XML-based technology

CoGenTex, Inc. MOQA User Interface: Accomplishments since December 2002 Workshop Implemented all features shown in mockup at December 2002 Workshop Incorporated all the feedback given at Workshop All the modalities in place, including integrated third party software, MapObjects from ESRI and Graph Editor Toolkit from Tom Sawyer Software. Fully integrated with the MOQA Fact Repository and working as a live system over the Internet and at the sites

CoGenTex, Inc. Benefits of Textual Summaries Textual descriptions complement graphics and tables by –highlighting salient facts and events –clustering data in meaningful ways Hypertext enables web-based data exploration via “drilldown” Text plays a crucial role in –search results justification when search involves inference –session report production

CoGenTex, Inc. CoGenTex’s NLG Expertise R&D for MOQA builds on and extends capabilities developed in previous efforts: –Web-based multimodal presentations which include automatically generated summaries –Generic text plans which support portability between subject domains –Explanation of expert system reasoning

CoGenTex, Inc. Query Interface Integrates with the NL question processor, which builds Text Meaning Representations (TMRs). Produces a corresponding structured query, to validate the NL question, and automatically paraphrases it in a disambiguated form of language. Paraphrases are a crucial feature of the planned ontology-based query clarification and repair dialogs

CoGenTex, Inc. Search Page with NL Query:

CoGenTex, Inc. (REQUEST-INFO-1007 (THEME (VALUE (COMMON MEET-WITH THEME))) (INSTANCE-OF (VALUE (COMMON REQUEST-INFO)))) (MEET-WITH-1007 (TIME (VALUE (COMMON DATE-1007 (< (FIND-ANCHOR-TIME *PAST-EVENT*))))) (AGENT (VALUE (COMMON HUMAN-1007))) (INSTANCE-OF (VALUE (COMMON MEET-WITH)))) (DATE-1007 (MONTH (VALUE (COMMON 1))) (YEAR (VALUE (COMMON 2003))) (INSTANCE-OF (VALUE (COMMON DATE)))) (HUMAN-1007 (AGENT-OF (VALUE (COMMON MEET-WITH-1007))) (OBJECT-NAME (VALUE (COMMON "Jalal Talabani"))) (INSTANCE-OF (VALUE (COMMON HUMAN)))) TMR for NL Query

CoGenTex, Inc. Underlying XML Query Use standard XML technology (browser, parsers, etc.) Support for : –Modularity: XML representation is viewable and exchangeable between subsystems –Automatic validation of query instances using a query class hierarchy described by XML schemas –Complex queries via use of logical expressions in XML

CoGenTex, Inc. Validation of NL Query

CoGenTex, Inc. Results Page Highlights Multimodal display with table, timeline, text, map, and relationship diagram Hyperlinks for drilldown Textual summaries highlight patterns in results (prompts for sorting/ filtering table)

CoGenTex, Inc. Query Results Page for Previous Query

CoGenTex, Inc. Person Details Display

CoGenTex, Inc. Expanded Relationship Graph

CoGenTex, Inc. Details Page with Arabic Alias

CoGenTex, Inc. Excerpt from Arabic Source

CoGenTex, Inc. Source Arabic Document

CoGenTex, Inc. Session History Captures sequence of queries and results Analyst can reorder/delete items to retain most important info Reports can be exported to text editor

CoGenTex, Inc. Session History Page

CoGenTex, Inc. R&D Plans for next 6 months Presentation of ontology-based inference for result justification Intelligent maintenance of session history and retrieved material to facilitate report production Presentation of search results for partially understood natural language queries Exploration of interface portability between subject domains based on automatic generation of interface core from the subject domain ontology

CoGenTex, Inc.

MOQA User Interface: Validation Validation by domain experts is crucial for producing an intelligent and ergonomic question answering interface –Highly interconnected multimodality offers new views on data and methods of browsing –Feedback is crucial for the search results justification for session report production

CoGenTex, Inc. Thank you!