1. CoGenTex, Inc. Ontology-based Multimodal User Interface in MOQA AQUAINT 18-Month Workshop San Diego, California Tanya Korelsky Benoit Lavoie Ted Caldwell

2. CoGenTex, Inc. Goals of MOQA Interface To enable the analyst in ergonomic browsing of an information-rich, ontology based repository To facilitate justification of search results and production of session reports

3. CoGenTex, Inc. Interface Features Support for both natural language (NL) queries and structured queries used for validation of NL queries Intuitive web-based multimodal interface for answers (search results) Tables, text, maps, timeline, and social network graphs interconnected by hyperlinks Natural language generation used in answers, answer justification, and for query validation Implemented using XML-based technology

4. CoGenTex, Inc. MOQA User Interface: Accomplishments since December 2002 Workshop Implemented all features shown in mockup at December 2002 Workshop Incorporated all the feedback given at Workshop All the modalities in place, including integrated third party software, MapObjects from ESRI and Graph Editor Toolkit from Tom Sawyer Software. Fully integrated with the MOQA Fact Repository and working as a live system over the Internet and at the sites

5. CoGenTex, Inc. Benefits of Textual Summaries Textual descriptions complement graphics and tables by –highlighting salient facts and events –clustering data in meaningful ways Hypertext enables web-based data exploration via “drilldown” Text plays a crucial role in –search results justification when search involves inference –session report production

6. CoGenTex, Inc. CoGenTex’s NLG Expertise R&D for MOQA builds on and extends capabilities developed in previous efforts: –Web-based multimodal presentations which include automatically generated summaries –Generic text plans which support portability between subject domains –Explanation of expert system reasoning

7. CoGenTex, Inc. Query Interface Integrates with the NL question processor, which builds Text Meaning Representations (TMRs). Produces a corresponding structured query, to validate the NL question, and automatically paraphrases it in a disambiguated form of language. Paraphrases are a crucial feature of the planned ontology-based query clarification and repair dialogs

8. CoGenTex, Inc. Search Page with NL Query:

9. CoGenTex, Inc. (REQUEST-INFO-1007 (THEME (VALUE (COMMON MEET-WITH- 1007.THEME))) (INSTANCE-OF (VALUE (COMMON REQUEST-INFO)))) (MEET-WITH-1007 (TIME (VALUE (COMMON DATE-1007 (< (FIND-ANCHOR-TIME *PAST-EVENT*))))) (AGENT (VALUE (COMMON HUMAN-1007))) (INSTANCE-OF (VALUE (COMMON MEET-WITH)))) (DATE-1007 (MONTH (VALUE (COMMON 1))) (YEAR (VALUE (COMMON 2003))) (INSTANCE-OF (VALUE (COMMON DATE)))) (HUMAN-1007 (AGENT-OF (VALUE (COMMON MEET-WITH-1007))) (OBJECT-NAME (VALUE (COMMON "Jalal Talabani"))) (INSTANCE-OF (VALUE (COMMON HUMAN)))) TMR for NL Query

10. CoGenTex, Inc. Underlying XML Query Use standard XML technology (browser, parsers, etc.) Support for : –Modularity: XML representation is viewable and exchangeable between subsystems –Automatic validation of query instances using a query class hierarchy described by XML schemas –Complex queries via use of logical expressions in XML

11. CoGenTex, Inc. Validation of NL Query

12. CoGenTex, Inc. Results Page Highlights Multimodal display with table, timeline, text, map, and relationship diagram Hyperlinks for drilldown Textual summaries highlight patterns in results (prompts for sorting/ filtering table)

13. CoGenTex, Inc. Query Results Page for Previous Query

14. CoGenTex, Inc. Person Details Display

15. CoGenTex, Inc. Expanded Relationship Graph

16. CoGenTex, Inc. Details Page with Arabic Alias

17. CoGenTex, Inc. Excerpt from Arabic Source

18. CoGenTex, Inc. Source Arabic Document

19. CoGenTex, Inc. Session History Captures sequence of queries and results Analyst can reorder/delete items to retain most important info Reports can be exported to text editor

20. CoGenTex, Inc. Session History Page

21. CoGenTex, Inc. R&D Plans for next 6 months Presentation of ontology-based inference for result justification Intelligent maintenance of session history and retrieved material to facilitate report production Presentation of search results for partially understood natural language queries Exploration of interface portability between subject domains based on automatic generation of interface core from the subject domain ontology

22. CoGenTex, Inc.

23. MOQA User Interface: Validation Validation by domain experts is crucial for producing an intelligent and ergonomic question answering interface –Highly interconnected multimodality offers new views on data and methods of browsing –Feedback is crucial for the search results justification for session report production

24. CoGenTex, Inc. Thank you!