1. AQUAINT R&D Program: “State of the Program” Phase I 18-Month Workshop 9-12 June 2003 Dr. John D. Prange AQUAINT Program Manager jprange@nsa.gov 301-688-7092 http://www.ic-arda.org

2. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 2 Focus of Workshop Technical Talks –AQUAINT Project Highlights; Not Status Reports –Written Project Status Reports included on Workshop CD Two Separate Demos Sessions Exercise Focused on Advanced Question Answering Scenarios Program Updates on: –Data- Testbed - Evaluations Birds of a Feather Sessions Wed Evening Update on Phase II Planning and Schedule

3. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 3 Still More to Talk About AQUAINT Program Symposium Series NRRC Summer Workshops Other QA Related Activities –AAAI Spring Symposium Workshop –NAACL-HLT 2003 Conference –Continuing evolution of the TREC QA Track –Question Answering Books in the works –Coordination with other Related Programs –The Learning Federation’s Question Answering Roadmap –Vulcan Inc. HALO Projects AQUAINT Program Phase I 24-Month Workshop A Couple of Special Recognitions

4. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 4 AQUAINT Symposium Series Need for Something More than Program Workshops –Program is sponsoring multiple approaches to Advanced Question Answering –Significant Progress is being made is multiple areas –My own very positive experience from Project Reviews/Site Visits –Smaller setting; comprehensive overview of project activities, approaches and successes; highly interactive setting 20-30 minute overview presentations in Workshop setting just aren’t the same The Idea of a Symposium Series is born

5. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 5 AQUAINT Spring Symposium - 2003 Rapidly Organized by ARDA with help from NIST (Donna Harman, Ellen Voorhees) and ARDA’s Northwest Regional Research Center (Rich Quadrel) Initial Symposium held at NIST, Gaithersburg, MD on Mon-Tues 5-6 May 2003 Format: –4 half-day detailed technical overview presentations by four AQUAINT Program teams –Attendance limited to AQUAINT Program PI’s, a few Government personnel and invited guests –Encouraged high degree of interaction between presenters and audience

6. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 6 AQUAINT Spring Symposium – 2003 (con’t) Presenters: –Language Computer Corporation (LCC) (Combination of 3 Projects) –University of Southern California / Information Science Institute (TextMap Project) –IBM T. J. Watson Research Center (PIQUANT Project) –Cycorp Inc. (QUIRK Project) Results –One significant glitch with Cycorp’s Internet connection –Lively “Discussions” & Interactions between researchers from different teams – It was great to see & hear! –Format proved to be right on the money –CD with all presentations will be published by NWRRC –Sessions videotaped; Future distribution is a possibility

7. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 7 AQUAINT Fall Symposium - 2003 Hugh Success of Spring Symposium implies we’ll try it again Tentative Dates: –8-9 October 2003 (Wed & Thurs) (Note: Yom Kippur is on Tues 7 Oct – Any other known conflicts??) –West Coast location likely – NWRRC is searching for a location Symposium Focus –Focus of Spring Symposium was on “End-to-end Systems” –What should the focus of Fall Symposium be? –Volunteers by teams to make Presentation? Symposium Format –Keep things pretty much the same – increase size only slightly to accommodate on a case-by-case basis more than just the PI from a given Project –Honorarium again used to cover travel costs

8. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 8 Northeast Regional Research Center http://nrrc.mitre.org Conduct 6-8 week workshops on multiple ARDA-related challenge problems –Three AQUAINT Focused Workshops conducted during Spring / Summer 2002 –Three AQUAINT Focused Workshops to be conducted during Summer 2003 Analytic Disciplines Day (3 Feb 2003) / Intelligence Analysis Futures Day (27 June 2003) –Periodic day-long set of talks focused on some aspect of Analysis –Held at MITRE-Bedford but video teleconferenced to a number of other MITRE locations Participation/Attendance open to interested contractors and to Government researchers/technologists/analysts Hosted By MITRE, Bedford, MA

9. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 9 FY2002 NRRC Wkshp Challenge Problems 1.Temporal Issues –Generate Sequence of events and activities along evolving timeline, resolving multiple levels of time references across series of documents/sources. Developed: 1. TimeML: Definition and design of a Metadata Standard for Markup of events, their temporal anchoring, and how they are related to each other in News articles. 2. TIMEBANK: Creation of a gold standard corpus of 300 articles marked up for temporal expressions, events, and basic temporal relations, based on the specification of TimeML. –Leader: James Pustejovsky, Brandeis University TERQAS Web Site: http://time2002.org

10. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 10 3.Re-Use of Accumulated Knowledge –Investigate strategies for structuring and maintaining previously generated knowledge for possible future use. E.g. previous knowledge might include questions and answers (original and amplified) as well as relevant and background information retrieved and processed. –Leaders: Marc Light, MITRE and Abraham Ittycheriah, IBM FY2002 NRRC Wkshp Challenge Problems 2.Multiple Perspectives –Develop approaches for handling situations where relevant information is obtained from multiple sources on the same topic but generated from different perspectives (e.g. cultural or political differences). –Leader: Jan Wiebe, University of Pittsburgh

11. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 11 2.Graphical Annotation Toolkit for TimeML –Address some specific annotation and user interface problems that arose in the context of the TERQAS workshop in 2002: 1. Development of a Text Segmented Closure Algorithm 2. Creation of a Semi-graphical Annotation Tool –Leader: James Pustejovsky, Brandeis University FY2003 NRRC Wkshp Challenge Problems 1.Toward More Reliable Information Retrieval Technology –Explore issues in IR performance and QA relating to performance variability as a function of topic, collection, queries & other contextual aspects –Leader: Donna Harman, NIST

12. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 12 3.Preparing to Explore a New Paradigm in Information Access: A Scenario Approach to Question- Answering –Jointly Explore with QA Researchers and Subject Matter Experts the nature and implications of the emerging QA Scenario Paradigm and report back on: 1. The new paradigm from the intelligence analysts’ perspective, 2. The more advanced capabilities being developed beyond simple factoid QA 3. The scenarios proposed by the Subject Matter Experts, 4. The range of QA capabilities needed to make scenario-based QA a reality. –Leader: Liz Liddy, Syracuse University FY2003 NRRC Wkshp Challenge Problems

13. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 13 Regional Research Centers: FY 2004 Workshop Challenge Problems Resource Enhancement Program’s Regional Research Centers (ARDA POC: Penny Lehtola) –Northeast: MITRE, Bedford, MA Mark Maybury Directorhttp://nrrc.mitre.org –Northwest: Battelle PNNL, Richland, WA Rich Quadrel, Director –Southern CA: SPAWAR, San Diego, CA John Audia, Director Call for Workshop Proposals in Fall 2003 Timeframe –Process concerning Call for Proposals is being refined –Strongly Encourage AQUAINT-Focused Workshops –Talk to me if you are interested; if you have an idea; if you want to brainstorm an idea –Will send out email to AQUAINT PI’s when Call for Proposal Process has been finalized

14. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 14 Related External Workshops Workshop on New Directions in Question Answering AAAI Spring Symposium 24-26 March 2003 Stanford University Palo Alto, CA Mark Maybury, MITRE Workshop Chair http://www.aaai.org/Symposia/Spring/2003/sss-03.html

15. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 15 Related External Conferences HLT (Human Language Technology) Conferences –Sponsored by NAACL & Government Funders DARPA, NSF, and ARDA; SIGIR and ISCA represented on HLT Advisory Committee –NAACL/HLT-2003:Edmonton, Alberta, Canada 27 May - 1 June 2003 http://www.hlt03.orghttp://www.hlt03.org –Solid participation from AQUAINT Program Number of Technical Talks Pre-Conference Tutorial on TimeML Post-Conference Workshop on Gazetteer Exploitation –NAACL/HLT-2004:Continuation endorsed by both NAACL and Government Funders Likely location:Boston, MA Likely Timeframe:2 nd Week in May 2004

16. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 16 Related External Conferences Text REtrieval Conferences (TREC) Question Answering (QA) Track –TREC managed by NIST –Co-Sponsored by DARPA and ARDA –TREC-2003:NIST, Gaithersburg, MD 18-21 November 2003 http://trec.nist.govhttp://trec.nist.gov –QA Track Question Types - 2003 Factoid Questions List Questions Definition Questions –Evolving Nature of QA Track in TREC & AQUAINT Program Evaluations

17. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 17 Question Answering Books TERQAS Book –Outgrowth of NRRC 2002 Summer Workshop –James Pustejovsky, Brandeis University, Editor Future Directions in Question Answering –Outgrowth of AAAI Spring Symposium 2003 –To be published by MIT Press – Mark Maybury, MITRE, Editor Advanced Question Answering Topics –Focus on newly emerging approaches to advanced QA –To be published by Kluwer Press –Co-Authors: Tomek Strzalkowski, SUNY Albany Sanda Harabagiu, University of Texas at Dallas

18. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 18 Coordination with Other Government Sponsors of Information Technology R&D Periodic Coordination Meetings Program Managers Involved –DARPA: Charles Wayne, Ted Senator –NSF: Gary Strong –ITIC:Art Becker –ARDA:John Prange, Lucy Nowell –Intelligence Community: Steve Dennis; J.K. Davis Programs Covered –DARPA:TIDES, EARS, EELD, Info Awareness + Others –NSF:Information Technology Research - ITR –ITIC:KD-D –ARDA:AQUAINT, NIMD + Others –IC:ACE + Others Some Coordination with DARPA’s DAML and RKF Programs

19. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 19 The Learning Federation Question Answering Roadmap Question Generation and Answering Systems R&D for Technology-Enabled Learning Systems –“An important component of advanced learning systems will be tools for answering learner queries. A key R&D issue is how to use what we know about question generation and question answering to design learning environments that guide learners to actively construct knowledge.” –Published a draft Roadmap Document: http://www.learningfederation.org/workshops/wks2/index.html –POC: Kay Howell; khowell@fas.orgkhowell@fas.org –Affiliated with Federation of American Scientists Workshops : –Oct 2002, Univ. of Memphis, Memphis, TN –April 2003, MITRE, Bedford, MA

20. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 20 Plans for AQUAINT Program Phase I 24-Month Workshop When & Where –2-5 Dec 2003 (Tues-Friday) –Opening Reception Evening of 1 Dec –MITRE, McLean, VA MITRE 1 Conference Facility (formerly Hayes Building) 7525 Colshire Drive McLean, VA 22102-7508 www.mitre.org/about/locations.html Workshop Focus –Final Overview Summary of Phase I –Project Summary Presentation (30-60 minutes each) Highlight significant research activities and results Focus on Contributions to and Impact on the AQUAINT Program Final System Demonstrations MITRE 1

21. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 21 Special Recognition: AQUAINT Program Executive Committee AQUAINT Program Executive Committee formed in January 2002; Meets Monthly Intelligence Community Members: –ARDA (John Prange; Heather McCallum-Bayliss-SETA; Paul Matthews- SETA) –CIA (John Donelon, Steve Maiorano, Jean-Michel Pomarede-SETA) –DIA (Kelcy Allwein) –NIMA (Duncan McCarthy) –NSA (Carol Van Ess-Dykema, J.K. Davis, Mike Blair-SETA) Other Members/Advisors: –NIST (Donna Harman, Ellen Voorhees) –MITRE (Scott Mardis, John Burger) –Tarragon Consulting (Richard Tong)

22. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 22 Special Recognition ! ! Uppsala University 24 Jan 2003 Web Cam “I have … climbed the Matterhorn, driven a Land Rover from London to Capetown, gotten attacked by a thousand people in Egypt, gotten stuck in quicksand in the interior of Iceland, flown in a Russian cargo plane to Timbuktu, followed orangutans around the Borneo rain forest, narrowly avoided being kidnapped in Yemen, etc.” Three Decades of work in AI and Computational Linguistics Information Seeking Dialogs FASTUS Scandinavia Venerable Swedish University Annual Site of Nobel Prize Ceremony First Some Hints...

23. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 23 Special Recognition ! ! First Some Hints... But What’s The Connection? Uppsala University 24 Jan 2003 Web Cam Three Decades of work in AI and Computational Linguistics Information Seeking Dialogs FASTUS Jerry Hobbs USC / ISI Scandinavia Venerable Swedish University Annual Site of Nobel Prize Ceremony “I have … climbed the Matterhorn, driven a Land Rover from London to Capetown, gotten attacked by a thousand people in Egypt, gotten stuck in quicksand in the interior of Iceland, flown in a Russian cargo plane to Timbuktu, followed orangutans around the Borneo rain forest, narrowly avoided being kidnapped in Yemen, etc.”

24. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 24 CONGRATULATIONS ! ! On January 24, 2003, Uppsala University in Sweden bestowed the title of Doctor of Philosophy honoris causa on Jerry Hobbs of USC / ISI and formerly of SRI International The AQUAINT Family extends our Congratulations to Jerry ! ! And if we haven’t embarrassed Jerry enough, here are some more pictures! ** ** From his colleagues at USC / ISI

25. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 25 Gets ready Receives degree The laurel wreath Our hero rehearses

26. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 26 Anna Sagvall-Hein of Uppsala University The Banquet in the Castle Jerry bought the artillery shell fired in his honor

27. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 27 Questions / Comments

28. AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 28 Contact Information Dr. John Prange, AQUAINT Program Director ARDA Web Pages:http://www.ic-arda.org Emailarda@nsa.gov JPrange@nsa.gov Phones:301-688-7092 800-276-3747 301-688-7410 (Fax) Mailing:ARDA (RA) Room 12A69 NBP#1 STE 6644 9800 Savage Road Fort Meade, MD 20755-6644