1. NRRC Summer ‘02 Workshop Proposal Temporal December 5, 2001 MITRE James Pustejovsky, Brandeis with input from: James Allan, UMASS Inderjeet Mani, MITRE Janet Hitzeman, MITRE John Frank, MetaCarta Erik Rauch, MetaCarta jamesp@cs.brandeis.edu

2. Page 2 Copyright © 2001 The MITRE Corporation. All rights reserved. Problem  Find most accurate time reference for events in news with the goal of answering temporally-based questions related to those events  Sample questions: -Is Gates currently CEO of Microsoft? -Were there any meetings between the terrorist hijackers and Iraq before the WTC event? -Did the Enron merger with Dynegy take place?

3. Page 3 Copyright © 2001 The MITRE Corporation. All rights reserved. Answers:  May be selections from Database entries  May be textual fragments from articles  May be abstracted and represented visually or graphically in a timeline or hyperbolic graph E2E2 E3E3 E1E1 Were there any meetings(e2,e1) between terrorist hijackers and Iraq before the WTC event (e3)?

4. Page 4 Copyright © 2001 The MITRE Corporation. All rights reserved. Workshop Goals:  TenseML: To Define and Design a Metadata Specification for Markup of events, their temporal anchoring, and how they are related to each other in News text.  Algorithm Design: Given the definition of TenseML, to design and develop algorithms for extracting such data from news texts.

5. Page 5 Copyright © 2001 The MITRE Corporation. All rights reserved. Preliminary Tasks:  Time stamping events (identifying an event and anchoring it in time);  Ordering events with respect to each other (relating more than one event in terms of precedence, overlap, and inclusion);  Temporal and event anaphora  Reasoning about the ramifications of an event (what is changed by virtue of an event);  Reasoning about the persistence of an event (how long does an event or the outcome of an event persist).

6. Page 6 Copyright © 2001 The MITRE Corporation. All rights reserved. TenseML:  DTD for event types (state, process, transition) - adopt a convention of how to tag events and aspect  DTD for temporal types - Intervals, points, scales  DTD for tense relations of precedence, overlap, inclusion, and identity  DTD for persistence and ramification consequences.  Basic Semantics for common English temporal expressions

7. Page 7 Copyright © 2001 The MITRE Corporation. All rights reserved. TIMEBANK Gold Standard  Create Gold standard -Human annotated table of temporally grounded events and temporal expressions: -300-500 articles tagged according to the markup language defined by workshop participants  Annotation of events, temporal references, and relative event orderings: -Specification to be determined -Draft versions in Mani and Wilson (2000) -Ferro, Mani, Sundheim, and Wilson (2001) -Filatova and Hovy (2001) -Ferro (2001) -Setzer and Gaizauskas (2001)

8. Page 8 Copyright © 2001 The MITRE Corporation. All rights reserved. Domain and Data sets  Project is likely to use the news domain  Possible data sets: -Reuters Newswire source (in NewsML and CCS) -Factiva Corpus -TDT -TIDES -TIDES TDT-2 temporal corpus

9. Page 9 Copyright © 2001 The MITRE Corporation. All rights reserved. Algorithm Design  Evaluated and Develop algorithms for extracting event and time expressions  Evaluate and Develop algorithms for question answering for event and time expressions

10. Page 10 Copyright © 2001 The MITRE Corporation. All rights reserved. Evaluation Methods  Accuracy of event/time table generation -Against gold standard  Accuracy of answers for questions -NB: Assume question classification already known -Against gold standard (real answer) -Degradation from gold standard to system output  Evaluation will be continuous throughout workshop with true blind final exam at the end  Evaluations will be performed by the participants

11. Page 11 Copyright © 2001 The MITRE Corporation. All rights reserved. Work Plan and Schedule  Mid-January: Kickoff Meeting: 1 week -Begin define and design of specification: annotation  March: 3-day meeting -Assess annotation effort  April: 1 week meeting -Test and extend existing algorithms  May: 1 week meeting -Deploy algorithms  June: 1 week meeting -Integration  July: 1 week meeting (Close up and evaluation) -Evaluation and Wrap-up

12. Page 12 Copyright © 2001 The MITRE Corporation. All rights reserved. Proposed Team Membership  Core Members -Robert Ingria, LingoMotors -Dragomir Radev, U. Michigan -Janet Hitzeman, MITRE -John Moynes, Hapax -Jose Castano, Brandeis -James Allen, Rochester -Rob Gaizauskas, Sheffield  Leader: James Pustejovsky, Brandeis  Additional Participants -Lisa Ferro, MITRE -Inderjeet Mani, MITRE -Antonio Sanfilippo, SRA -Len Schubert, Rochester

13. Page 13 Copyright © 2001 The MITRE Corporation. All rights reserved. Impact  Better understanding of limits of current technology  Preliminary capabilities for answering questions using that output  Groundwork for event ordering capabilities  Specification for Temporal and Event Markup  Gold Standard Corpus that can be used by anyone in the community (TIMEBANK)  Add a new dimension to the kinds of Q&A possible

14. Page 14 Copyright © 2001 The MITRE Corporation. All rights reserved. Resources and Cost  Brought by the team -Expertise in Event Representation and Reasoning -Event Extraction Capabilities (LingoMotors) -TEMPEX time annotation (MITRE) -ACE specifications  Costs -Participants’ salary -Corpora, likely to be quite cheap -TenseML Annotation, from LDC or local annotators -Computers and storage -Coordination help -Travel, for the separate meetings