TTO3 Semantic Annotation Analysis Bonnie Dorr September 9, 2008

Analysis of Annotation Study Interannotator Agreement Multi-translation Equivalence Multi-language Equivalence Modality95%75% equivalenceNA (no Arabic) Temporal Type95%100% equivalence62% equivalence Inherent Time96%93% equivalence85% equivalence Temporal Parent85%100% equivalence92% equivalence Temporal Relations97%100% equivalence75% equivalence BeliefNA (not doubly annotated)100% equivalence90.5% equivalence Dialog ActsNA (not doubly annotated)NA (no multi-trans)NA (no parallel docs) Event CorefNA (not doubly annotated)100% equivalenceNA (no parallel docs)

Annotation Agreement: UMBC Modality *Loose = Within.2 ModalityItemStrictLoose*Example Belief (13)96%46%100%Expect =.9 Epistemic (8)94%63%88%May =.8 Epiteuctic (5)91%34%95%Get = 1 Obligative (62)99%34%98%Should =.7 Permissive (15)91%60%100%Allow =.8 Potential (31)98%74%87%Can =.8 Volitive (19)95%37%95%Prefer =.7 Evaluative (11)100% Best = 1 OVERALL (164)97%52%95%

Labeling Equivalence: UMBC Modality ModalityItemStrictLoose*Example Potential/Permissive (1)100%0% Can/Allow =.8 Volitive (2)100%50%100%Want =.7 Evaluative (1)100% Best = 1 OVERALL (4)100%50%75% *Loose = Within.2

Modality Example 1 Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center." INFORM-100(announce) HUM-92(Shiekh Mohammed) CEREMONY-102(ceremony) MODALITY-103(want)[TYPE=VOLITIVE,VALUE=1] CHANGE-3(make) GEO-POL-ENT-106(Dubai) LOCATION-OF-5() COMMERCE-EVENT-108(trade) Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy". ASSERTIVE-ACT-177(say) HUM-172(Shiekh Mohamed) CEREMONY-179(ceremony) MODALITY-180(wish)[TYPE=VOLITIVE,VALUE=1] EVENT-182(make) GEO-POL-ENT-184(Dubai) CENTER-POINT-186(center) ECONOMY-187(economy) Equivalence: Strict = 100%, Loose = 100%

Modality Example 2 The minister, {who has his own website}, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies." ASSERTIVE-ACT-70(say) MINISTER-67(minister) MODALITY-71(want)[TYPE=VOLITIVE,VALUE=.8] MODALITY-6()[TYPE=EVALUATIVE,VALUE=1] GEO-POL-ENT-74(Dubai) FOR-PROF-CORP-79(company) TECHNOLOGY-78(technology) The minister {who has a personal website on the internet}, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies. ASSERTIVE-ACT-941(say) MINISTER-936(minister) DISCOURSE-940(further) MODALITY-942(want)[TYPE=VOLITIVE,VALUE=1] CHANGE-EVENT-944(become) MODALITY-2()[TYPE=EVALUATIVE,VALUE=1] GEO-POL-ENT-943(Dubai) FOR-PROF-CORP-949(company) Equivalence: Strict = 0%, 100%, Loose = 100%, 100% Note: {} units omitted for simplificity

Modality Example 3 He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies. ASSERTIVE-ACT-157(say) HUMAN-156(he) MODALITY-170(would be able)[TYPE=PERMISSIVE,VALUE=.8] SET-168() CAUSE-BY-169(set up)[FOR-PROFIT-158, OFFICE-161] HIRE-162(employ) [SET-163(staff)] OWN-164(own) [DEVICE-165(equipment) [ZONE-167(zone)]] He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners. ASSERTIVE-ACT-598(say) HUMAN-596(he) MODALITY-599(would be possible)[TYPE=POTENTIAL,VALUE=.6] OWN-604(establish)[FOR-PROFIT-600, OFFICE-605] HIRE-606(employment) OWN-607(own) [SET-608(machinery) [PLACE-609(zone)]] Equivalence: Strict = 0%, Loose = 0%

Annotation Agreement: BBN’s Temporal Annotations

Labeling Equivalence: BBN’s Temporal Annotations

E1: Sheikh Mohammed, who is also the Defense Minister(وزير) announced(واعلن) at the inauguration ceremony(التدشين) that we want(نريد) to make(نجعل) Dubai a new trading center. E2: Sheikh Mohamed, who is also the Defense Minister(وزير) said(واعلن) during the launching ceremony(التدشين) we wish(نريد) to make(نجعل) Dubai a new center of modern economy TimeUnit TypeRelationParent minister(الوزير)* Present.StateAfter/Before/Conc announced announced(واعلن) Past.Say*Before ceremony(التدشين) Unspec.EventAfter/Before/Conc announced* want(نريد) Present.State*After/Before/Conc announced make(نجعل) Unspec.State*After* want ANNOUNCE (واعلن) [Past.Say, Before( )] MINISTER (الوزير) [Present.State, After/Before/Conc(ANNOUNCED)] CEREMONY (التدشين) [Unspec.Event, After/Before/Conc(ANNOUNCED)] WANT (نريد) [Present.State, After/Before/Conc(ANNOUNCED)] MAKE (نجعل) [Unspec.State, After(WANT)] Temporal Relation Example 1: Multi-translation and Multilingual announced ceremony minister TIME want make Equivalence Arabic-EnglishEnglish-English Type Match25%100% Parent Match75%100% Relation Match67%100%

E1: The minister who has(يملك) his own website also said(واضاف) I want( اريد ) Dubai to be the best place(تصبح) in the world for companies. E2: The minister who has(يملك) a personal website further said(واضاف) he wanted( اريد ) Dubai to become the best place(تصبح) in the world for companies. TimeUnit Type Relation Parent has(يملك) Present.State* After/Before/Concurrent said said(واضاف) Past.Say* After (prev sent)* want(اريد) Present.State* + After/Before/Concurrentsaid place(تصبح) Unspec.State* Afterwant SAID (واضاف) [Past.Say, After( )] HAS (يملك) [Present.State, After(SAID)] WANT (اريد) [Present.State, After/Before/Concurrent(SAID)] PLACE (تصبح) [Unspec.State, After(WANT)] Temporal Relation Example 2: Multi-translation and Multilingual Equivalence Arabic-EnglishEnglish-English Type Match0%75% Parent Match75%100% Relation Match100% said want has TIME place

E1: He said(وقال) companies engaged in commerce would be able(سيكون) to set up(وتقيم) offices, employ(وتستخدم) staff and own(وتمتلك) equipment in the open zone. E2: He said(وقال) that it would be possible(سيكون) for companies working in the electronic trade sector to establish(وتقيم) offices here and provide(وتستخدم) employment and own(وتمتلك) machinery in the zone. TimeUnit TypeRelationParent said(وقال) Past.SayBefore* * possible(سيكون) Future.StateAftersaid establish(وتقيم) Unspec.EventAfter possible provide(وتستخدم) Unspec.EventConcurrent establish own(وتمتلك) Unspec.EventConcurrent provide SAID (وقال) [Past.Say, Before( )] POSSIBLE (سيكون) [Future.State, After(SAID)] ESTABLISH (وتقيم) [Unspec.Event, After(POSSIBLE)] PROVIDE (وتستخدم) [Unspec.Event, Concurrent(ESTABLISH)] OWN (وتمتلك) [Unspec.Event, Concurrent(PROVIDE)] Temporal Relation Example 3: Multi-translation and Multilingual said establish possible TIME provide own Equivalence Arabic-EnglishEnglish-English Type Match100% Parent Match80%100% Relation Match100%

Labeling Equivalence: CMU and Columbia’s Committed Belief

Committed Belief Example 1 E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center." E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy". [WILL INSERT EXAMPLE SOON]

Committed Belief Example 2 E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies." E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies. [WILL INSERT EXAMPLE SOON]

Committed Belief Example 3 E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies. E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners. [WILL INSERT EXAMPLE SOON]

Labeling Equivalence: CMU Event Coref *Loose = arguments unordered and/or extra unused arguments

Event Coref Example 1 E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center." E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy". [WILL INSERT EXAMPLE SOON]

Event Coref Example 2 E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies." E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies. [WILL INSERT EXAMPLE SOON]

Event Coref Example 3 E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies. E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners. [WILL INSERT EXAMPLE SOON]

Remaining Analyses Additional multilingual and multi-translation sets Non-parallel documents about the same event Additional genre: Informal texts Additional phenomena: Coreference