Assessing the Impact of Frame Semantics on Textual Entailment Authors: Aljoscha Burchardt, Marco Pennacchiotti, Stefan Thater, Manfred Pinkal Saarland Univ, Germany in Natural Language Engineering 1 (1) pp1-25 As (mis-)interpreted by Peter Clark

The Textual Entailment Task  Syntactically, the players have moved.  from a syntactic point of view, T and H differ  But semantically, the players are still the same  from a semantic point of view, T and H are the same  So, want to identify and match on semantic, not syntactic, level:  Need for “frame semantics”  (syntax) X drown Y → (semantics) cause* drown victim  (syntax) X drown in Y → (semantics) victim drown in cause T: A flood drowned 11 people. H: 11 people drowned in a flood. T: flood drown people H: people drown in flood * if X is inanimate (otherwise role is killer)

Frame Semantic Resources  PropBank:  thematic roles (arg0, arg1, …):  arg0 search arg1 for arg2 (“Mary searched the room for the ring”)  arg0 search for arg2 (“Fred searched for the ring”)  BUT roles are verb-specific (and names are overloaded)  arg0 seek arg1 (“Mary sought the ring”)  No guarantee arg0 means the same in different verbs  Note: thematic roles like “agent” are necessarily verb- specific:  Fred sold a car to John. John bought a car from Fred.  Thematic roles: Fred, John are both agents.  Case/semantic roles: Fred is the buyer, John is the seller.

Frame Semantic Resources  FrameNet:  Semantic roles are shared among verbs  several verbs map to the same Frame  Frames organized in a taxonomy  Roles organized in a taxonomy  Doesn’t contain subcategorization templates for semantic role labeling  causer kill victim  But does contain role-labeled examples, from which semantic role labeling algorithms can be learned

Example Frame in FrameNet

 So: it seems FrameNet should really help!

 Even more, FrameNet has (limited) inferential connections T: Wyniemko, now 54 and living in Rochester Hills, was arrested and tried in 1994 for a rape in Clinton Township. H: Wyniemko was accused of rape.

But, limited success in practice  PropBank used by several systems, including the RTE3 winner:  but unclear how much PropBank contributed  FrameNet used in SALSA (Burchardt and Frank)  Shalmaneser + Detour for Semantic Role Labeling (SRL)  (Detour boosts SRL when training examples are missing)  SALSA:  find matching semantic roles  see if the role fillers match  machine learning approach:  for set of known matching fillers  (i) compute features  (ii) learn which weighted sum of features implies match  But SALSA didn’t do significantly better than simple lexical overlap

Possible reasons for “failure”  Poor coverage of FrameNet  Decision of applicable Frame is poor  Semantic Role Labeling is poor  Role filler matching is poor  How to distinguish between these?  Create FATE, an annotated RTE corpus  Only annotated the “relevant” parts of the sentences

FATE  Annotated RTE2 corpus (400+ve, 400-ve exs)  Good interannotator agreement  ~2 months work to create  4488 frames, 9512 roles annotated in the corpus  includes 373 (8%) Unknown_Frame  1% Unknown_Role  → FrameNet coverage is good for this data!  Still, not always clear-cut:  Annotator: EXPORT; Shalmaneser: SENDING  SENDING is still plausible Cars exported by Japan increased

1. How do automatic and manual annotation compare? Does SALSA pick the right frame? When it picks the right frame, does assign the right roles? When it picks the right frame and role, does it get the right filler (i.e., the same head noun as the gold standard)   Fred sold the book on the shelf to Mary sellergoodsbuyer Commercial_Transaction

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T 2.The Frame’s roles used in H to also be in T 3.The role fillers in H to match those in T  These may also be true if H isn’t entailed by T  BUT: presumably with less probability

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) 2.The Frame’s roles used in H to also be in T (more often) 3.The role fillers in H to match those in T (more often)  These may also be true if H isn’t entailed by T  BUT: presumably with less probability  Also: compare with simple word overlap

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) Yes…. (Note: low difference here reflects that T and H typically talk about the same thing)

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) …but not much more than word overlap… (Not really surprising, as frames are picked based on words)

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) Also the hierarchy doesn’t help much here

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) 2.The Frame’s roles used in H to also be in T (more often) Again, low difference suggests that the roles talked about in T and H are usually the same For pairs which have a Frame in common between T and H:

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) 2.The Frame’s roles used in H to also be in T (more often) 3.The role fillers in H to match those in T (more often) 

Results  If H is entailed by T, then we expect 1.The Frame for H to also be in T (more often) 2.The Frame’s roles used in H to also be in T (more often) 3.The role fillers in H to match those in T (more often) T: An avalanche has struck a popular skiing resort in Australia, killing at least 11 people. H: Humans died in an avalanche. T: Virtual reality is used to train surgeons, pilots, astronauts, police officers, first-responders, and soldiers. H: Soldiers are trained using virtual reality. student Some difficult cases:

Results  Also, even if we had perfect frame, role, and filler matching, entailment does not always follow:  Negation:  Modality:

Conclusions 1.FrameNet’s coverage is good 2.Frame Semantic Analysis (frame/role/filler selection) is mediocre  3.Simple lexical overlap at the frame level don’t outperform simple lexical overlap at the syntactic level  4.Need better modeling:  wider context (negation, modalities)  role filler matching (semantic matching, e.g., WordNet)  more knowledge in FrameNet, e.g., implications  e.g., kill → die, arrest → accuse

(Extra slides)

The Textual Entailment Task: More complex example  Again, need to match semantic roles:  Again need for “frame semantics”  (syntax) X kill Y → (semantics) cause kill victim  (syntax) X died in Y → (semantics) protagonist died in cause  ALSO:  progagonist isa victim, Killing → Death T: An avalanche has struck a popular skiing resort in Australia, killing at least 11 people. H: Humans died in an avalanche. T: avalanche kill people H: human die in avalanche