TextInfer 2011 – Bar Ilan University 1 Towards a probabilistic Model for Lexical Entailment Eyal Shnarch, Jacob Goldberger, Ido Dagan

TextInfer 2011 – Bar Ilan University 2 Entailment at the lexical level Obama gave a speech last night in the Israeli lobby conference In his speech at the American Israel Public Affairs Committee yesterday, the president challenged … Barack Obamas AIPAC address... AIPAC Israeli lobby American Israel Public Affairs Committee address speech Barack Obama the president Obama

TextInfer 2011 – Bar Ilan University 3 Lexical-level systems are very handy Important component within a full inference system Pose hard-to-beat baselines –(Mirkin et. al 2009, Majumdar and Bhattacharyya 2010) Can be used in cases where there are no deep analysis tools for target language –e.g. no parser

TextInfer 2011 – Bar Ilan University 4 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Modeling entailment at the lexical level

TextInfer 2011 – Bar Ilan University 5 Mostly heuristic: Percent covered/un-covered –(Majumdar and Bhattacharyya, 2010, Clark and Harrison, 2010) Similarity estimation –(Corley and Mihalcea, 2005; Zanzotto and Moschitti,2006) Vector space –(MacKinlay and Baldwin, 2009) Lexical entailment scores

TextInfer 2011 – Bar Ilan University 6 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Terminology rule lexical resource chain rule 2 rule 1

TextInfer 2011 – Bar Ilan University 7 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Goal – a probabilistic model 1.Distinguish resources reliability levels 2.Consider transitive chains length 3.Consider multiple evidence Addressing:

TextInfer 2011 – Bar Ilan University 8 Entailment validation process t1t1 tmtm titi h1h1 hnhn hjhj t chain … … …… A hypothesis is entailed if all its terms are entailed A single term is entailed if at least one of its evidence is a valid entailment chain A chain is valid if all its rule steps are valid The validity of a rule depends on the reliability of the resource which provided it

TextInfer 2011 – Bar Ilan University 9 Probabilistic model for Lexical Entailment t1t1 tmtm titi h1h1 hnhn hjhj t AND OR chain … … …… validity prob. of a rule step r is the reliability of the resource R(r) which suggested it EM to estimate parameter set if entailment holds

TextInfer 2011 – Bar Ilan University 10 Lets try a concrete example The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group * numbers in blue are parameter values found by our model

TextInfer 2011 – Bar Ilan University 11 Results on RTE are nice, but… F 1 % Model RTE 6RTE Avg. of all systems Base Prob Best lexical system Best full system F1F1

TextInfer 2011 – Bar Ilan University 12 Extension 1: relaxing with noisy-AND noisy- final AND gate demands the entailment of all hypothesis terms sentence level entailment is possible even if not all terms are entailed this strict demand is especially unfair for longer hypotheses

TextInfer 2011 – Bar Ilan University 13 Better results with extension 1 F 1 % Model RTE 6RTE Avg. of all systems Base Prob Base Prob. + noisy-AND Best lexical system Best full system * * * significant improvement over base prob. according to Mc-Nemars test with p<0.01 F1F1

TextInfer 2011 – Bar Ilan University 14 Extension 2: terms independence assumption uncovered term covered term As T covers more terms of H – our belief in each rule application increases

TextInfer 2011 – Bar Ilan University 15 Same (better) results with extension 2 F 1 % Model RTE 6RTE Avg. of all systems Base Prob Base Prob. + noisy-AND Base Prob. + coverage normalization Best lexical system Best full system * * F1F1

TextInfer 2011 – Bar Ilan University 16 Putting it all together is best Negative result: F1 usually decreases when allowing chains * * F1F1

TextInfer 2011 – Bar Ilan University 17 Summary Learns for each lexical resource an individual reliability value Considers multiple evidence and chain length Two extensions which brings us to… Performance is in line with best entailment systems A probabilistic model: noisy-

TextInfer 2011 – Bar Ilan University 18 Future work Better model for transitivity noisy-AND for chains too Verify rule application in a specific context next talk by Shachar Mirkin Test with other application data sets passage retrieval for QA Integrate into a full entailment system Thank you!