1. TextInfer 2011 – Bar Ilan University 1 Towards a probabilistic Model for Lexical Entailment Eyal Shnarch, Jacob Goldberger, Ido Dagan

2. TextInfer 2011 – Bar Ilan University 2 Entailment at the lexical level Obama gave a speech last night in the Israeli lobby conference In his speech at the American Israel Public Affairs Committee yesterday, the president challenged … Barack Obamas AIPAC address... AIPAC Israeli lobby American Israel Public Affairs Committee address speech Barack Obama the president Obama

3. TextInfer 2011 – Bar Ilan University 3 Lexical-level systems are very handy Important component within a full inference system Pose hard-to-beat baselines –(Mirkin et. al 2009, Majumdar and Bhattacharyya 2010) Can be used in cases where there are no deep analysis tools for target language –e.g. no parser

4. TextInfer 2011 – Bar Ilan University 4 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Modeling entailment at the lexical level

5. TextInfer 2011 – Bar Ilan University 5 Mostly heuristic: Percent covered/un-covered –(Majumdar and Bhattacharyya, 2010, Clark and Harrison, 2010) Similarity estimation –(Corley and Mihalcea, 2005; Zanzotto and Moschitti,2006) Vector space –(MacKinlay and Baldwin, 2009) Lexical entailment scores

6. TextInfer 2011 – Bar Ilan University 6 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Terminology rule lexical resource chain rule 2 rule 1

7. TextInfer 2011 – Bar Ilan University 7 The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group Goal – a probabilistic model 1.Distinguish resources reliability levels 2.Consider transitive chains length 3.Consider multiple evidence Addressing:

8. TextInfer 2011 – Bar Ilan University 8 Entailment validation process t1t1 tmtm titi h1h1 hnhn hjhj t chain … … …… A hypothesis is entailed if all its terms are entailed A single term is entailed if at least one of its evidence is a valid entailment chain A chain is valid if all its rule steps are valid The validity of a rule depends on the reliability of the resource which provided it

9. TextInfer 2011 – Bar Ilan University 9 Probabilistic model for Lexical Entailment t1t1 tmtm titi h1h1 hnhn hjhj t AND OR chain … … …… validity prob. of a rule step r is the reliability of the resource R(r) which suggested it EM to estimate parameter set if entailment holds

10. TextInfer 2011 – Bar Ilan University 10 Lets try a concrete example The presidents car got stuck in Ireland, surrounded by many people Obamas Cadillac got stuck in Dublin in a large Irish crowd social group * numbers in blue are parameter values found by our model

11. TextInfer 2011 – Bar Ilan University 11 Results on RTE are nice, but… F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 47.644.4Best lexical system 48.045.6Best full system F1F1

12. TextInfer 2011 – Bar Ilan University 12 Extension 1: relaxing with noisy-AND noisy- final AND gate demands the entailment of all hypothesis terms sentence level entailment is possible even if not all terms are entailed this strict demand is especially unfair for longer hypotheses

13. TextInfer 2011 – Bar Ilan University 13 Better results with extension 1 F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 43.144.6Base Prob. + noisy-AND 47.644.4Best lexical system 48.045.6Best full system * * * significant improvement over base prob. according to Mc-Nemars test with p<0.01 F1F1

14. TextInfer 2011 – Bar Ilan University 14 Extension 2: terms independence assumption uncovered term covered term As T covers more terms of H – our belief in each rule application increases

15. TextInfer 2011 – Bar Ilan University 15 Same (better) results with extension 2 F 1 % Model RTE 6RTE 5 33.830.5Avg. of all systems 38.536.2Base Prob. 43.144.6Base Prob. + noisy-AND 44.742.8Base Prob. + coverage normalization 47.644.4Best lexical system 48.045.6Best full system * * F1F1

16. TextInfer 2011 – Bar Ilan University 16 Putting it all together is best Negative result: F1 usually decreases when allowing chains * * F1F1

17. TextInfer 2011 – Bar Ilan University 17 Summary Learns for each lexical resource an individual reliability value Considers multiple evidence and chain length Two extensions which brings us to… Performance is in line with best entailment systems A probabilistic model: noisy-

18. TextInfer 2011 – Bar Ilan University 18 Future work Better model for transitivity noisy-AND for chains too Verify rule application in a specific context next talk by Shachar Mirkin Test with other application data sets passage retrieval for QA Integrate into a full entailment system Thank you!