1. Natural Logic for Textual Inference Bill MacCartney and Christopher D. Manning NLP Group Stanford University 29 June 2007

2. 2 Inferences involving monotonicity Few states completely forbid casino gambling. OK Few western states completely forbid casino gambling. Few or no states completely forbid casino gambling. Few states completely forbid gambling. No Few states completely forbid casino gambling for kids. Few states or cities completely forbid casino gambling. Few states restrict gambling. What kind of textual inference system could predict this? Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

3. 3 Textual inference: a spectrum of approaches robust, but shallow deep, but brittle natural logic lexical/ semantic overlap Jijkoun & de Rijke 2005 patterned relation extraction Romano et al. 2006 pred-arg structure matching Hickl et al. 2006 FOL & theorem proving Bos & Markert 2006 Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

4. 4 What is natural logic? A logic whose vehicle of inference is natural language No formal notation:       Just words & phrases: All men are mortal … Focus on a ubiquitous category of inference: monotonicity I.e., reasoning about the consequences of broadening or narrowing the concepts or constraints in a proposition Precise, yet sidesteps difficulties of translating to FOL: idioms, intensionality and propositional attitudes, modalities, indexicals,reciprocals, scope ambiguities, quantifiers such as most, reciprocals, anaphoric adjectives, temporal and causal relations, aspect, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, … Aristotle, Lakoff, van Benthem, Sánchez Valencia 1991 Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

5. 5Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

6. 6 The entailment relation:  In natural logic, entailment is defined as an ordering relation over expressions of all semantic types (not just sentences) categorysemantic typeexample(s) common nouns etet penguin  bird adjectives etet tiny  small intransitive verbs etet hover  fly transitive verbs eeteet kick  strike temporal & locative modifiers (e  t)  (e  t) this morning  today in Beijing  in China connectives tttttt and  or quantifiers (e  t)  t (e  t)  (e  t)  t everyone  someone all  most  some Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

7. 7 Monotonicity of semantic functions Upward-monotone (  M) The default: “bigger” inputs yield “bigger” outputs Example: broken. Since chair  furniture, broken chair  broken furniture Heuristic: in a  M context, broadening edits preserve truth Downward-monotone (  M) Negatives, restrictives, etc.: “bigger” inputs yield “smaller” outputs Example: doesn’t. While hover  fly, doesn’t fly  doesn’t hover Heuristic: in a  M context, narrowing edits preserve truth Non-monotone (#M) Superlatives, some quantifiers ( most, exactly n ): neither  M nor  M Example: most. While penguin  bird, most penguins # most birds Heuristic: in a #M context, no edits preserve truth In compositional semantics, meanings are seen as functions, and can have various monotonicity properties: Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

8. 8 Downward monotonicity few athletes  few sprinters restrictive quantifiers: no, few, at most n prohibit weapons  prohibit guns negative & restrictive verbs: lack, fail, prohibit, deny without clothes  without pants prepositions & adverbs: without, except, only drug ban  heroin ban negative & restrictive nouns: ban, absence [of], refusal If stocks rise, we’ll get real paid  If stocks soar, we’ll get real paid the antecedent of a conditional didn’t dance  didn’t tango explicit negation: no, n’t Downward-monotone constructions are widespread! Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

9. 9 Monotonicity of binary functions Some quantifiers are best viewed as binary functions Different arguments can have different monotonicities all  All ducks fly  All mallards fly  All ducks move some  Some mammals fly  Some animals fly  Some mammals move no  No dogs fly  No poodles fly  No dogs hover not every  Not every bird flies  Not every animal flies  Not every bird hovers Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

10. 10 Composition of monotonicity Composition of functions  composition of monotonicity Sánchez Valencia: a precise monotonicity calculus for CG Few  forbid  states completely casino gambling +++––– Few states completely forbid casino gambling o MM MM #M MM MM MM MM MM MM Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

11. 11 The NatLog System linguistic pre-processing alignment entailment classification 1 2 3 textual inference problem prediction Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

12. 12 Step 1: Linguistic Pre-processing Tokenize & parse input sentences (future: & NER & coref & …) Identify & project monotonicity operators Problem: PTB-style parse tree  semantic structure! Few states completely forbid casino gambling JJ NNS RB VBD NN NN NP ADVP NP VP S +++––– Solution: specify projections in PTB trees using Tregex Few  forbid  states completely casino gambling few pattern: JJ < /^[Ff]ew$/ arg1:  M on dominating NP __ >+(NP) (NP=proj !> NP) arg2:  M on dominating S __ >+(/.*/) (S=proj !> S) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

13. 13 Step 2: Alignment Alignment = a sequence of atomic edits [cf. Harmeling 07] Atomic edits over token spans: DEL, INS, SUB, ADV Limitations: no easy way to represent movement no alignments to non-contiguous sets of tokens Benefits: well-defined sequence of intermediate forms can use adaptation of Levenshtein string-edit DP We haven’t (yet) invested much effort here Few states completely forbid casino gambling Few states have completely prohibited gambling ADV SUBADVINSDEL Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

14. 14 Step 3: Entailment Classification Atomic edits  atomic entailment problems Feature representation Basic features: edit type, monotonicity, “light edit” feature Lexical features for SUB edits: lemma sim, WN features Decision tree classifier Trained on small data set designed to exercise feature space Outputs an elementary entailment relation:   = # | Composition of atomic entailment predictions Fairly intuitive:  º   ,  º   #,  º =  =, etc. Composition yields global entailment prediction for problem Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

15. 15 predict featurize Entailment model example typeINS monodown isLighttrue Few states completely forbid casino gambling. Few states have completely prohibited gambling. SUBINSDEL typeSUB monodown isLightfalse lemSim0.375 wnSyn1.0 wnAnto0.0 wnHypo0.0 typeDEL monoup isLightfalse compose = (equivalent)  (forward) = (equivalent)  (forward) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

16. 16 The FraCaS test suite FraCaS: mid-90s project in computational semantics 346 “textbook” examples of textual inference problems No delegate finished the report. Some delegate finished the report on time. Smith believed that ITEL had won the contract in 1992. ITEL won the contract in 1992. 9 sections: quantifiers, plurals, anaphora, ellipsis, … 3 possible answers: yes, no, unknown (not balanced!) 55% single-premise, 45% multi-premise (excluded) unk no Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

17. 17 Results on FraCaS §Category#Acc. 1Quantifiers4484.09 2Plurals2441.67 3Anaphora650.00 4Ellipsis2528.00 5Adjectives1560.00 6Comparatives1668.75 7Temporal3661.11 8Verbs862.50 9Attitudes955.56 “Applicable”: 1, 5, 67576.00 All sections18359.56 yesunknototal yes6240—102 unk1545—60 no613221 total90912183 guess gold by section confusion matrix Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

18. 18 The RTE3 test suite RTE: more “natural” textual inference problems Much longer premises: average 35 words (vs. 11) Binary classification: yes and no RTE problems not ideal for NatLog Many kinds of inference not addressed by NatLog Big edit distance  propagation of errors from atomic model Maybe we can achieve high precision on a subset? Strategy: hybridize with broad-coverage RTE system As in Bos & Markert 2006 Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

19. 19 A hybrid RTE system using NatLog NatLog pre-processing alignment classification {yes, no} Stanford pre-processing alignment classification [– , +  ] threshold (balanced) {yes, no} xx threshold (optimized) {yes, no} Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

20. 20 Results on RTE3 RTE3 Development Set (800 problems) System% yes precisio n recall accurac y Stanford50.2568.6666.9967.25 NatLog18.0076.3926.7058.00 Hybrid, balanced 50.0069.7567.7268.25 Hybrid, optimized 55.1369.1674.0369.63 RTE3 Test Set (800 problems) System% yes precisio n recall accurac y Stanford50.0061.7560.2460.50 NatLog23.8868.0631.7157.38 Hybrid, balanced 50.0064.5062.9363.25 Hybrid, optimized 54.1363.7467.3263.62 25 extra problems (significant, p < 0.01) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

21. 21Conclusion Natural logic enables precise reasoning about monotonicity, while sidestepping the difficulties of translating to FOL. The NatLog system successfully handles a broad range of such inferences, as demonstrated on the FraCaS test suite. Future work: Add proof search, to handle multiple-premise inference problems Consider using CCG parses to facilitate monotonicity projection Explore the use of more sophisticated alignment models Bring factive & implicative inferences into the NatLog framework :-) Thanks! Questions? Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion