1. Containment, Exclusion, and Implicativity: A Model of Natural Logic for Textual Inference Bill MacCartney and Christopher D. Manning NLP Group Stanford University 14 February 2008

2. 2 The textual inference task Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Does premise P justify an inference to hypothesis H? An informal, intuitive notion of inference: not strict logic Focus on local inference steps, not long chains of deduction Robust, accurate textual inference could enable: Question answering [Harabagiu & Hickl 06] Semantic search Customer email response Relation extraction (database building) Document summarization A limited selection of data sets: RTE, FraCaS, KBE

3. 3 Some simple inferences No state completely forbids casino gambling. OK No western state completely forbids casino gambling. Few or no states completely forbid casino gambling. No state completely forbids gambling. No No state completely forbids casino gambling for kids. No state or city completely forbids casino gambling. No state restricts gambling. What kind of textual inference system could predict this? Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

4. 4 Textual inference: a spectrum of approaches robust, but shallow deep, but brittle natural logic lexical/ semantic overlap Jijkoun & de Rijke 2005 patterned relation extraction Romano et al. 2006 semantic graph matching Hickl et al. 2006 MacCartney et al. 2006 Burchardt & Frank 2006 FOL & theorem proving Bos & Markert 2006 Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

5. 5 What is natural logic? (natural logic  natural deduction) Lakoff (1970) defines natural logic as a goal (not a system) to characterize valid patterns of reasoning via surface forms (syntactic forms as close as possible to natural language) without translation to formal notation:       A long history traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, … van Benthem & Sánchez Valencia (1986-91): monotonicity calculus Precise, yet sidesteps difficulties of translating to FOL: idioms, intensionality and propositional attitudes, modalities, indexicals, reciprocals, scope ambiguities, quantifiers such as most, reciprocals, anaphoric adjectives, temporal and causal relations, aspect, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, … Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

6. 6 Monotonicity calculus (Sánchez Valencia 1991) Entailment as semantic containment: rat < rodent, eat < consume, this morning < today, most < some Monotonicity classes for semantic functions Upward monotone: some rats dream < some rodents dream Downward monotone: no rats dream > no rodents dream Non-monotone: most rats dream # most rodents dream Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion +––––++++ Handles even nested inversions of monotonicity Every state forbids shooting game without a hunting license But lacks any representation of exclusion Garfield is a cat < Garfield is not a dog

7. 7 Implicatives & factives Work at PARC, esp. Nairn et al. 2006 Explains inversions & nestings of implicatives & factives Ed did not forget to force Dave to leave  Dave left Defines 9 implication signatures “Implication projection algorithm” Bears some resemblance to monotonicity calculus But, fails to connect to containment or monotonicity John refused to dance  John didn’t tango Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

8. 8Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

9. 9 A theory of natural logic Three elements: 1.an inventory of entailment relations semantic containment relations of Sánchez Valencia plus semantic exclusion relations 2.a concept of projectivity explains entailments compositionally generalizes Sánchez Valencia’s monotonicity classes generalizes Nairn et al.’s implication signatures 3.a weak proof procedure composes entailment relations across chains of edits Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

10. 10 Entailment relations in past work X is a man X is a woman X is a hippo X is hungry X is a fish X is a carp X is a crow X is a bird X is a couch X is a sofa Yes entailment No non-entailment 2-way RTE1,2,3 Yes entailment No contradiction Unknown non-entailment 3-way FraCaS, PARC, RTE4? P = Q equivalence P < Q forward entailment P > Q reverse entailment P # Q non-entailment containment Sánchez-Valencia Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

11. 11 16 elementary set relations Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion ?? ?? QQ PP P Q P and Q can represent sets of entities (i.e., predicates) or of possible worlds (propositions) cf. Tarski’s relation algebra

12. 12 16 elementary set relations Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion ?? ?? QQ PP P Q P ^ Q P _ Q P = Q P > Q P < Q P | QP # Q P and Q can represent sets of entities (i.e., predicates) or of possible worlds (propositions) cf. Tarski’s relation algebra

13. 13 7 basic entailment relations Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion symbol name example2-way3-way P = Q equivalence couch = sofa yes P < Q forward (strict) crow < bird yes P > Q reverse (strict) European > French nounk P ^ Q negation (exhaustive exclusion) human ^ nonhuman no P | Q alternation (non-exhaustive exclusion) cat | dog no P _ Q cover (non-exclusive exhaustion) animal _ nonhuman nounk P # Q independence hungry # hippo nounk

14. 14 Relations for all semantic types The entailment relations are defined for expressions of all semantic types (not just sentences) categorysemantic typeexample(s) common nouns etet penguin < bird adjectives etet tiny < small intransitive verbs etet hover < fly transitive verbs eeteet kick < strike temporal & locative modifiers (e  t)  (e  t) this morning < today in Beijing < in China connectives tttttt and < or quantifiers (e  t)  t (e  t)  (e  t)  t everyone < someone all < most < some Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

15. 15 Monotonicity of semantic functions Upward-monotone (  M) The default: “bigger” inputs yield “bigger” outputs Example: broken. Since chair  furniture, broken chair  broken furniture Heuristic: in a  M context, broadening edits preserve truth Downward-monotone (  M) Negatives, restrictives, etc.: “bigger” inputs yield “smaller” outputs Example: doesn’t. While hover  fly, doesn’t fly  doesn’t hover Heuristic: in a  M context, narrowing edits preserve truth Non-monotone (#M) Superlatives, some quantifiers ( most, exactly n ): neither  M nor  M Example: most. While penguin  bird, most penguins # most birds Heuristic: in a #M context, no edits preserve truth Sánchez Valencia’s monotonicity calculus assigns semantic functions to one of three monotonicity classes: Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

16. 16 Downward monotonicity at most two people < at most two men restrictive quantifiers: no, few, at most n prohibit weapons < prohibit guns negative & restrictive verbs: lack, fail, prohibit, deny without clothes < without pants prepositions & adverbs: without, except, only drug ban < heroin ban negative & restrictive nouns: ban, absence [of], refusal If stocks rise, we’ll get real paid < If stocks soar, we’ll get real paid the antecedent of a conditional didn’t dance < didn’t tango explicit negation: no, n’t Downward-monotone constructions are widespread! Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

17. 17 Generalizing to projectivity Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion How do the entailments of a compound expression depend on the entailments of its parts? How does the entailment relation between (f x) and (f y) depend on the entailment relation between x and y (and the properties of f)? Monotonicity gives partial answer (for =,, #) But what about the other relations (^, |, _)? We’ll categorize semantic functions based on how they project the basic entailment relations

18. 18 downward monotonicity Example: projectivity of not Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion swaps these too projectionexample =  = not happy = not glad <  > didn’t kiss > didn’t touch >  < isn’t European < isn’t French #  # isn’t swimming # isn’t hungry ^  ^ not human ^ not nonhuman |  _ not French _ not German _  | not more than 4 | not less than 6

19. 19 switch blocks, not swaps downward monotonicity Example: projectivity of refuse Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion projectionexample =  = <  > refuse to tango > refuse to dance >  < #  # ^  | refuse to stay | refuse to go |  # refuse to tango # refuse to waltz _  #

20. 20 Projecting entailment relations upward Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Nobody can enter without a shirt < Nobody can enter without clothes < < > > > Assume idealized semantic composition trees Propagate lexical entailment relations upward, according to projectivity class of each node on path to root a shirtnobodycanwithoutenter @ @ @ @ clothesnobodycanwithoutenter @ @ @ @

21. 21 A weak proof procedure 1.Find sequence of edits connecting P and H Insertions, deletions, substitutions, … 2.Determine lexical entailment relation for each edit Substitutions: depends on meaning of substituends Deletions: < by default: red socks < socks But some deletions are special: not hungry ^ hungry Insertions are symmetric to deletions: > by default 3.Project up to find entailment relation across each edit 4.Compose entailment relations across sequence of edits Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

22. 22 Composing entailment relations Relation composition: if a R b and b S c, then a ? C cf. Tarski’s relation algebra Many compositions are intuitive = º =  = < º <  < < º =  < ^ º ^  = Some less obvious, but still accessible | º ^  < fish | human, human ^ nonhuman, fish < nonhuman But some yield unions of basic entailment relations! | º |  {=,, |, #} (i.e. the non-exhaustive relations) Larger unions convey less information (can approx. with #) This limits power of proof procedure described  Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

23. 23 Implicatives & factives Nairn et al. 2006 define nine implication signatures These encode implications (+, –, o) in + and – contexts Refuse has signature –/o: refuse to dance implies didn’t dance didn’t refuse to dance implies neither danced nor didn’t dance Signatures generate different relations when deleted Deleting –/o generates | Jim refused to dance | Jim danced Jim didn’t refuse to dance _ Jim didn’t dance Deleting o/– generates Jim didn’t dance (Factives are only partly explained by this account) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

24. 24Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

25. 25 The NatLog system linguistic analysis alignment lexical entailment classification 1 2 3 textual inference problem prediction Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion entailment projection entailment composition 4 5

26. 26 Step 1: Linguistic analysis Tokenize & parse input sentences (future: & NER & coref & …) Identify items w/ special projectivity & determine scope Problem: PTB-style parse tree  semantic structure! No state completely forbids casino gambling DT NNS RB VBD NN NN NP ADVP NP VP S +++––– Solution: specify scope in PTB trees using Tregex No  forbid  state completely casino gambling no pattern: DT < /^[Nn]o$/ arg1:  M on dominating NP __ >+(NP) (NP=proj !> NP) arg2:  M on dominating S __ > (S=proj !> S) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

27. 27 Step 2: Alignment Phrase-based alignments: symmetric, many-to-many Can view as sequence of atomic edits: DEL, INS, SUB, MAT Ordering of edits defines path through intermediate forms Need not correspond to sentence order Ordering of some edits can influence effective projectivity for others We use heuristic reordering of edits to simplify this Decomposes problem into atomic entailment problems We haven’t (yet) invested much effort here Experimental results use alignments from other sources Few states completely forbid casino gambling Few states have completely prohibited gambling MAT SUBMATINSDEL Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

28. 28 Running example Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion P Jimmy Dean refused to movewithoutbluejeans H James Dean didn’tdancewithoutpants edit index 12345678 edit type SUBDELINS SUBMATDELSUB OK, the example is contrived, but it compactly exhibits containment, exclusion, and implicativity

29. 29 Step 3: Lexical entailment classification Predict basic entailment relation for each edit, based solely on lexical features, independent of context Feature representation: WordNet features: synonymy, hyponymy, antonymy Other relatedness features: Jiang-Conrath (WN-based), NomBank String and lemma similarity, based on Levenshtein edit distance Lexical category features: prep, poss, art, aux, pron, pn, etc. Quantifier category features Implication signatures (for DEL edits only) Decision tree classifier Trained on 2,449 hand-annotated lexical entailment problems >99% accuracy on training data — captures relevant distinctions Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

30. 30 Running example Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion P Jimmy Dean refused to movewithoutbluejeans H James Dean didn’tdancewithoutpants edit index 12345678 edit type SUBDELINS SUBMATDELSUB lex feats strsim = 0.67 implic: +/o cat:au x cat:ne g hypohyper lex entrel =|=^>=<<

31. 31 Step 4: Entailment projection Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion P Jimmy Dean refused to movewithoutbluejeans H James Dean didn’tdancewithoutpants edit index 12345678 edit type SUBDELINS SUBMATDELSUB lex feats strsim = 0.67 implic: +/o cat:au x cat:ne g hypohyper lex entrel =|=^>=<< project -ivity  atomic entrel =|=^<=<<

32. 32 interestingfinal answer Step 5: Entailment composition Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion P Jimmy Dean refused to movewithoutbluejeans H James Dean didn’tdancewithoutpants edit index 12345678 edit type SUBDELINS SUBMATDELSUB lex feats strsim = 0.67 implic: +/o cat:au x cat:ne g hypohyper lex entrel =|=^>=<< project -ivity  atomic entrel =|=^<=<< compo -sition =||<<<<<

33. 33Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

34. 34 The FraCaS test suite FraCaS: mid-90s project in computational semantics 346 “textbook” examples of textual inference problems examples on next slide 9 sections: quantifiers, plurals, anaphora, ellipsis, … 3 possible answers: yes, no, unknown (not balanced!) 55% single-premise, 45% multi-premise (excluded) Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

35. 35 FraCaS examples P No delegate finished the report. H Some delegate finished the report on time. no P At most ten commissioners spend time at home. H At most ten commissioners spend a lot of time at home. yes P Either Smith, Jones or Anderson signed the contract. H Jones signed the contract. unk P Dumbo is a large animal. H Dumbo is a small animal. no P ITEL won more orders than APCOM. H ITEL won some orders. yes P Smith believed that ITEL had won the contract in 1992. H ITEL won the contract in 1992. unk Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

36. 36 high precision even outside areas of expertise 27% error reduction in largest category, all but one correct high accuracy in sections most amenable to natural logic Results on FraCaS Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion System#prec %rec %acc % most common class18355.7100.055.7 MacCartney & M. 0718368.960.859.6 this work18389.365.770.5 §Category#prec %rec %acc % 1Quantifiers4495.2100.097.7 2Plurals2490.064.375.0 3Anaphora6100.060.050.0 4Ellipsis25100.05.324.0 5Adjectives1571.483.380.0 6Comparatives1688.9 81.3 7Temporal3685.770.658.3 8Verbs880.066.762.5 9Attitudes9100.083.388.9 1, 2, 5, 6, 910890.485.587.0

37. 37 FraCaS confusion matrix Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion yesnounktotal yes67431102 no116421 unk774660 total752781183 guess gold

38. 38Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

39. 39 The RTE3 test suite RTE: more “natural” textual inference problems Much longer premises: average 35 words (vs. 11) Binary classification: yes and no RTE problems not ideal for NatLog Many kinds of inference not addressed by NatLog: paraphrase, temporal reasoning, relation extraction, … Big edit distance  propagation of errors from atomic model Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

40. 40 RTE3 examples Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion P As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuela’s populist president is using an energy windfall to win friends and promote his vision of 21st-century socialism. H Hugo Chávez acts as Venezuela's president. yes P Democrat members of the Ways and Means Committee, where tax bills are written and advanced, do not have strong small business voting records. H Democrat members had strong small business voting records. no (These examples are probably easier than average for RTE.)

41. 41 Results on RTE3 data Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion systemdata% yesprec %rec %acc % RTE3 best (LCC)test80.0 RTE3 average top 5test71.0 RTE3 average all 26test61.7 NatLogdev22.573.932.359.3 test26.470.136.159.4 (each data set contains 800 problems) Accuracy is unimpressive, but precision is relatively high Maybe we can achieve high precision on a subset? Strategy: hybridize with broad-coverage RTE system As in Bos & Markert 2006

42. 42 A simple bag-of-words model Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Dogshatefigs Dogs do n’t like fruit max1.000.250.40 maxIDF P(p|H ) P(P|H ) 1.000.431.00 0.670.110.96 0.330.050.950.43 0.25 0.71 0.400.460.66 IDF0.430.550.80 P(h|P)1.000.470.48 P(H|P ) 0.23 1.000.000.33 0.670.00 0.330.250.00 0.25 0.00 0.40 similarity scores on [0, 1] for each pair of words (I used a really simple-minded similarity function based on Levenshtein string-edit distance) max sim for each hyp word how rare each word is = (max sim)^IDF =  h P(h|P) P H

43. 43 Results on RTE3 data Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion systemdata% yesprec %rec %acc % RTE3 best (LCC)test80.0 RTE3 average top 5test71.0 RTE3 average all 26test61.7 NatLogdev22.573.932.359.3 test26.470.136.159.4 BoW (bag of words)dev50.670.168.9 test51.262.470.063.0 (each data set contains 800 problems) +10 probs

44. 44 Combining BoW & NatLog Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion MaxEnt classifier BoW features: P(H|P), P(P|H) NatLog features: 7 boolean features encoding predicted entailment relation

45. 45 Results on RTE3 data Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion systemdata% yesprec %rec %acc % RTE3 best (LCC)test80.0 RTE3 average top 5test71.0 RTE3 average all 26test61.7 NatLogdev22.573.932.359.3 test26.470.136.159.4 BoW (bag of words)dev50.670.168.9 test51.262.470.063.0 BoW + NatLogdev50.771.470.470.3 test56.163.069.063.4 (each data set contains 800 problems) +11 probs +3 probs

46. 46 Problem: NatLog is too precise? Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Error analysis reveals a characteristic pattern of mistakes: Correct answer is yes Number of edits is large (>5) (this is typical for RTE) NatLog predicts < or = for all but one or two edits But NatLog predicts some other relation for remaining edits! Most commonly, it predicts > for an insertion (e.g., RTE3_dev.71) Result of relation composition is thus #, i.e. no Idea: make it more forgiving, by adding features Number of edits Proportion of edits for which predicted relation is not < or =

47. 47 Results on RTE3 data Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion systemdata% yesprec %rec %acc % RTE3 best (LCC)test80.0 RTE3 average top 5test71.0 RTE3 average all 26test61.7 NatLogdev22.573.932.359.3 test26.470.136.159.4 BoW (bag of words)dev50.670.168.9 test51.262.470.063.0 BoW + NatLogdev50.771.470.470.3 test56.163.069.063.4 BoW + NatLog + otherdev52.770.972.670.5 test58.763.072.264.0 (each data set contains 800 problems) +13 probs +8 probs

48. 48Outline Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

49. 49 What natural logic can’t do Not a universal solution for textual inference Many types of inference not amenable to natural logic Paraphrase: Eve was let go = Eve lost her job Verb/frame alternation: he drained the oil < the oil drained Relation extraction: Aho, a trader at UBS… < Aho works for UBS Common-sense reasoning: the sink overflowed < the floor got wet etc. Also, has a weaker proof theory than FOL Can’t explain, e.g., de Morgan’s laws for quantifiers: Not all birds fly = Some birds don’t fly Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion

50. 50 What natural logic can do Natural logic enables precise reasoning about containment, exclusion, and implicativity, while sidestepping the difficulties of translating to FOL. The NatLog system successfully handles a broad range of such inferences, as demonstrated on the FraCaS test suite. Ultimately, open-domain textual inference is likely to require combining disparate reasoners, and a facility for natural logic is a good candidate to be a component of such a system. Future work: phrase-based alignment for textual inference :-) Thanks! Questions? Introduction Foundations of Natural Logic The NatLog System Experiments with FraCaS Experiments with RTE Conclusion