1 RTE April 10, 2006 An approach based on Logic Forms and WordNet relationships to Textual Entailment performance O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar GPLSI – Natural Language Processing and Information Systems Group

2 Index System Architecture Derivation of the Logic Forms Computation of Similarity Measures between Logic Forms Result Analysis Conclusions and Future Work

3 Text Derivation of the Logic Forms System Architecture Hypothesis LF HypothesisLF Text Computation of similarity measures between Logic Forms Entailment? YESNO score

4 Derivation of the Logic Forms The Logic Forms are derived through an analysis of dependency relationships between the words of the sentence Employs a set of rules that infer several aspects such as the assert, its type, its identifier and the relationships between the different asserts in the logic form The Logic Forms are based on the logic form format defined in the eXtended WordNet

5 As an example “A shark attacked a human being” Dependency tree Logic Form shark:NN(x1) attack:VB(e1,x1,x3) human:NN(x2) NNC(x3,x2,x4) being:NN(x4) a:Det shark:N attack:V human:U being:N S det Lex-mod obj det a:Det Derivation of the Logic Forms

6 Computation of Similarity Measures between LF The method Focused on the entailment between the verbs (verbs generally govern the meaning of sentences) Firstly analyses the relation between the verbs of the two logic forms derived from the text and the hypothesis Secondly, if there is a relation between the verbs, then the method will analyse the similarity relations between all predicates which depending on the two verbs

7 simWeight = 0 Tvb = obtainVerbs(T) Hvb = obtainVerbs(H) for i = 0... size(Tvb) do for j = 0... size(Hvb) do if calcSim(Tvb(i),Hvb(j)) > 0 then simWeight += calcSim(Tvb(i),Hvb(j)) Telem = obtainElem(Tvb(i)) Helem = obtainElem(Hvb(j)) simWeight += calcSim(Telem,Helem) end if end for if simWeight > threshold then return TRUE else return FALSE end if Computation of Similarity Measures between LF

8 In order to obtain the similarity between the predicates of the logic forms (calcSim(x,y)), two approaches have been implemented Based on WordNet relations Based on Lin’s measure A Word Sense Disambiguation module was not employed The first 50% of the WordNet senses were taken into account The threshold, which determines if the text entails the hypothesis, has been obtained empirically using the development data Computation of Similarity Measures between LF

9 Based on WordNet relations For which concept (word#sense): Obtaining the relations among other concepts through the synsets Each relation has an associated weight Synonymy (0.9), Hypernymy (0.8), Hyponymy and Entailment (0.7), Meronymy and Holonymy (0.5) The length of the path that relates the two different concepts must be lower or equal than 4 synsets Computation of Similarity Measures between LF

10 Based on WordNet relations The weight of the path between two different concepts is calculated as the product of the weights associated to the relations connecting the intermediate synsets This weight indicates the relation between two concepts Computation of Similarity Measures between LF

11 Based on WordNet relations Example cable_car#n#3 subway#n#3  0.5*0.8*0.7=0.28 cable_car#n#1 railway#n#3 funicular#n#3 subway#n#3 holonymy Hypernymy Hyponymy Computation of Similarity Measures between LF

12 Based on Lin’s measure Lin’s similarity measure as implemented in WordNet::Similarity Open source software package Lin’s similarity measure augments the information content of the least common subsumer (LCS is the most specific concept that two concepts share as an ancestor) of the two concepts with the sum of the information content of the concepts Computation of Similarity Measures between LF

13 T: Five US soldiers were killed in the capital and insurgents blasted polling stations across the country kill:VB NNC five:NN NNC us:NN soldier:NN Ø in:IN capital:NN and:CC blast:VB insurgent:NN NNC polling:U... An example

14 H: Five US soldiers were killed kill:VB NNC five:NN NNC us:NNsoldier:NN Ø An example

15 kill#v kill#v  1 Verbs of H  killVerbs of T  kill, blast kill NNC five NNC us soldier Ø in capital and blast insurgents NNC polling... kill NNC five NNC ussoldier Ø Relation ~ 1 kill#v blast#v  0,34... Accumulating weights > Threshold  ENTAILMENT An example

16 Results (RTE2 dev&test) Run1 (Lin’s measure) Development data Accuracy: Test data Accuracy: Average Precision: Run2 (WN relations) Development data Accuracy: Test data Accuracy: Average Precision: Result Analysis

17 The empirical threshold of the development data A value of 0.24 Result Analysis

18 Conclusions and Future Work Our system derives the logic forms for the text/hypothesis pair and computes the similarity between them The similarity is computed using two different approaches: Lin’s similarity measure WordNet relation-based similarity Our system provides a score showing the semantic similarity between two logic forms

19 Conclusions and Future Work The run using Lin’s similarity measure achieves better results than the approach based onWordNet relations, both when tested on development, as well as test data This slight loss of accuracy is due to the fact that our WordNet relations approach attempts to establish an objective semantic comparison between the logic forms rather than an entailment relation

20 Conclusions and Future Work As a future work: Performing a deeper study about the most suitable WordNet relations for recognising textual entailment. Perhaps only hypernymy, synonymy and entailment relations between the text and the hypothesis would be more suitable for the entailment phenomenon Testing how other natural language processing tools can help in detecting textual entailment. For example, using a Named Entity Recognizer could help in detecting entailment between two segment of text

21 RTE April 10, 2006 Thank you very much O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar GPLSI – Natural Language Processing and Information Systems Group