Recognising Textual Entailment Johan Bos School of Informatics University of Edinburgh Scotland,UK

Textual Entailment Example 1 (TRUE) Text: His family has steadfastily denied the charges. Hypothesis: The charges were denied by his family.

Textual Entailment Example 2 (TRUE) Text: In 1998, the general Assembly of the Nippon Sei Ko Kai (Anglican Church in Japan) voted to accept female priests. Hypothesis: The Anglican church in Japan approved the ordination of women.

Textual Entailment Example 3 (FALSE) Text: The city Tenochtitlan grew rapidly and was the center of the Aztec’s great empire. Hypothesis: Tenochtitlan quickly spread over the island, marshes, and swamps.

Textual Entailment Example 4 (FALSE) Text: Clinton’s new book is not big seller here. Hypothesis: Clinton’s book is a big seller.

Approach in a Nutshell Compute semantic representations for Text and Hypothesis Use logical inference (theorem proving) to determiner whether T entails H Compare this with a shallow approach (word overlap) Use Machine Learning to combine logical inference with shallow word overlap

Talk Outline Compositional Semantics –Discourse Representation Theory –Combinatorial Categorial Grammar –Lambda Calculus as “glue” Inference –Theorem Proving –Model Building –Approximating Entailment Evaluation

Use dataset (training and test set) of the RTE challenge organised by PASCAL network Baseline is 50% Dataset is based on different tasks –CD (Comparable documents) –QA (Question answering) –IE (Information extraction) –MR (machine translation –RC (reading comprehension –PP (paraphrase acquisition –IR (information retrieval)

Machine Learning Each entailment example pair is expressed as a feature vector Train a decision tree for classification into TRUE and FALSE WEKA (Witten & Frank 2000) Test on the test set Evaluation measures: –Accuracy (% correct judgements) –CWS (confidence-weighted score)

Features Shallow features –Word overlap between text and hypothesis –Length of text and hypothesis Deep semantic features –uninformative/inconsistent (theorem prover) –Domainsize, modelsize, and absolute and relative differences between text and hypothesis (model builder)

Results AccuracyCWS Shallow Deep Hybrid (S+D) Hybrid+Task

Conclusions Hybrid approach combines shallow analysis with both theorem proving and model building and achieves high accuracy scores (compared to other systems) Need more work on computing appropriate background knowledge Future work also includes task-based evaluation, for instance in a QA system

Acknowledgements Joined work with –James Curran (Sydney University) –Steve Clark (University of Oxford) –Katja Markert (University of Leeds) –Patrick Blackburn (LORIA, Nancy)