Relation Alignment for Textual Entailment Recognition Cognitive Computation Group, University of Illinois Experimental ResultsTitle Mark Sammons, V.G.Vinod Vydiswaran, Tim Vieira, Nikhil Johri, Ming-Wei Chang, Dan Goldwasser, Vivek Srikumar, Gourab Kundu, Yuancheng Tu, Kevin Small, Joshua Rule, Quang Do, Dan Roth  Text The RATER System The RATER system was trained using the RTE5 Development corpus and evaluated on the RTE5 Test corpus. We compare the system’s performance against a ‘smart’ lexical baseline that uses WordNet-based similarity resources. In addition, we carried out an ablation study with three versions of the system: without WordNet-based resources (“No WN”); without Named Entity resources (“No NE”); and with simple Named Entity similarity (“Basic NE”). After the submission deadline, we augmented the shallow semantic predicates in the full system using Coreference information to create predicates spanning multiple sentences (“+Coref”). Table 1 shows the performance of these variants of the system on the Development corpus, while table 2 shows the results on the Test corpus. Performance is consistent with the expected behavior of the system; as semantic resources are removed, system performance declines. Wordnet (Miller et al. 1990), Named Entity (Ratinov and Roth, 2009), and Coreference (Bengtson and Roth, 2009) each make a significant contribution to overall performance. Table 1: RTE5 2-way Task Results (Dev. Corpus)  Identify clear roles for Alignment in Textual Entail- ment systems: filter and decider  Propose an alignment framework to leverage focused knowledge resources, avoid canonization Contributions RTE5 Test SystemAllQAIEIR Baseline Submtd* No NE* Basic NE No WN Coref RTE5 Development SystemAllQAIEIR Baseline Submtd* No NE* Basic NE No WN Coref Recognizing Textual Entailment The task of Recognizing Textual Entailment frames Natural Language Text understanding as recognizing when two text spans express the same meaning. In the example below, the text span ‘T’ contains the meaning of the text span ‘’H’, so a successful RTE system would say that T entail s H. T: The Shanghai Co-operation Organization (SCO), is a fledgling association that binds Russia, China and four other nations. H: China is a member of SCO. Most successful systems share a basic assumption: that semantics is largely compositional, meaning that we can combine the results of local entailment decisions to reach a global decision. Many systems share the same basic architecture: 1.Preprocess the TE pair with a range of NLP tools 2.Determine some structure over each sentence in the Entailment pair 3.Align some level of structure in the Hypothesis with structure in the Text 4.Either: directly compute entailment result based on alignment (either online or in batch mode) OR: Extract features using alignment (and possibly other resources), and determine the label of the TE pair based on this feature representation. (Zanzotto et al. 2006) take the first approach, computing the ‘best’ alignment for each pair, then learning a classifier over all aligned pairs in a corpus, thereby using alignment directly to determine the entailment label. Others, such as (Hickl et al. 2007, de Marneffe et al 2008) use alignment as a filtering step to select among possible feature sources. (Zanzotto and Moschitti 2006). explain their alignment as capturing valid and invalid syntactic transformations across many entailment pairs. (de Marneffe et al.) propose an alignment task that is separate from the entailment decision, in which elements in the Hypothesis are paired with the most relevant elements of the Text. We believe that Alignment is a valuable inference framework in RTE, but found problems with existing approaches when we tried to incorporate new analysis and comparison resources. In the present work, we share our insights about the Alignment process and its relation to Textual Entailment decisions. The RATER system first annotates entailment pairs with a suite of NLP analytics, generating a multi-view representation mapping each analysis to the original text. Resource-specific metrics are then used to compare constituents in each (T,H) paired view (e.g., NE metrics are used to compare constituents in the T, H Named Entity views) to build a match graph. An Aligner then selects edges from these graphs (see panel below). Features are then extracted over the resulting set of alignments, and used to train a classifier which is used to label examples. Table 2: RTE5 2-way Task Results (Test Corpus) Selected References Marie-Catherine de Marneffe, Trond Grenager, Bill MacCartney, Daniel Cer, Daniel Ramage, Chloe Kiddon, and Christopher D. Manning: Aligning semantic graphs for textual inference and machine reading. In AAAI Spring Symposium at Stanford, Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi: Recognizing textual entailment with LCC’s groundhog system. In Proc. of the 2nd PASCAL Challenges Workshop on Recognizing Textual Entailment, Fabio Massimo Zanzotto and Alessandro Moschitti: Automatic learning of textual entailments with cross-pair similarities. In Proceedings of the 21st Intl. Conf. on Computational Linguistics and 44th Annual Meeting of the ACL, L. Ratinov and D. Roth: Design challenges and misconceptions in named entity recognition. In Proc. of CoNLL E. Bengtson and D. Roth: Understanding the value of features for coreference resolution, in EMNLP In the alignment step, instead of aligning only a single shallow or unified representation (as previous alignment systems have done), RATER divides the set of views in to groups, and computes a separate alignment for each group (groups contain analysis sources for which the comparison metrics share a common output scale). Within each alignment, RATER selects the edges that maximize match score while minimizing the distance of mapped constituents in the text from each other; the objective function is given below. The selected constituents of H must respect the constraint that each token in H may be mapped to at most one token in T. Alignment over Multiple Views Figure 3: Example showing multiple alignments over different views of the entailment pair Figure 2: Objective function for Alignment Figure 1: Architecture of the RATER system