We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byGraham Voshall
Modified about 1 year ago
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use Evaluation and STS Workshop on a Pipeline for Semantic Text Similarity (STS) March 12, 2012 Sherri Condon The MITRE Corporation
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use Page 2 Evaluation and STS Wish-list ■Valid: measure what we think we’re measuring (definition) ■Replicable: same results for same inputs (annotator agreement) ■Objective: no confounding biases (from language or annotator) ■Diagnostic –Not all evaluations achieve this –Understanding factors and relations ■Generalizable –Makes true predictions about new cases –Functional: if not perfect, good enough ■Understandable: –Meaningful to stakeholders –Interpretable components ■Cost effective
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use ■Meaning as extension: same/similar denotation –Anaphora/coreference and time/date resolution –The evening star happens to be the morning star –“Real world” knowledge = true in this world ■Meaning as intension –Truth (extension) in the same/similar possible worlds –Compositionality: inference and entailment ■Meaning as use –Equivalence for all the same purposes in all the same contexts –“Committee on Foreign Affairs, Human Rights, Common Security and Defence Policy” vs “Committee on Foreign Affairs” –Salience, application specificity, implicature, register, metaphor ■Yet remarkable agreement in intuitions about meaning Quick Foray into Philosophy Page 3
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use ■Computer vision through a human lens –Recognize events in video as verbs –Produce text descriptions of events in video ■Comparing human descriptions to system descriptions raises all the STS issues –Salience/importance/focus (A gave B a package. They were standing.) –Granularity of description (car vs. red car, woman vs. person) –Knowledge and inference (standing with motorcycle vs. sitting on motorcycle: motorcycle is stopped) –Unequal text lengths ■Demonstrates value of/need for understanding these factors DARPA Mind’s Eye Evaluation Page 4
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use ■Basic similarity scores based on dependency parses –Scores increase for matching predicates and arguments –Scores decrease for non-matching predicates and arguments –Accessible syn-sets and ontological relations expand matches ■Salience/importance –Obtain many “reference” descriptions –Weight predicates and arguments based on frequency ■Granularity of description –Demonstrates influence of application context –Program focus is on verbs, so nouns match loosely ■Regularities in evaluation efforts that depend on semantic similarity promise common solutions (This is work with Evelyne Tzoukermann and Dan Parvaz) Mind’s Eye Text Similarity Page 5
© 2012 The MITRE Corporation. All rights reserved. For Internal MITRE Use Predicate ArgumentWeight123456 dobjclosecan1x dobjcloseit3 dobjholdlid6x dobjplaceit1 dobjplacelid2 dobjputit1xxx dobjputlid5x dobjtakelid1 nsubjclosewoman1 nsubjholdwoman2x Nominal errors0 33 5 Verbal Errors01116 Predicate errors1.5053239 Total Score15-6.52.51-4-14 Test Sentences Page 6
11 Thoughts on STS regarding Machine Reading Ralph Weischedel 12 March 2012.
Using Semantic Relations to Improve Passage Retrieval for Question Answering Tom Morton.
Knowledge Structure Vijay Meena ( ) Gaurav Meena ( )
Lecture 1 Ling 442. What is meaning? Meaning – what the speaker wants to convey Literal meaning vs. pragmatic meaning (1)I forgot the paper. Literal meaning:
“You cannot teach anybody anything. You can only help them discover it within themselves.” ~~ Galileo.
State Testing Updates. 5 th and 8 th Grade Writing Assessments Will be given sometime in April Will be passage-based Will involve two tests per grade.
Testing 05 Test Methods. Considerations in Test Methods Like traits tested, test methods also affect test performance. Test methods: features of the test.
Meaning. Deictics Are words, phrases and features of grammar that have to be interpreted in relation to the situation in which they are uttered such.
Applying Educational Research A Practical Guide EdAd 692 Research in Educational Leadership.
Knowledge Representation Use of logic. Artificial agents need Knowledge and reasoning power Can combine GK with current percepts Build up KB incrementally.
Ambiguity, Generality, and Definitions Sign In Review: Vagueness and Semantic Ambiguity Syntactic Ambiguity Arguments and Ambiguity Definitions For next.
Understanding Naturally Conveyed Explanations of Device Behavior Michael Oltmans and Randall Davis MIT Artificial Intelligence Lab.
Meaning and Language Part 1. Plan We will talk about two different types of meaning, corresponding to two different types of objects: –Lexical Semantics:
Specialized models and ranking for coreference resolution Pascal Denis ALPAGE Project Team INRIA Rocquencourt F Le Chesnay, France Jason Baldridge.
MITRE © 2001 The MITRE Corporation. ALL RIGHTS RESERVED. What Works, What Doesn’t -- And What Needs to Work Lynette Hirschman Information Technology Center.
The meaning of Language Chapter 5 Semantics and Pragmatics Week10 Nov.19 th -23 rd.
Discussion of open issues. Cases of meaning equivalence Creative Language use – Metaphorical language Al7ayah ba2a lwnha bamby 5ales [Egyptian Arabic.
Day 3. Standards Reading: 1.0 Word Analysis, Fluency, and Systematic Vocabulary Development- Students apply their knowledge of word origins to determine.
Grammars Grammars can get quite complex, but are essential. Syntax: the form of the text that is valid Semantics: the meaning of the form – Sometimes semantics.
Summarizing Encyclopedic Term Descriptions on the Web from Coling 2004 Atsushi Fujii and Tetsuya Ishikawa Graduate School of Library, Information and Media.
COGEX at the Second RTE Marta Tatu, Brandon Iles, John Slavick, Adrian Novischi, Dan Moldovan Language Computer Corporation April 10 th, 2006.
For Monday Read chapter 26 Last Homework –Chapter 23, exercise 7.
Chapter 5 The Meaning of Language Don’t Be Afraid..!
1 1 Introduction: Terminological Remarks. 2 Language, Mind, World Famous triangle: Famous triangle:Language Mind World How does the mind relate to the.
LECTURE 2: SEMANTICS IN LINGUISTICS ENGLISH STUDY PROGRAM FKIP – SRIWIJAYA UNIVERSITY 1.
Annotating and measuring Temporal relations in texts Philippe Muller and Xavier Tannier IRIT,Université Paul Sabatier COLING 2004.
Event-Centric Summary Generation Lucy Vanderwende, Michele Banko and Arul Menezes One Microsoft Way, WA, USA DUC 2004.
The interface between model-theoretic and corpus-based semantics Sebastian Pado.
Prefix Definition: in front, against, in the way of.
Words Words Words! Helping ELL Students Develop Vocabulary.
For Monday Read chapter 26 Homework: –Chapter 23, exercises 8 and 9.
Using Information Extraction for Question Answering Done by Rani Qumsiyeh.
Click the globe beside the matching definition.
Newmark: semantic & communicative translation. Notable Opinions 1. The success of equivalent effect is ‘illusory’ 2. The most crucial problem in translation.
1 First Order Logic CS 171/271 (Chapter 8) Some text and images in these slides were drawn from Russel & Norvig’s published material.
© McGraw-Hill Higher Education. All rights reserved. Chapter 4 Validity.
An Approach to Evaluate Data Trustworthiness Based on Data Provenance Department of Computer Science Purdue University.
March 1, 2009 Dr. Muhammed Al-Mulhem 1 ICS 482 Natural Language Processing Semantics (Chapter 17) Muhammed Al-Mulhem March 1, 2009.
PHILOSOPHY OF LANGUAGE Some topics and historical issues of the 20 th century.
Using Semantic Relations to Improve Information Retrieval Tom Morton.
Indirect Supervision Protocols for Learning in Natural Language Processing II. Learning by Inventing Binary Labels This work is supported by DARPA funding.
Illinois-Coref: The UI System in the CoNLL-2012 Shared Task Kai-Wei Chang, Rajhans Samdani, Alla Rozovskaya, Mark Sammons, and Dan Roth Supported by ARL,
Incident Threading for News Passages (CIKM 09) Speaker: Yi-lin,Hsu Advisor: Dr. Koh, Jia-ling. Date:2010/06/14.
Programming Languages and Design Lecture 3 Semantic Specifications of Programming Languages Instructor: Li Ma Department of Computer Science Texas Southern.
Short Text Similarity with Word Embedding Date: 2016/03/28 Author: Tom Kenter, Maarten de Rijke Source: CIKM’15 Advisor: Jia-Ling Koh Speaker: Chih-Hsuan.
Lecture 3 Ling 442. exercises 1.What is the difference between implicature and entailment? 2.What is the difference between presuppositions and entailments?
Defining Discourse. What is Discourse? Consider the following definitions of discourse: 1.Discourse is language above the sentence or above the clause.
CS305: HCI in SW Development Evaluation (Return to…) We’ve had an introduction to evaluation. Now for more details on…
For Monday Read chapter 24, sections 1-3 Homework: –Chapter 23, exercise 8.
Empirical Methods in Information Extraction Claire Cardie Appeared in AI Magazine, 18:4, Summarized by Seong-Bae Park.
© 2017 SlidePlayer.com Inc. All rights reserved.