1. Linguistic Resources for the 2013 TAC KBP Sentiment SF Evaluation Joe Ellis (presenter), Jeremy Getman, Jonathan Wright, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania, USA

2. 2013 Source Corpus LanguageGenreDocuments English Newswire1,000,257 Web Text 999,999 Discussion Forums 99,063 TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

3. Query Selection  1 - Select queries and reference docs Four slots used in Sentiment Slot Filling positive-towards, positive-from negative-towards, negative-from Sentiment Slot Filling queries comprised of Entity – Slot Rich queries (at least 2-3 instances of sentiment in source corpora) Sentiment defined as a positive or negative emotion, evaluation, or judgment.  2 - Link namestrings to KB or mark as NIL TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

4. Query Selection  1 - Select queries and reference docs Four slots used in Sentiment Slot Filling positive-towards, positive-from negative-towards, negative-from Sentiment Slot Filling queries comprised of Entity – Slot Rich queries (at least 2-3 instances of sentiment in source corpora) Sentiment defined as a positive or negative emotion, evaluation, or judgment.  2 - Link namestrings to KB or mark as NIL TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

5. Annotation  For each query annotator spends up to 2 hours searching corpus for instances of sentiment of the correct directionality and polarity TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Ronnie James Dio neg-from: InfraBlue cavfancier

6. Assessment  Assess validity of fillers & justification from humans & systems Filler Correct – meets the slot requirements and supported in document Wrong – doesn’t meet slot requirements and/or not supported in doc Inexact – otherwise correct, but is incomplete, includes extraneous text, or is not the most informative string in the document Predicate Correct, Wrong, Inexact-Short, Inexact-Long Subject/Object Correct, Wrong, Inexact Ignore TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

7. Justification  Justification is the string(s) of text that show a relation is true Predicate: Includes all three pieces of information necessary to justify the entity/slot/filler relation Subject: proves the entity’s involvement in the relation Object: proves the filler’s involvement in the relation Each part can be comprised of up to two, discontiguous strings Predicate 1: Westboro Baptist Church said they oppose anyone they believe worships Satan. Predicate 2: The fundamentalist church said that includes Ronnie James Dio, who died Sunday. TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 New in 2013: Ronnie James Dio neg-from cavfancier

8. 2013 SSF Discoveries  The reclassification of top-level governments of GPEs as GPEs themselves proved particularly beneficial in Sentiment SF Examples like (1) are much more prevalent than examples like (2) (1) The Palestinian government has denounced what it calls the Israeli army's 'current practice of shoot now and ask questions later.‘ (2) We're kinda like David Hasselhoff; where we're big in Germany, but nobody else cares. Especially useful since actions as indicators of sentiment were invalid e.g. - Israel launched an air strike against Syria.  -Towards slots significantly less productive than -From slots Most annotatable sentiment is in discussion forum But post authors can only be fillers, not query entities Neither NIL or non-NIL – UNKNOWN! TAC KBP Evaluation Workshop – NIST, November 18-19, 2013

9. 2013 SSF Discoveries  and tags seemed to be difficult to parse Found what I think is some backwards speech on the fade-out at the very end of Ronnie James Dio's Holy Diver album. Anyone? Just listened and can't tell. Great album, though. Love that guy.  Seemingly difficult to extract succinct justification from DF docs Predicate strings consisting of entire posts: Elizabeth Warren's a senator from Massachusetts. She's a great American patriot. No one fights harder for the middle class. More ‘Ignore’ assessments in SSF than in other tasks TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Ronnie James Dio pos-from Monketey Ghost ? beachnut ? Discussion Forums

10. Delivered 2013 Resources TAC KBP Evaluation Workshop – NIST, November 18-19, 2013 Corpus TitleTypeLDC CatalogLanguageSize TAC 2013 KBP English Sentiment Slot Filling Training Queries and Annotations TrainingLDC2013E78English160 Queries TAC 2013 KBP English Sentiment Slot Filling Evaluation Queries and Annotations EvaluationLDC2013E89English160 Queries TAC 2013 KBP English Sentiment Slot Filling Evaluation Assessment Results EvaluationLDC2013E100English 5,160 Assessments