Word Sense and Subjectivity (Coling/ACL 2006) Janyce Wiebe Rada Mihalcea University of Pittsburgh University of North Texas Acknowledgements: This slide is created based on the presentation slides from

2/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

3/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

4/33 Motivation Growing interest in the automatic extraction of opinions, emotions, and sentiments in text  subjectivity Continuing interest in word sense Though both are concerned with text meaning, they have mainly been investigated independently

5/33 Subjectivity Labels on Senses Alarm: S: Alarm, dismay, consternation – (fear resulting from the awareness of danger) 驚慌 O: Alarm, warning device, alarm system – (a device that signals the occurrence of some undesirable event) 警報器 Interest: S: Interest, involvement -- (a sense of concern with and curiosity about someone or something; "an interest in music") 愛好 O: Interest -- (a fixed charge for borrowing money; usually a percentage of the amount borrowed; "how much interest do you pay on your mortgage?") 利息

6/33 WSD using Subjectivity Tagging Sense 4 “a sense of concern with and curiosity about someone or something” S Sense 1 “a fixed charge for borrowing money” O WSD System Sense 4 Sense 1? Sense 1 Sense 4? The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest.

7/33 Sense 4 “a sense of concern with and curiosity about someone or something” S Sense 1 “a fixed charge for borrowing money” O WSD using Subjectivity Tagging The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest. WSD System Sense 4 Sense 1? Sense 1 Sense 4? Subjectivity Classifier S O

8/33 Sense 4 “a sense of concern with and curiosity about someone or something” S Sense 1 “a fixed charge for borrowing money” O WSD using Subjectivity Tagging The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest. WSD System Sense 4 Sense 1? Sense 1 Sense 4? Subjectivity Classifier S O

9/33 Subjectivity Classifier Subjectivity Tagging using WSD The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest. O S? S O?

10/33 Subjectivity Classifier S Sense 4 “a sense of concern with and curiosity about someone or something” O Sense 1 “a fixed charge for borrowing money” Subjectivity Tagging using WSD The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest. WSD System Sense 4 Sense 1 O S? S O?

11/33 Subjectivity Classifier S Sense 4 “a sense of concern with and curiosity about someone or something” O Sense 1 “a fixed charge for borrowing money” Subjectivity Tagging using WSD The notes do not pay interest. He spins a riveting plot which grabs and holds the reader’s interest. WSD System Sense 4 Sense 1 O S? S O?

12/33 Goals Explore interactions between word sense and subjectivity Can subjectivity labels be assigned to word senses? Manually Automatically Can subjectivity analysis improve word sense disambiguation? Can word sense disambiguation improve subjectivity analysis?  Future work

13/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

14/33 Subjectivity Tagging Prior art: identify words and phrases associated with subjectivity  subjectivity classification of sentences, clauses, phrases, or word instances in context Here: subjectivity labels are applied to word senses

15/33 Annotation Scheme Assigning subjectivity labels to WordNet senses S: Subjective O: Objective B: Both Annotators are given the synset and its hypernym

16/33 Senses Definitions When the sense is used in a text or conversation … S: expect it to express subjectivity, and expect the phrase/sentence containing it to be subjective. O: don ’ t expect it to express subjectivity and, if the phrase/sentence containing it is subjective, the subjectivity is due to something else. Ex: Will someone shut that darn alarm off? B: covers both subjective and objective usages

17/33 Annotated Data 64 words; 354 senses Balanced subset 32 words; 138 senses  2 judges  Agreement The ambiguous nouns of the SENSEVAL-3 English Lexical Task 20 words; 117 senses  2 judges  WSD w/ subj. info. Others 12 words; 99 senses  1 judge  Auto subjectivity annotation on word sense

18/33 Agreement Results (tag U removed  95%, : 0.90)

19/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

20/33 Main idea Assess the subjectivity of a word sense based on information about the subjectivity of: a set of distributionally similar words (DSW) on unannotated corpus (BNC) in a corpus annotated with subjective expressions (MPQA)

21/33 Algorithm_ diagram Unannotated Corpus (BNC) dsw 1 inst 1 (S) Annotated dsw 1 inst 2 (O) Corpus dsw 2 inst 1 (S) (MPQA) word sense w i alarm w 1 : 驚慌 w 2 : 警報器 DSW = {detector, panic} [-1, 1] [highly objective, highly subjective]

22/33 Algorithm_ pseudo code

23/33 Evaluation No DSW instances for 82 senses  auto assignment of subjectivity for =272 word senses Calculate subj scores for all word senses, sort them, and assign a subj label to the top N word sense  calculate the precision of the algorithm at different points of recall Baseline: random selection of S labels Number of assigned S labels matches number of S labels in the gold standard (recall = 1.0)

24/33 PR curves use only those DSWs that lead to the highest similarity (link)link

25/33 How about the # of DSW? represents the value where precision and recall become equal

26/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

27/33 WSD + Subj. Augment an existing WSD system with a feature reflecting the subjectivity of the context of the ambiguous word Compare the performance of original and subjectivity-aware WSD systems The 20 ambiguous nouns of the SENSEVAL-3 English Lexical Task

28/33 WSD using Subjectivity Tagging Subjectivity Classifier S, O, or B S Original WSD System Subjectivity Aware WSD System Sense 4Sense 1 Sentence i “interest” Sense 1 “a sense of concern with and curiosity about someone or something” Sense 4 “a fixed charge for borrowing money”

29/33 Results Words with Only O Senses Words with S and O Senses the most frequent sense

30/33 Outline Motivation and Goals Assigning Subjectivity Labels to Word Senses Manually Automatically Word Sense Disambiguation using Automatic Subjectivity Analysis Conclusions

31/33 Conclusion Can subjectivity labels be assigned to word senses? Manually Good agreement; Kappa=0.74 Very good when uncertain cases removed; Kappa=0.90 Automatically Method substantially outperforms baseline Showed feasibility of assigning subjectivity labels to the fine-grained level of word senses

32/33 Conclusion Can subjectivity analysis improve word sense disambiguation? Improves performance, but mainly for words with both S and O senses (4.3% error reduction; significant (p < 0.05)) Performance largely remains the same or degrades for words that don ’ t Assign subjectivity labels to WordNet; WSD system should consult WordNet tags to decide when to pay attention to the contextual subjectivity feature.

33/33 Thank you

34/33 Method – Optional Variation w 1 dsw 1 dsw 2 dsw 3 w 2 dsw 1 dsw 2 dsw 3 w 3 dsw 1 dsw 2 dsw 3 if k is in a subjective expression: subj += sim(w i,dsw j ) else: subj -= sim(w i,dsw j ) “Selected” back