1. Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis Theresa Wilson Janyce Wiebe Paul Hoffmann University of Pittsburgh

2. HLT-EMNLP 20052 Introduction Sentiment analysis task of identifying positive and negative opinions, emotions, and evaluations  How detailed? Depends on the application. Flame detection, review classification  document-level analysis Question answering, review mining  sentence or phrase-level analysis

3. HLT-EMNLP 20053 Question Answering Example African observers generally approved of his victory while Western Governments denounced it. Q: What is the international reaction to the reelection of Robert Mugabe as President of Zimbabwe?

4. HLT-EMNLP 20054 Most approaches use a lexicon of positive and negative words Prior polarity: out of context, positive or negative beautiful  positive horrid  negative A word may appear in a phrase that expresses a different polarity in context Contextual polarity “Cheers to Timothy Whitfield for the wonderfully horrid visuals.” Prior Polarity versus Contextual Polarity

5. HLT-EMNLP 20055 Example Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.

6. HLT-EMNLP 20056 Example Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable.

7. HLT-EMNLP 20057 Philip Clap, President of the National Environment Trust, sums up well the general thrust of the reaction of environmental movements: there is no reason at all to believe that the polluters are suddenly going to become reasonable. Example prior polarity Contextual polarity

8. HLT-EMNLP 20058 Goal of Our Research Automatically distinguish prior and contextual polarity

9. HLT-EMNLP 20059 Approach Use machine learning and variety of features Achieve significant results for a large subset of sentiment expressions Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

10. HLT-EMNLP 200510 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions

11. HLT-EMNLP 200511 Manual Annotations Need: sentiment expressions with contextual polarity  positive and negative expressions of emotions, evaluations, stances Had: subjective expression annotations in MPQA Opinion Corpus http://nrrc.mitre.org/NRRC/publications.htm http://nrrc.mitre.org/NRRC/publications.htm  words/phrases expressing emotions, evaluations, stances, speculations, etc. sentiment expressions  subjective expressions Decision: annotate subjective expressions in MPQA Corpus with their contextual polarity

12. HLT-EMNLP 200512 Annotation Scheme Mark polarity of subjective expressions as positive, negative, both, or neutral African observers generally approved of his victory while Western governments denounced it. Besides, politicians refer to good and evil … Jerome says the hospital feels no different than a hospital in the states. positive negative both neutral

13. HLT-EMNLP 200513 Annotation Scheme Judge the contextual polarity of sentiment ultimately being conveyed They have not succeeded, and will never succeed, in breaking the will of this valiant people. positive

14. HLT-EMNLP 200514 Agreement Study 10 documents with 447 subjective expressions Kappa: 0.72 (82%) Remove uncertain cases  at least one annotator marked uncertain (18%) Kappa: 0.84 (90%) (But all data included in experiments)

15. HLT-EMNLP 200515 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

16. HLT-EMNLP 200516 Corpus 425 documents from MPQA Opinion Corpus 15,991 subjective expressions in 8,984 sentences Divided into two sets Development set 66 docs / 2,808 subjective expressions Experiment set 359 docs / 13,183 subjective expressions Divided into 10 folds for cross-validation

17. HLT-EMNLP 200517 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

18. HLT-EMNLP 200518 Prior-Polarity Subjectivity Lexicon Over 8,000 words from a variety of sources Both manually and automatically identified Positive/negative words from General Inquirer and Hatzivassiloglou and McKeown (1997) All words in lexicon tagged with: Prior polarity: positive, negative, both, neutral Reliability: strongly subjective (strongsubj), weakly subjective (weaksubj)

19. HLT-EMNLP 200519 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions

20. HLT-EMNLP 200520 Experiments  Give each instance its own label Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Both Steps: BoosTexter AdaBoost.HM 5000 rounds boosting 10-fold cross validation

21. HLT-EMNLP 200521 Definition of Gold Standard Given an instance inst from the lexicon: if inst not in a subjective expression: goldclass( inst ) = neutral else if inst in at least one positive and one negative subjective expression: goldclass( inst ) = both else if inst in a mixture of negative and neutral: goldclass( inst ) = negative else if inst in a mixture of positive and neutral: goldclass( inst ) = positive else: goldclass( inst ) = contextual polarity of subjective expression

22. HLT-EMNLP 200522 Features Many inspired by Polanya & Zaenen (2004): Contextual Valence Shifters Example:little threat little truth Others capture dependency relationships between words Example: wonderfully horrid pos mod

23. HLT-EMNLP 200523 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

24. HLT-EMNLP 200524 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Word token terrifies Word part-of-speech VB Context that terrifies me Prior Polarity negative Reliability strongsubj Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

25. HLT-EMNLP 200525 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Binary features: Preceded by adjective adverb (other than not) intensifier Self intensifier Modifies strongsubj clue weaksubj clue Modified by strongsubj clue weaksubj clue Dependency Parse Tree Thehumanrights report poses asubstantial challenge … det adj mod adj det subjobj p Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

26. HLT-EMNLP 200526 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Binary features: In subject The human rights report poses In copular I am confident In passive voice must be regarded Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Thehumanrights report poses asubstantial challenge … det adj mod adj det subjobj p

27. HLT-EMNLP 200527 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Count of strongsubj clues in previous, current, next sentence Count of weaksubj clues in previous, current, next sentence Counts of various parts of speech Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

28. HLT-EMNLP 200528 1. Word features 2. Modification features 3. Structure features 4. Sentence features 5. Document feature Document topic (15) economics health Kyoto protocol presidential election in Zimbabwe … Example: The disease can be contracted if a person is bitten by a certain tick or if a person comes into contact with the blood of a congo fever sufferer. Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

29. HLT-EMNLP 200529 Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Results 1a

30. HLT-EMNLP 200530 Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Results 1b

31. HLT-EMNLP 200531 Step 2: Polarity Classification Classes positive, negative, both, neutral Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances 19,5065,671

32. HLT-EMNLP 200532 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

33. HLT-EMNLP 200533 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter Word token terrifies Word prior polarity negative Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

34. HLT-EMNLP 200534 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter Binary features: Negated For example: not good does not look very good  not only good but amazing Negated subject No politically prudent Israeli could support either of them. Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

35. HLT-EMNLP 200535 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter Modifies polarity 5 values: positive, negative, neutral, both, not mod substantial: negative Modified by polarity 5 values: positive, negative, neutral, both, not mod challenge: positive substantial (pos) challenge (neg) Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

36. HLT-EMNLP 200536 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter Conjunction polarity 5 values: positive, negative, neutral, both, not mod good: negative good (pos) and evil (neg) Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

37. HLT-EMNLP 200537 Word token Word prior polarity Negated Negated subject Modifies polarity Modified by polarity Conjunction polarity General polarity shifter Negative polarity shifter Positive polarity shifter General polarity shifter pose little threat contains little truth Negative polarity shifter lack of understanding Positive polarity shifter abate the damage Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

38. HLT-EMNLP 200538 Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Results 2a

39. HLT-EMNLP 200539 Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances Results 2b

40. HLT-EMNLP 200540 Ablation experiments removing features: 1. Negated, negated subject 2. Modifies polarity, modified by polarity 3. Conjunction polarity 4. General, negative, positive polarity shifters Corpus Lexicon Neutral or Polar? Step 1 Contextual Polarity? Step 2 All Instances Polar Instances

41. HLT-EMNLP 200541 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions

42. HLT-EMNLP 200542 Previous Work Learn prior polarity of words and phrases e.g., Hatzivassiloglou & McKeown (1997), Turney (2002) Sentence-level sentiment analysis e.g., Yu & Hatzivassiloglou (2003), Kim & Hovy (2004) Phrase-level contextual polarity classification e.g., Yi et al. (2003)

43. HLT-EMNLP 200543 At HLT/EMNLP 2005 Popescu & Etizioni: Extracting Product Features and Opinions from Reviews Choi, Cardie, Riloff & Patwardhan: Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns Alm, Roth & Sproat: Emotions from Text: Machine Learning for Text-based Emotion Prediction

44. HLT-EMNLP 200544 Outline Introduction Manual Annotations Corpus Prior-Polarity Subjectivity Lexicon Experiments Previous Work Conclusions

45. HLT-EMNLP 200545 Conclusions Presented a two-step approach to phrase- level sentiment analysis 1. Determine if an expression is neutral or polar 2. Determines contextual polarity of the ones that are polar Automatically identify the contextual polarity of a large subset of sentiment expression

46. HLT-EMNLP 200546 Thank you

47. HLT-EMNLP 200547 Acknowledgments This work was supported by Advanced Research and Development Activity (ARDA) National Science Foundation