Presentation is loading. Please wait.

Presentation is loading. Please wait.

A cognitive study of subjectivity extraction in sentiment annotation Abhijit Mishra 1, Aditya Joshi 1,2,3, Pushpak Bhattacharyya 1 1 IIT Bombay, India.

Similar presentations


Presentation on theme: "A cognitive study of subjectivity extraction in sentiment annotation Abhijit Mishra 1, Aditya Joshi 1,2,3, Pushpak Bhattacharyya 1 1 IIT Bombay, India."— Presentation transcript:

1 A cognitive study of subjectivity extraction in sentiment annotation Abhijit Mishra 1, Aditya Joshi 1,2,3, Pushpak Bhattacharyya 1 1 IIT Bombay, India 2 Monash University, Australia 3 IITB-Monash Research Academy At 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, ACL 2014, Baltimore

2 Subjectivity Extraction Goal: To identify subjective portions of text

3 Motivation Strong AI suggests that a machine must be perform sentiment analysis in a manner and accuracy similar to human beings Do humans perform subjective extraction as well? A “cognitive study” of subjectivity extraction in sentiment annotation

4 Outline Sentiment Oscillations & Subjectivity Extraction Experiment Setup Anticipation & Homing Conclusion & Future Work

5 Sentiment Oscillations & subjectivity extraction Subjective documents may be: Humans perform subjectivity extraction either as a result of “anticipation” or as “homing”. Which of the two methods are adopted depends on the linear/oscillating nature of the subjective document. Linear: The story was captivating. The actors did a great job. I absolutely loved the movie! Oscillating: The story was captivating. Only if they had better actors. But then I enjoyed the movie, on the whole.

6 Experiment Setup (1/2) A human annotator reads a document and predicts its sentiment A Tobii T120 eye-tracker records eye movements while he/she reads the document * No time restriction, no user input required: to minimize errors.

7 Experiment Setup (2/2) Dataset – 3 Movie reviews in English from imdb – One linear, one oscillating, one between the two extremes (D0, D1, D2 respectively) Three documents? Really?! – To eliminate predictability – To reduce errors due to fatigue 12 human annotators (P0,.. P11 respectively)

8 Observations: Anticipation (1/2) In case of linear subjective documents, an annotator reads some sentences and begins to skip sentences.

9 Observations: Anticipation (2/2) DocumentLengthAverage number of non-unique sentences read by participants D01021 D D

10 Observations: Homing (1/3) In case of oscillating subjective documents, an annotator (a) first reads all sentences, (b) revisits some sentences again

11 Observations: Homing (2/3) Considerable overlap between sentences that are read in the second pass All of them are subjective. ParticipantTFD-SEPTFDTFC-SE P P P P Reading statistics for D1 TFD: Total fixation duration for subjective extract; PTFD: Proportion of total fixation duration = (TFD)/(Total duration); TFC-SE: Total fixation count for subjective extract

12 Observations: Homing (3/3) Homing at a sub-sentence level – Sarcasm Multiple regressions around the sarcasm portion for participant P1, document D1 Participant P1 does not correctly detect the sentiment of the document – Thwarting

13 Conclusion & Future Work Based on how sentiment changes through a document, humans may perform subjectivity extraction as a result of anticipation or homing Applications: – Pricing models for crowd-sourced annotation – Sentiment classifiers that incorporate “sentiment runlengths”

14 References WikiSent : Weakly Supervised Sentiment Analysis Through Extractive Summarization With Wikipedia, Subhabrata Mukherjee and Pushpak Bhattacharyya, ECML PKDD 2012 A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts, Bo Pang, Lillian Lee, ACL 2004


Download ppt "A cognitive study of subjectivity extraction in sentiment annotation Abhijit Mishra 1, Aditya Joshi 1,2,3, Pushpak Bhattacharyya 1 1 IIT Bombay, India."

Similar presentations


Ads by Google