Presentation is loading. Please wait.

Presentation is loading. Please wait.

6 th June, Defence CBRN Winterbourne Gunner, UK Prof. Tom Jackson, Dr. Ann O’Brien, Dr. Martin Sykora and Dr. Suzanne Elayan Extracting the Meaning Of.

Similar presentations


Presentation on theme: "6 th June, Defence CBRN Winterbourne Gunner, UK Prof. Tom Jackson, Dr. Ann O’Brien, Dr. Martin Sykora and Dr. Suzanne Elayan Extracting the Meaning Of."— Presentation transcript:

1 6 th June, Defence CBRN Winterbourne Gunner, UK Prof. Tom Jackson, Dr. Ann O’Brien, Dr. Martin Sykora and Dr. Suzanne Elayan Extracting the Meaning Of Terse Information in a Visualisation of Emotion

2 Contents Introduction to EMOTIVE System Overview and Evaluation Visual Interpretation Conclusions

3 Egyptian activist; “We use Facebook to schedule our protests, Twitter to coordinate and YouTube to tell the world.” (Meier 2011) Social Media – Polling public opinion: O’Connor et al. 2010, Tumasjan et al. 2010, Cheong et al. 2011, Lansdall-Welfare et al Social Media is first to break-the-news! Mumbai attacks, where individuals on location broke the news via Twitter. - July 2009 Jakarta bombings, where Twitter broke the news. - Even earthquakes, ranging from seismic intensity scale 3 or more, were reported quicker by Twitter users as opposed to the relevant Japanese agency. Commercial Interest: Attensity, Crimson Hexagon, Sysomos, Brandwatch, Vocus, Socialradar, Radian 6,... Crisis mapping community: National security social media monitoring in the literature – reviewed in Sykora et al. (2013) IntroductionSystem Overview Interpretation Conclusions

4 EMOTIVE (Extracting the Meaning Of Terse Information in a Visualisation of Emotion) The system’s main aims, are Emotion, Geo-location and Filtering / Keyphrase extraction from sparse messages (i.e. Tweets). Finally displaying a spectrum of emotions relating to current events on a dynamic geo-map interface for interpretation and interactive exploration by an analyst. IntroductionSystem Overview Interpretation Conclusions

5 1. M6 Megabus Bomb-Alarm incident 2.Serious Kings Cross Tube Station Overcrowding (just ahead of the summer Olympics) Belfast Riots 4.Woolwich Soldier Murder 4.September 2012 English Floods (River Ouse, River Weaver, River Severn, etc.) 5.UK Snow Disruption Al-Hilli Bomb (Claygate) Scare – Alps Shooting and Al-Hilli spy accusations 7.PC Manchester Shootings Aftermath and Guns for Police debate (Funeral of the Police Officers – and reactions towards Dale Cregan) 8.Olympic Parade, Closing / Opening Ceremony – Olympics / Paralympics coverage 9.Horsemeat scandal 10.Government Reshuffle 11.Nick Clegg – apology incident 13.Job Losses / Unemployment (reactions to JJB Sports bankruptcy) 14.DNC (USA Democratic National Convention 2012) Watching the news in Tescos, bit confused about what's happened on the M6 toll road Splashy♥ ❤ 2001, terrorists hijack planes. 2012, terrorism alert on a Megabus on the M6 toll. This recession has hit hard. wow it is quite emotional, starting to well up here #olympicparade so proud of our athletes and our nation IntroductionSystem Overview Interpretation Conclusions

6 Emotion extraction, prior work: Notions of affect and sentiment have been rather simplified in current state-of-the-art, often confined to their assumed overall polarity (i.e. positive / negative), Thelwall (2012) Another problem with polarity-centric sentiment classifiers is that they generally encompass a vague notion of polarity that bundles together emotion, states and opinion There is no common agreement about which features are the most relevant in the definition of an emotion and which are the relevant emotions and their names, Grassi (2009) Comparison: de Choudhury and Counts (2012) & Thelwall et al. (2012) IntroductionSystem Overview Interpretation Conclusions

7 EMOTIVE emotion detection is completely finalised & following fine-grained explicit emotions are extracted from sparse text messages:  Anger, Disgust, Fear, Happiness, Sadness, Surprise (Ekman’s 6 basic emotions) + Shame, and Confusion.  Shame – common on Twitter  Confusion – useful for situational awareness, Oh et al. (2011) IntroductionSystem Overview Interpretation Conclusions

8  The ontology contains over 300 emotional terms, with many intensifier, conjunction, negation and interjection words and phrases. It also contains information on the perceived strength of emotions, and some linguistic analysis related information.  Example Emotion Terms from the Ontology: Anger (e.g. enraged, infuriated, peeved, in a tizzy…) Confusion(e.g. chaotic, distracted, perplexed, confuzzled…) Disgust(e.g. appalling, beastly, bullshit, scuzzy…) Fear(e.g. cold feet, goose bumpy, petrified, scary…) Happiness(e.g. blissful, chuffed, delighted, in high spirits…) Sadness(e.g. depressed, devastating, duff, grief stricken…) Shame(e.g. abashment, degrading, hang head in shame, scandalous…) Surprise (e.g. astonished, disbelief, gobsmacked, off guard…) IntroductionSystem Overview Interpretation Conclusions

9  Study of language performed by an English language and literature PhD level research associate, with training in linguistics and discourse analysis, during a three month time-period. 600MB of cleaned Tweets on 63 different UK-specific topics / search-terms datasets Focused on identifying commonly used explicit expressions of emotion OOV (Out of Vocabulary) terms, Wordnet synset synonym lists of emotional expressions, Dictionary.com, Thesaurus.com, the Oxford English online dictionary, Urbandictionary.com, Internetslang.com… Emotional terms and activation levels identified and used in work by Choudhury et al. (2012) and lexicon lists of intensifiers, negators & words of basic sentiment used in SentiStrength-2, Thelwall et al. (2012) were also reviewed. IntroductionSystem Overview Interpretation Conclusions

10

11 A custom NLP-pipeline was developed to facilitate the ontology driven linguistic analysis and emotion extraction.  We developed a Brill-tagger, with pre-tagged tokens from a tri-gram tagger, with backoff to bi-gram, uni-gram and a simple regex pattern tagger.  This custom POS tagger is faster in the multiples and achieved a successful 0.88 accuracy in respect to Gimpel et al. (2010) – keeping in mind, the baseline on tweet style messages is around 0.7, Ritter et al. (2011) – as our POS tagger was evaluated on unseen Twitter datasets used in Gimpel et al. (2010). Our POS-tagger is faster in the multiples to Gimpel et al. (2010) Accuracy is equivalent to Gimpel et al. (2010)  The ontology is loaded into efficient data-structures (Hashtables and Tries), the custom NLP-pipeline is speed optimised and processes around 1500 tweets / sec, on a single avg. machine. IntroductionSystem Overview Interpretation Conclusions

12  On an initial golden-dataset (annotated by 2 human annotators) of emotive tweets the technique achieved excellent results, F-measure =.962:  This is an extremely high f-measure illustrating the successful nature of the ontology.  To compare our high f-measure to another approach, fine-grained emotion detection from Choudhury et al. (2012) achieved;.744 /.668 (f-measure),.830 /.658 (precision) and.674 /.680 (recall); direct matching / stemmed matching, respectively.  EMOTIVE’s emotion strength scoring approach was evaluated against SentiStrength-2 (Thelwall et al. 2012): a consistent and statistically significant correlation was found; which indicates that we are measuring in line with a sentiment scoring state-of-the-art system. Recall, precision and f-measure, were computed using an equivalent approach as used in CoNNL-2003 shared task on NER (Tjong et al. 2003). IntroductionSystem Evaluation Interpretation Conclusions

13 Woolwich Soldier Killing (Lee Rigby) – #woolwich, Anjem Choudary The brutal murder sparked a storm of emotional reactions of Sadness, Disgust and Surprise. At the same time the controversial cleric Anjem Choudary was most often mentioned with extreme emotions of Anger and Disgust. Example reactions to Anjem Choudary. I'm quite angry that Anjem Choudary is on Newsnight tonight - I can only imagine how furious Muslims he falsely claims to speak for must be [anger].And I'm angry that Anjem Choudary is aloud to preach hate in our towns and city's It's the government we should be angry with not a religion [anger].Anjem Choudary, gfy. Ruining the 'Choudhary' name for all of us, you complete bastard, it's sickening #woolwich so sad, and so wrong that ANJEM CHOUDARY can get air time saying muslims around the world will call them heroes what a twat. [sadness]

14 Woolwich Soldier Killing (Lee Rigby) – #woolwich.My heart goes to the soldiers family, friends, the people of #woolwich & all those effected, so pretty much everyone. Terribly sad news. Military commanders tell soldiers told not to wear their uniforms in public until further notice #Woolwich" - Sad times :( [sadness].The fact the soldier was a father upsets me further. Maybe it shouldn't but it does #woolwich [sadness].My heart goes to the soldiers family, friends, the people of #woolwich & all those effected, so pretty much everyone. Terribly sad news. Military commanders tell soldiers told not to wear their uniforms in public until further notice #Woolwich" - Sad times :( [sadness].The fact the soldier was a father upsets me further. Maybe it shouldn't but it does #woolwich [sadness].#woolwich absolutely disgusting scene yesterday. Jst so annoying [disgust].#Woolwich - so awful. Strength to the victim's families. [disgust].Can't believe the news about yesterday's #woolwich attack. Disgusting. Some people are so sick!! [disgust].#woolwich absolutely disgusting scene yesterday. Jst so annoying [disgust].#Woolwich - so awful. Strength to the victim's families. [disgust].Can't believe the news about yesterday's #woolwich attack. Disgusting. Some people are so sick!! [disgust].Enjoyed last event in response to @ihrc, Syed Ammar & Sheikh Panju.. article on the town I'm proud to live in. We love #Woolwich [happiness]http://t.co/W0MXkxIqvm.The family of murdered soldier pay tribute. Rebecca Rigby: "I love Lee, I always will and I'm proud to be his wife." #woolwich [happiness].Enjoyed last event in response to @ihrc, Syed Ammar & Sheikh Panju.. article on the town I'm proud to live in. We love #Woolwich [happiness]http://t.co/W0MXkxIqvm.The family of murdered soldier pay tribute. Rebecca Rigby: "I love Lee, I always will and I'm proud to be his wife." #woolwich [happiness].#Woolwich Attack: New Shocking Video of Terrorists Charging at Police Car, Getting Shot: #EDL [surprise].Utterly astonished to see some videos popping up claiming the #Woolwich attack was a hoax with the media and government colluding! [surprise].#Woolwich Attack: New Shocking Video of Terrorists Charging at Police Car, Getting Shot: #EDL [surprise].Utterly astonished to see some videos popping up claiming the #Woolwich attack was a hoax with the media and government colluding! #woolwich She was arrested on Wednesday after apparently asking police for protection when her malicious tweet prompted angry backlash. [anger] Dyka Ayan Hassan from Harrow, 21, arrested for a malicious #woolwich She was arrested on Wednesday after apparently asking police for protection when her malicious tweet prompted angry backlash. [anger] Dyka Ayan Hassan from Harrow, 21, arrested for a malicious "Free speech in Britain is threatened by the influence of Muslims in the media" YouGov question. Wow. #woolwich [fear].Following the #Woolwich incident, people in #Britain are anxious. Reports of a man with an axe at #LondonBridge is making people nervous. "Free speech in Britain is threatened by the influence of Muslims in the media" YouGov question. Wow. #woolwich [fear].Following the #Woolwich incident, people in #Britain are anxious. Reports of a man with an axe at #LondonBridge is making people nervous. [fear]

15 Achieved excellent F-measures on Tweets Evaluated against two other systems from Literature, with good results: emotion & strength scores detection A Golden dataset (620 tweets) of explicit emotions, intensifiers and related entities, annotated by two human-expert annotators with over 98% agreement. Social media monitoring system – EMOTIVE  Aiding analysts to interpret live events  Learning and predicting from previous datasets IntroductionSystem Overview Interpretation Conclusions

16 Thanks

17 References Choudhury M. and Counts S., The Nature of Emotional Expression in Social Media: Measurement, Inference and Utility, Technical Report: Microsoft. Drummond T., Vocabulary of Emotions [Online], North Seattle Community College, [last viewed ]. Available from Ekman P., All emotions are basic. The nature of emotion: Fundamental questions Gimpel K., Schneider N., O'Connor B., Das D., Mills D., Eisenstein J., Heilman M., Yogatama D., Flanigan J. and Smith N., Part-of-speech tagging for twitter: Annotation, features, and experiments, Technical Report. Grassi M., Developing HEO human emotions ontology, Biometric ID Management and Multimodal Communication, Springer Berlin Heidelberg, pp Izard C. E., Emotion theory and research: Highlights, unanswered questions, and emerging issues. Annual Review of Psychology 60, O'Connor B., Krieger M. and Ahn D., TweetMotif: Exploratory Search and Topic Summarization for Twitter, Proceedings of the International AAAI Conference on Weblogs and Social Media, Washington DC (USA) Oh O., Agrawal M. and Rao H., Information control and terrorism: Tracking the Mumbai terrorist attack through twitter, Information Systems Frontiers 13, pp Potts C., Potts Twitter-aware Tokeniser - [last viewed ] Plutchik R., Emotion: A Psychoevolutionary Synthesis. Longman Higher Education. Ritter A., Clark S., Mausam and Etzioni O., Named Entity Recognition in Tweets: An Experimental Study, Proceedings of Conference on Empirical Methods in Natural Language Processing, Edinburgh (UK) Sykora M., Jackson T. W., O’Brien A. and Elayan S., National Security and Social Media Monitoring: A Presentation of the EMOTIVE and Related Systems, submitted to EISIC 2013, Uppsala (Sweden). Thelwall M., Buckley K. and Paltoglou G., Sentiment Strength Detection for the Social Web, Journal of the American Society for Information Science and Technology 63, pp Tjong E. F., Sang K., Meulder F. D., Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition, Proceedings of the ACL seventh conference on Natural Language Learning.


Download ppt "6 th June, Defence CBRN Winterbourne Gunner, UK Prof. Tom Jackson, Dr. Ann O’Brien, Dr. Martin Sykora and Dr. Suzanne Elayan Extracting the Meaning Of."

Similar presentations


Ads by Google