Joint Models of Disagreement and Stance in Online Debate Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, Marilyn Walker University of California,

Slides:

Advertisements

Similar presentations

Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California

Advertisements

The Software Infrastructure for Electronic Commerce Databases and Data Mining Lecture 4: An Introduction To Data Mining (II) Johannes Gehrke

Document Summarization using Conditional Random Fields Dou Shen, Jian-Tao Sun, Hua Li, Qiang Yang, Zheng Chen IJCAI 2007 Hao-Chin Chang Department of Computer.

On-line learning and Boosting

Jointly Identifying Temporal Relations with Markov Logic Katsumasa Yoshikawa †, Sebastian Riedel ‡, Masayuki Asahara †, Yuji Matsumoto † † Nara Institute.

1.Accuracy of Agree/Disagree relation classification. 2.Accuracy of user opinion prediction. 1.Task extraction performance on Bing web search log with.

Unifying MAX SAT, Local Consistency Relaxations, and Soft Logic with Hinge-Loss MRFs Stephen H. Bach Bert Huang Lise Getoor Maryland Virginia Tech UC Santa.

Indian Statistical Institute Kolkata

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

Confidence-Weighted Linear Classification Mark Dredze, Koby Crammer University of Pennsylvania Fernando Pereira Penn  Google.

Foundations of Comparative Analytics for Uncertainty in Graphs Lise Getoor, University of Maryland Alex Pang, UC Santa Cruz Lisa Singh, Georgetown University.

Markov Logic: A Unifying Framework for Statistical Relational Learning Pedro Domingos Matthew Richardson

Logistic Regression Principal Component Analysis Sampling TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA A A A.

Logistics Course reviews Project report deadline: March 16 Poster session guidelines: – 2.5 minutes per poster (3 hrs / 55 minus overhead) – presentations.

Scalable Text Mining with Sparse Generative Models

Seven Lectures on Statistical Parsing Christopher Manning LSA Linguistic Institute 2007 LSA 354 Lecture 7.

Online Stacked Graphical Learning Zhenzhen Kou +, Vitor R. Carvalho *, and William W. Cohen + Machine Learning Department + / Language Technologies Institute.

SI485i : NLP Set 12 Features and Prediction. What is NLP, really? Many of our tasks boil down to finding intelligent features of language. We do lots.

STRUCTURED PERCEPTRON Alice Lai and Shi Zhi. Presentation Outline Introduction to Structured Perceptron ILP-CRF Model Averaged Perceptron Latent Variable.

Opinion mining in social networks Student: Aleksandar Ponjavić 3244/2014 Mentor: Profesor dr Veljko Milutinović.

Classifiers, Part 1 Week 1, video 3:. Prediction  Develop a model which can infer a single aspect of the data (predicted variable) from some combination.

Overview: Humans are unique creatures. Everything we do is slightly different from everyone else. Even though many times these differences are so minute.

Graphical models for part of speech tagging

Margin Learning, Online Learning, and The Voted Perceptron SPLODD ~= AE* – 3, 2011 * Autumnal Equinox.

HyPER: A Flexible and Extensible Probabilistic Framework for Hybrid Recommender Systems Pigi Kouki, Shobeir Fakhraei, James Foulds, Magdalini Eirinaki,

Bayesian networks Classification, segmentation, time series prediction and more. Website: Twitter:

 Text Representation & Text Classification for Intelligent Information Retrieval Ning Yu School of Library and Information Science Indiana University.

Recognizing Activities of Daily Living from Sensor Data Henry Kautz Department of Computer Science University of Rochester.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.6: Linear Models Rodney Nielsen Many of.

Predictive Design Space Exploration Using Genetically Programmed Response Surfaces Henry Cook Department of Electrical Engineering and Computer Science.

Chapter6. Statistical Inference : n-gram Model over Sparse Data 이 동 훈 Foundations of Statistic Natural Language Processing.

Empirical Research Methods in Computer Science Lecture 7 November 30, 2005 Noah Smith.

1 Generative and Discriminative Models Jie Tang Department of Computer Science & Technology Tsinghua University 2012.

Today Ensemble Methods. Recap of the course. Classifier Fusion

Exploiting Context Analysis for Combining Multiple Entity Resolution Systems -Ramu Bandaru Zhaoqi Chen Dmitri V.kalashnikov Sharad Mehrotra.

+ Collective Spammer Detection in Evolving Multi-Relational Social Networks Shobeir Fakhraei (University of Maryland) James Foulds (University of California,

Graphical Models over Multiple Strings Markus Dreyer and Jason Eisner Dept. of Computer Science, Johns Hopkins University EMNLP 2009 Presented by Ji Zongcheng.

Department of Electrical Engineering and Computer Science Kunpeng Zhang, Yu Cheng, Yusheng Xie, Doug Downey, Ankit Agrawal, Alok Choudhary {kzh980,ych133,

Learning TFC Meeting, SRI March 2005 On the Collective Classification of “Speech Acts” Vitor R. Carvalho & William W. Cohen Carnegie Mellon University.

CHAPTER 6 Naive Bayes Models for Classification. QUESTION????

Template-Based Event Extraction Kevin Reschke – Aug 15 th 2013 Martin Jankowiak, Mihai Surdeanu, Dan Jurafsky, Christopher Manning.

Online Learning Rong Jin. Batch Learning Given a collection of training examples D Learning a classification model from D What if training examples are.

Carolyn Penstein Rosé Language Technologies Institute Human-Computer Interaction Institute School of Computer Science With funding from the National Science.

Recognizing Stances in Online Debates Unsupervised opinion analysis method for debate-side classification. Mine the web to learn associations that are.

Click to Add Title A Systematic Framework for Sentiment Identification by Modeling User Social Effects Kunpeng Zhang Assistant Professor Department of.

Weakly Supervised Models of Aspect-Sentiment for Online Course Discussion Forums ARTI RAMESH SHACHI H. KUMAR JAMES FOULDS LISE GETOOR.

DeepDive Model Dongfang Xu Ph.D student, School of Information, University of Arizona Dec 13, 2015.

Human and Optimal Exploration and Exploitation in Bandit Problems Department of Cognitive Sciences, University of California. A Bayesian analysis of human.

Contribution and Proposed Solution Sequence-Based Features Collective Classification with Reports Results of Classification Using Reports Collective Spammer.

Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Mining Advisor-Advisee Relationships from Research Publication.

Department of Computer Science The University of Texas at Austin USA Joint Entity and Relation Extraction using Card-Pyramid Parsing Rohit J. Kate Raymond.

1 Relational Factor Graphs Lin Liao Joint work with Dieter Fox.

Brief Intro to Machine Learning CS539

Lecture 7: Constrained Conditional Models

Sentiment analysis algorithms and applications: A survey

School of Computer Science & Engineering

Learning Coordination Classifiers

Data Mining Jim King.

Adversarial Learning for Neural Dialogue Generation

CSC 594 Topics in AI – Natural Language Processing

CSCI 5822 Probabilistic Models of Human and Machine Learning

Conditional Random Fields

Learning Emoji Embeddings Using Emoji Co-Occurrence Network Graph

Speaker: Jim-An Tsai Advisor: Professor Jia-ling Koh

Biased Random Walk based Social Regularization for Word Embeddings

Conditional Random Fields

Discriminative Probabilistic Models for Relational Data

Introduction to Sentiment Analysis

Dan Roth Department of Computer Science

Stance Classification of Ideological Debates

Presentation transcript:

Joint Models of Disagreement and Stance in Online Debate Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, Marilyn Walker University of California, Santa Cruz Virginia Institute of Technology

Social media sites for debating issues Valuable resources for: –Argumentation –Dialogue –Sentiment –Opinion mining 2 Online Debate Forums

CreateDebate.org 3

4 Debate topic

CreateDebate.org 5 Debate topic Posts

CreateDebate.org 6 Debate topic Posts Replies

CreateDebate.org 7 Debate topic Posts Replies Reply polarity

4Forums.com 8

9 Quotation

Graph of posts: tree structure Online Debate Forums 10 Graph of users: loopy structure = reply link

Graph of posts: tree structure Online Debate Forums 11 Graph of users: loopy structure

Zoom in on an Example 12

Zoom in on an Example 13

I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Zoom in on an Example 14

I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Stance Classification 15 Useful for advocacy and get-out-the-vote campaigns

I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Stance Classification 16 Study argumentation and dialogue Posts express stance

I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Disagreement Classification 17 Study argumentation and dialogue Disagree on stance

I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Disagreement Classification 18 Study argumentation and dialogue Disagree on stance Posts express disagreement

Stance Classification Targets 19 Stance Author-level Post-level Disagreement Author-level Post-level Textual

Stance Disagrees Classification Targets 20 Stance Author-level Post-level Disagreement Author-level Post-level Textual

Stance Disagrees Classification Targets 21 Stance Author-level Post-level Disagreement Author-level Post-level Textual

Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 22

Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 23

Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 24

Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 25

Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 26

Stance Classification: “Teach the Controversy” Previous work employs many modeling strategies How best to model stance in online debate? Answers may be different to Congressional debates –Links have different semantics –Posts much shorter than speeches. Many posts per author –Dialogue is informal 27

Stance Classification: “Teach the Controversy” Previous work employs many modeling strategies How best to model stance in online debate? Answers may be different to Congressional debates –Links have different semantics –Posts much shorter than speeches. Many posts per author –Dialogue is informal 28

Stance Modeling at author-level or post-level? 29 [Hasan and Ng 2013] [Other Related Work] Modeling Question 1)

Stance Modeling at author-level or post-level? 30 [Hasan and Ng 2013] [Other Related Work] Modeling Question 1)

Stance Modeling Question 2) 31 [Walker et al. 2012, Hasan and Ng 2013] Collective classification vs. local classification?

Stance Modeling Question 2) 32 [Walker et al. 2012, Hasan and Ng 2013 ] [Walker et al. 2012] Collective classification vs. local classification?

Stance Disagrees 33 Jointly model disagreement together with stance? [Abbott et. al Linguistic Features], [Burfoot et. al 2011 for Congressional Debates] Modeling Question 3)

Stance Disagrees 34 Jointly model disagreement together with stance? [Abbott et. al Linguistic Features], [Burfoot et. al 2011 for Congressional Debates] Modeling Question 3) Stance Disagrees Stance

Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options 35

Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options 36

Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options –Modeling recommendations 37

Author Post Local Collective Joint Author Local Author Coll. Author Joint Post Local Post Joint Post Coll. Modeling Granularity Statistical Models All Combinations of Models 38

Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields 39 Bach et. al (2015), ArXiV Open source software: 5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight

Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields : Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight Predicates are continuous Random Variables!

Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields : Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight Predicates are continuous Random Variables! Relaxations of Logical Operators

Probabilistic Soft Logic (PSL) Rules instantiated with variables from real network : Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Stance Disagrees

Probabilistic Soft Logic (PSL) Rules instantiated with variables from real network : Disagrees(, ) ^ Pro( )  ~Pro( ) … Continuous Random Variables!

Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI Bach et al. (2015), ArXiV Conditional random field over continuous RVs in [0,1]

Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI Conditional random field over continuous RVs in [0,1] Feature function for each instantiated rule 5.0: Disagrees(, ) ^ Pro( )  ~Pro( )

Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI Conditional random field over continuous RVs in [0,1] Feature functions are hinge-loss functions

Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI Encodes distance to satisfaction of each instantiated rule Linear function

Fast Inference in Hinge-loss MRFs Bach et al. NIPS 12, Bach et al. UAI Convex, continuous inference objective… Convex optimization! Solved using efficient, parallelizable algorithm: Alternating Direction Method of Multipliers (ADMM)

Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Obama Bush believe Constructing Local Predictors 49 Bag-of-words

Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Logistic Regression Obama Bush believe Constructing Local Predictors 50 Pro Not Pro Bag-of-words Training Labels

Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Logistic Regression Observed Prediction Probabilities Obama Bush believe Constructing Local Predictors 51 Pro Not Pro Bag-of-words Training Labels LocalPro: 0.8 LocalPro: 0.1

PSL Rules Shared by All Models 52 Stance LocalPro: 0.8 LocalPro: 0.1

PSL Rules for Simple Collective Models 53 Stance Disagrees: 1.0 LocalPro: 0.8 LocalPro: 0.1

Joint Disagreement Models 54 Stance Disagrees LocalPro: 0.8 LocalPro: 0.1 LocalDis: 0.5

Joint Disagreement Models 55 Stance Disagrees LocalPro: 0.8 LocalPro: 0.1 LocalDis: 0.5

Details4Forums.comCreateDebate.org TopicsAbortion, Gay Marriage, Evolution, Gun Control Abortion, Gay Rights, Obama, Marijuana Avg. Users/Topic Avg. Posts/User 194 Evaluation - Datasets 56

Evaluation Settings Prediction Tasks: Author Stance, Post Stance 57

Evaluation Settings Prediction Tasks: Author Stance, Post Stance 58 Ground Truth for CreateDebate.org: Stance Majority

Evaluation Settings Prediction Tasks: Author Stance, Post Stance 59 Ground Truth for CreateDebate.org: Stance Majority Ground Truth for 4Forums: Stance

60 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org

61 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Post < Author

62 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Post < Author Author-Joint Model is best

63 Accuracy Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Post Stance Prediction – CreateDebate.org Post < Author (still!) Author-Joint Model still best!

64 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Local < Collective < Joint

65 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Naïve collective harmful at author level! Author Stance Prediction – 4Forums.com

66 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – 4Forums.com Naïve collective harmful at author level!

Details4Forums.comCreateDebate.org % Opposite Stance Posts % Opposite Stance Authors Explanation for Naïve Collective’s Performance 67

Details4Forums.comCreateDebate.org % Opposite Stance Posts % Opposite Stance Authors Explanation for Naïve Collective’s Performance 68 Naïve collective assumption mostly true for posts

Details4Forums.comCreateDebate.org % Opposite Stance Posts % Opposite Stance Authors Explanation for Naïve Collective’s Performance 69 Naïve collective assumption mostly true for posts Assumption doesn’t hold at author level!

70 I agree with everything except the last part. Safe gun storage is very important… I agree with everything except the last part. Safe gun storage is very important… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… Benefit of Disagreement Prediction Agree Anti Gun Control

71 I agree with everything except the last part. Safe gun storage is very important… I agree with everything except the last part. Safe gun storage is very important… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… Benefit of Disagreement Prediction Agree Anti Gun Control

Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 72

Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 73

Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 74

Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 75 Thank you for your attention!