Title: The Author-Topic Model for Authors and Documents Authors: Rosen-Zvi, Griffiths, Steyvers, Smyth Venue: the 20th Conference on Uncertainty in Artificial Intelligence Year: 2004 To give you guys a little personal connection… Presenter: Peter Wu Date: Apr 7, 2015

Title: The Author-Topic Model for Authors and Documents Authors: Rosen-Zvi, Griffiths, Steyvers, Smyth Venue: the 20th Conference on Uncertainty in Artificial Intelligence Year: 2004 This paper introduced a new version of generative model for topic modeling Presenter: Peter Wu Date: Apr 7, 2015

Title: The Author-Topic Model for Authors and Documents Authors: Rosen-Zvi, Griffiths, Steyvers, Smyth, Venue: the 20th Conference on Uncertainty in Artificial Intelligence Year: 2004 Extension: added the modeling of authors interest Built upon the original topic model… The extension part is that they added… If you are familiar with the original LDA… Presenter: Peter Wu Date: Apr 7, 2015 Blei, Ng, & Jordan. "Latent dirichlet allocation." the Journal of machine Learning research 3 (2003): 993-1022.

Outline Motivation Model formulation Parameter estimation Evaluation Application

Outline Motivation Model formulation Parameter estimation Evaluation Generative process; plate notation; a comparison with LDA Parameter estimation Gibbs sampling Evaluation Application Describe how the generative process works; and a convenient visualization of such process; then comparison to show the difference and innovation of this paper

Motivation Learning the interests of authors is a fundamental problem raised by large collection of documents. Previous works usually adopt a discriminative approach and features chosen are usually superficial. The authors introduced a generative model that represents each author with a distribution of weights over latent topics. First, it’s an important task because we have large collections; we want to learn the interests but not obvious Of course there are people ; The notion of authors interests is captured. Note that the topics are latent clusters

Motivation Learning the interests of authors is a fundamental problem raised by large collection of documents. Previous works usually adopt a discriminative approach and features chosen are usually superficial. The authors introduced a generative model that represents each author with a distribution of weights over latent topics. Unsupervised clustering algorithm: (only) the number of topics T needs to be specified.

Model Formulation It’s storytelling time for a generative model! Suppose we have a corpus of D documents that: spans a vocabulary of V words is collectively composed by A authors In this corpus, each document d: contains words wd (a subset of the V words with cardinality Nd; order doesn’t matter) is composed by authors ad (a subset of the A authors) We see what we observe and hypothesize a story of how what we see is generated; what we observe here is…

Model Formulation It’s storytelling time for a generative model! This is what we observe! It’s storytelling time for a generative model! Suppose we have a corpus of D documents that: spans a vocabulary of V words is collectively composed by A authors In this corpus, each document d: contains words wd (a subset of the V words with cardinality Nd; order doesn’t matter) is composed by authors ad (a subset of the A authors)

Model Formulation It’s storytelling time for a generative model! This is what we observe! It’s storytelling time for a generative model! Suppose we have a corpus of D documents that: spans a vocabulary of V words is collectively composed by A authors In this corpus, each document d: contains words wd (a subset of the V words with cardinality Nd; order doesn’t matter) is composed by authors ad (a subset of the A authors) How could such a corpus be created? Now we ask the question how could such a corpus be created

Model Formulation (Cont’d) How could what we observe be created? We introduce a latent layer of topic clusters, whose number T is specified by human, just like any unsupervised clustering algorithm (e.g., k-means). Suppose each of the A authors writes about the T topics with different probabilities: author k (𝑘∈ 1,…,𝐴 ) writes about topic j (𝑗∈ 1,…,𝑇 ) with probability 𝜃 𝑘𝑗 . Probabilities 𝜃 𝑘𝑗 form an A×T matrix, representing author-topic distributions. Suppose each of the T topics is represented by a distribution of different weights over the V words in the vocabulary: given a topic j (𝑗∈ 1,…,𝑇 ), word wm (𝑚∈ 1,…,𝑉 ) has a probability 𝜑 𝑗𝑚 to be used. Probabilities 𝜑 𝑗𝑚 form a T×V matrix, representing topic-word distributions. To generate each word in each document: Uniformly choose an author x among the document’s authors ad Sample a topic z from author x’s topic distribution 𝜽 𝑥∙ (author x’s row in the A×T matrix) Sample a word from topic z’s word distribution 𝝋 𝑧∙ (topic j’s row in the T×V matrix) Where does these latent topics come into play? It comes between authors and words and generate two distributions. Now we can start the generative process, or drawing; Now a quick quiz

Model Formulation (Cont’d) How could what we observe be created? We introduce a latent layer of topic clusters, whose number T is specified by human, just like any unsupervised clustering algorithm (e.g., k-means). Suppose each of the A authors writes about the T topics with different probabilities: author k (𝑘∈ 1,…,𝐴 ) writes about topic j (𝑗∈ 1,…,𝑇 ) with probability 𝜃 𝑘𝑗 . Probabilities 𝜃 𝑘𝑗 form an A×T matrix, representing author-topic distributions. Suppose each of the T topics is represented by a distribution of different weights over the V words in the vocabulary: given a topic j (𝑗∈ 1,…,𝑇 ), word wm (𝑚∈ 1,…,𝑉 ) has a probability 𝜑 𝑗𝑚 to be used. Probabilities 𝜑 𝑗𝑚 form a T×V matrix, representing topic-word distributions. To generate each word in each document: Uniformly choose an author x among the document’s authors ad Sample a topic z from author x’s topic distribution 𝜽 𝑥∙ (author x’s row in the A×T matrix) Sample a word from topic z’s word distribution 𝝋 𝑧∙ (topic j’s row in the T×V matrix) What distribution is this?

Model Formulation (Cont’d) How could what we observe be created? We introduce a latent layer of topic clusters, whose number T is specified by human, just like any unsupervised clustering algorithm (e.g., k-means). Suppose each of the A authors writes about the T topics with different probabilities: author k (𝑘∈ 1,…,𝐴 ) writes about topic j (𝑗∈ 1,…,𝑇 ) with probability 𝜃 𝑘𝑗 . Probabilities 𝜃 𝑘𝑗 form an A×T matrix, representing author-topic distributions. Suppose each of the T topics is represented by a distribution of different weights over the V words in the vocabulary: given a topic j (𝑗∈ 1,…,𝑇 ), word wm (𝑚∈ 1,…,𝑉 ) has a probability 𝜑 𝑗𝑚 to be used. Probabilities 𝜑 𝑗𝑚 form a T×V matrix, representing topic-word distributions. To generate each word in each document: Uniformly choose an author x among the document’s authors ad Sample a topic z from author x’s topic distribution 𝜽 𝑥∙ (author x’s row in the A×T matrix) Sample a word from topic z’s word distribution 𝝋 𝑧∙ (topic j’s row in the T×V matrix) Multinomial!

Plate Notation Generative process conveniently visualized Matrix 𝜃 and 𝜑 are the parameters we need to estimate in order to learn about the authors’ interests and topic patterns of the observed corpus Vector 𝛼 and 𝛽 are called called Dirichlet priors, which are hyper-parameters governing the multinomial distribution documented in each row of matrix 𝜃 and 𝜑. They are pre-specified to be symmetric parameters and we don’t need to estimate them. All what we discussed last slide can be visualized by this diagram called plate notation We can see that to generate the words matrix theta and fi are the parameters we need to estimate What are alpha and beta?

A Crash Course on Dirichlet Distribution Beta distribution Dirichlet distribution Parameters: 𝜶= 𝛼 1 ,…,𝛼 𝐾 >0,𝐾≥2, determining the shape of the PDF Support: 𝒙={𝑥 1 ,…, 𝑥 𝐾 }∈ 0,1 , 𝑖=1 𝐾 𝑥 𝑖 =1 PDF: Sampling result: a vector of values between 0 and 1 that sum up to 1 that can be used as the 𝒑={𝑝 1 ,…, 𝑝 𝐾 } parameters in multinomial distribution 𝑀𝑢𝑙𝑡𝑖(𝑛,𝒑) Parameters: 𝛼>0, 𝛽>0, determining the shape of the PDF Support: 𝑥∈[0,1] PDF: Sampling result: a value between 0 and 1 that can be used as the p parameter in binomial distribution 𝐵(𝑛,𝑝) For convenience we set

A Comparison between ATF and LDA Author-Topic Model (Rosen-Zvi et al, 2004) Latent Dirichlet Allocation (Blei et al, 2003) Now we finished formulating the model we’ll move onto parameter estimation

Outline Motivation Model formulation Parameter estimation Evaluation Generative process; plate notation; comparison with LDA Parameter estimation Gibbs sampling Evaluation Application

Parameter Estimation Parameters: Strategy: all elements 𝜃 𝑘𝑗 and 𝜑 𝑗𝑚 in the two matrices 𝜃 and 𝜑 Strategy: instead of estimating 𝜃 and 𝜑 directly, sample the authorship and topic assignment for each word in each document, and estimate 𝜃 and 𝜑 from the sample of assignments. The strategy is kind of complicated but look at it a little more closely

Parameter Estimation Parameters: Strategy: all elements 𝜃 𝑘𝑗 and 𝜑 𝑗𝑚 in the two matrices 𝜃 and 𝜑 Strategy: instead of estimating 𝜃 and 𝜑 directly, sample the authorship and topic assignment for each word in each document, and estimate 𝜃 and 𝜑 from the sample of assignments. “instead of estimating 𝜃 and 𝜑 directly” Why?  Intractable! “sample the authorship and topic assignment for each word in each document” How?  Gibbs Sampling! “estimate 𝜃 and 𝜑 from the sample of assignments” How?  We have a formula for this.

“Sample the authorship and topic assignment for each word in each document” How to do this? Gibbs Sampling! Gibbs sampling: When sampling from a joint distribution is impossible or hard, iteratively draw and update sample from conditional distributions and it will converge to a sample as if drawn from the joint distribution. Toy example: Sampling from a bivariate joint distribution 𝑝 𝜃 1 ,𝜃 2 Real application in Author-Topic Model: For each document d, we want to draw samples from 𝑝 𝑥 1 ,𝑧 1 ,… ,𝑥 𝑁 𝑑 , 𝑧 𝑁 𝑑 |𝒘 𝑑 , 𝒂 𝑑 , but it’s not feasible; Instead, we initialize with random assignments for all words, and; Update the authorship and topic assignment for each word in document d, ( 𝑥 𝑖 ,𝑧 𝑖 ), 𝑖=1,…, 𝑁 𝑑 , with a sample from a conditional distribution. So gibbs sampling is the practice of iteratively

“Sample the authorship and topic assignment for each word in each document” How to do this? Gibbs Sampling! Gibbs sampling: When sampling from a joint distribution is impossible or hard, iteratively draw and update sample from conditional distributions and it will converge to a sample as if drawn from the joint distribution. Toy example: Sampling from a bivariate joint distribution 𝑝 𝜃 1 ,𝜃 2 Real application in Author-Topic Model: For each document d, we want to draw samples from 𝑝 𝑥 1 ,𝑧 1 ,… ,𝑥 𝑁 𝑑 , 𝑧 𝑁 𝑑 |𝒘 𝑑 , 𝒂 𝑑 , but it’s not feasible; Instead, we initialize with random assignments for all words, and; Update the authorship and topic assignment for each word in document d, ( 𝑥 𝑖 ,𝑧 𝑖 ), 𝑖=1,…, 𝑁 𝑑 , with a sample from a conditional distribution: We want to sample the author and topic assignment The distribution of the authorship and topic assignment for one word conditioned on all other words and the assignments for all other words

“Sample the authorship and topic assignment for each word in each document” 𝐶 𝑚𝑗 𝑊𝑇 : the number of times a word m is assigned to topic j 𝐶 𝑘𝑗 𝐴𝑇 : the number of times (a word m with) topic j is assigned to author k 𝐶 𝑚𝑗 𝑊𝑇 +𝛽 𝑚′ 𝐶 𝑚′𝑗 𝑊𝑇 +𝑉𝛽 : smoothed probability of a word being sampled given a topic 𝐶 𝑘𝑗 𝐴𝑇 +𝛼 𝑗′ 𝐶 𝑘𝑗′ 𝐴𝑇 +𝑇𝛼 : smoothed probability of a topic being sampled given an author The product will converge after many iterations the distribution of the authorship and topic assignment for one word conditioned on all other words and the assignments for all other words

“Sample the authorship and topic assignment for each word in each document” 𝐶 𝑚𝑗 𝑊𝑇 : the number of times a word m is assigned to topic j 𝐶 𝑘𝑗 𝐴𝑇 : the number of times a word m with topic j is assigned to author k 𝐶 𝑚𝑗 𝑊𝑇 +𝛽 𝑚′ 𝐶 𝑚′𝑗 𝑊𝑇 +𝑉𝛽 : smoothed probability of a word being sampled given a topic 𝐶 𝑘𝑗 𝐴𝑇 +𝛼 𝑗′ 𝐶 𝑘𝑗′ 𝐴𝑇 +𝑇𝛼 : smoothed probability of a topic being sampled given an author The product will converge after many iterations the distribution of the authorship and topic assignment for one word conditioned on all other words and the assignments for all other words The two fractions are the probabilities documented in the two matrices! Doesn’t this sound familiar?

“Estimate 𝜃 and 𝜑 from the sample of assignments” How to do this? We have a formula for this, which is the converged probabilities/weights in the last step: Sure enough, we use the formula for the conditional distribution for the parameters

Outline Motivation Model formulation Parameter estimation Evaluation Generative process; plate notation; comparison with LDA Parameter estimation Gibbs sampling Evaluation Given a test document and its author(s), calculated perplexity score Application Predict the authors of a test document 1. Perplexity: the probability that the test document is generated by the generative model we built 2. To predict the authors, simply rank the perplexity score given different authors, the ones with lower perplexities are more likely to be the real authors

Takeaway By incorporating in the generative process a word-level author choosing and topic choosing according to an author-topic distribution, the Author-Topic Model manages to learn the relationship between authors and topics, and topic and words.

Takeaway By incorporating in the generative process a word-level author choosing and topic choosing according to an author-topic distribution, the Author-Topic Model manages to learn the relationship between authors and topics, and topic and words. Gibbs sampling is a solution for the difficulty of sampling from joint multivariate distributions and is used for inferring parameter values for generative models. The Author-Topic Model can also be used to predict authors of an unseen documents The learned relationship between authors and topics solved the problem raised in the motivation part

Thank you! Questions?