Download presentation
Presentation is loading. Please wait.
1
An Overview of Topic Modeling
Weifeng Li1,2 and Hsinchun Chen1 1 Artificial Intelligence Laboratory, The University of Arizona 2 University of Georgia
2
Acknowledgements Many of the pictures, results, and other materials are taken from: David Blei, Princeton University The Stanford Natural Language Processing Group
3
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
4
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
5
Introduction and Motivation
As more information is becoming easily available, it is difficult to find and discover what we need. Topic models are a suite of algorithms for discovering the main themes that pervade a large and other wise unstructured collection of documents. Among these algorithms, Latent Dirichlet Allocation (LDA), a technique based in Bayesian Modeling, is the most commonly used nowadays. Topic models can be applied to massive collections of documents to automatically organize, understand, search, and summarize large electronic archives. Especially relevant in today’s “Big Data” environment.
6
Introduction and Motivation
Each topic is a distribution of words; each document is a mixture of corpus-wide topics; and each word is drawn from one of those topics.
7
Introduction and Motivation
In reality, we only observe documents. The other structures are hidden variables. Our goal to infer the hidden variables.
8
Introduction and Motivation: 100-topic LDA, 17,000 Science articles
The resulting output from an LDA model would be sets of topics containing keywords which would then be manually labeled. On the left are the inferred topic proportions for the example article from the pervious figure.
9
Use Cases of Topic Modeling
Topic models have been used to: Annotate documents and images Organize and browse large corpora Model topic evolution Categorize source code archives Discover influential articles
10
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
11
Probabilistic Modeling Overview
Modeling: treat the data as arising from a generative process that includes hidden variables. This defines a joint distribution over both the observed and the hidden variables. Inference: infer the conditional distribution (posterior) of the hidden variables given the observed variables. Analysis: check the fit of the model; make prediction based on new data; explore the properties of the hidden variables. Modeling Inference Analysis Blei
12
Latent Dirichlet Allocation: Assumptions
LDA is a generative Bayesian model for topic modeling, which is built on the following assumptions: Assumptions on all variables: Word: the basic unit of discrete data Document: a collection of words (exchangeability assumption) Corpus: a collection of documents Topic (hidden): a distribution over words & the number of topics 𝐾 is known. Assumptions on how texts are generated: Dirichlet Dist. (next slide) For each topic 𝑘, draw a multinomial over words 𝛽 𝑘 ~𝐷𝑖𝑟 𝜂 For each document 𝑑, Draw a document topic proportion 𝜽 𝑑 ~𝐷𝑖𝑟 𝛼 For each word 𝑤 𝑑,𝑛 : Draw a topic 𝑧 𝑑,𝑛 ~𝑀𝑢𝑙𝑡𝑖 𝜽 𝑑 Draw a word 𝑤 𝑑,𝑛 ~𝑀𝑢𝑙𝑡𝑖( 𝛽 𝑧 𝑑,𝑛 )
13
Dirichlet Distribution: Dir(𝜶)
Named after Peter G. L. Dirichlet and often denoted as Dir(𝜶); A family of continuous multivariate probability distributions parameterized by a vector 𝜶 of positive reals. 𝑝 𝜽 𝜶 = Γ( 𝑘 𝛼 𝑘 ) 𝑘 Γ( 𝛼 𝑘 ) 𝑘 𝜃 𝑘 𝛼 𝑘 −1 Dir(𝜶) is the multivariate generalization of the beta distribution. Dirichlet distributions are often used as prior distributions in Bayesian statistics. Dirichlet distribution is the conjugate prior of the categorical distribution and multinomial distribution.( Conjugates distributions: the posterior distributions are in the same family as the prior distribution.)
14
LDA: Probabilistic Graphical Model
Per-document topics proportions 𝜃 𝑑 is a multinomial distribution, which is generated from Dirichlet distribution parameterized by 𝛼. Smilarly, topics 𝛽 𝑘 is also a multinomial distribution, which is generated from Dirichlet distribution parameterized by 𝜂. For each word 𝑛, its topic 𝑍 𝑑,𝑛 is drawn from document topic proportions 𝜃 𝑑 . Then, we draw the word 𝑊 𝑑,𝑛 from the topic 𝛽 𝑘 , where 𝑘= 𝑍 𝑑,𝑛 .
15
The Graphical Model for LDA: Joint Distribution
This distribution specifies a number of dependencies that define LDA (as shown in the plate diagram).
16
Inference Objective: computing the conditional distribution (posteriors) of the topic structure given the observed documents. 𝑝 𝜷,𝜽,𝒛 𝒘,𝛼 = 𝑝(𝜷,𝜽,𝒛,𝒘|𝛼) 𝑝(𝒘|𝛼) 𝑝(𝜷,𝜽,𝒛,𝒘|𝛼): the joint distribution of all the random variables, which is easy to compute 𝑝(𝒘|𝛼): the marginal probability of observations (the probability of seeing the observed corpus under any topic model), which is intractable. In theory, 𝑝(𝒘|𝛼) is computed by summing the joint distribution over every possible combination of 𝜷,𝜽,𝒛, which is exponentially large. Approximation methods: search over the topic structure Sampling-based algorithms attempt to collect samples from the posterior to approximate it with an empirical distribution. Variational algorithms posit a parameterized family of distributions over the hidden structure and then find the member of that family that is closest to the posterior.
17
More on Approximation Methods
In Sampling-based algorithms, Gibbs sampling is the most commonly used: Approximating the posterior with samples. Construct a Markov chain—a sequence of random variables, each dependent on the previous—whose limiting distribution is the posterior. The Markov chain is defined on the hidden topic variables for a particular corpus, and the algorithm is to run the chain for a long time, collect samples from the limiting distribution, and then approximate the distribution with the collected samples (see Steyers & Griffiths, 2006). Variational algorithms are a deterministic alternative to sampling-based algorithms. Posit a parametrized family of distributions over the hidden structure and then find the member of that family that is closet to the posterior. The inference problem is transformed to an optimization problem. Coordinate ascent variational inference algorithm for LDA (see Blei, Ng, and Jordan, 2003)
18
Model Evaluation: Perplexity
Perplexity is the most typical evaluation of LDA models (Bao & Datta, 2014; Blei et al., 2003). Perplexity measures the modeling power by calculating the inverse log- likelihood of unobserved documents (an decreasing function). (Higher likelihood, the better model) Better models have lower perplexity, suggesting less uncertainties about the unobserved document. Average log-likelihood of all unobserved document Log-likelihood of each unobserved document Wd: words in document d; Nd: Length of document d The figure compares LDA with other topic modeling approaches. The LDA model is consistently better than all other benchmark approaches. Moreover, as the number of topics go up, the LDA model becomes better (i.e., the perplexity decreases.)
19
Model Evaluation: Topic Coherence
Topic coherence evaluates the semantic nature of the learned topics. Specifically, it measures the semantic similarity among the top keywords of a topic. Topic coherence has shown to be correlated with human evaluations of topic quality. Topic coherence a topic 𝜷 𝑘 is calculated by: coherence 𝜷 𝑘 = ( 𝑤 𝑖 , 𝑤 𝑗 )∈ 𝑉 𝑛 score( 𝑤 𝑖 , 𝑤 𝑗 ) where 𝑉 𝑛 is the top 𝑛 keywords of the topic 𝜷 𝑘 There are two commonly used score metrics: The Extrinsic UCI Metric (Newman et al. 2010): score 𝑤 𝑖 , 𝑤 𝑗 = log 𝑝( 𝑤 𝑖 , 𝑤 𝑗 ) 𝑝 𝑤 𝑖 𝑝( 𝑤 𝑗 ) where 𝑝( 𝑤 𝑖 , 𝑤 𝑗 ) is the word co-occurrence probability of word pair 𝑤 𝑖 , 𝑤 𝑗 estimated from an external corpus (e.g., Wikipedia) and 𝑝( 𝑤 𝑖 ) is the probability of word 𝑤 𝑖 in the external corpus. The Intrinsic UMass Metric (Mimno et al. 2011): score 𝑤 𝑖 , 𝑤 𝑗 = log (D 𝑤 𝑖 , 𝑤 𝑗 +1) D( 𝑤 𝑗 ) where D( 𝑤 𝑖 , 𝑤 𝑗 ) counts the number of documents word pair 𝑤 𝑖 , 𝑤 𝑗 co-occurred and D( 𝑤 𝑗 ) counts the number of documents containing 𝑤 𝑗 . Newman, D., Lau, J. H., Grieser, K., & Baldwin, T. (2010, June). Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (pp ). Association for Computational Linguistics. Mimno, D., Wallach, H. M., Talley, E., Leenders, M., & McCallum, A. (2011, July). Optimizing semantic coherence in topic models. In Proceedings of the conference on empirical methods in natural language processing (pp ). Association for Computational Linguistics. Newman, D., Lau, J. H., Grieser, K., & Baldwin, T. (2010). Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (pp ). ACL. Mimno, D., Wallach, H. M., Talley, E., Leenders, M., & McCallum, A. (2011, July). Optimizing semantic coherence in topic models. In Proceedings of the conference on empirical methods in natural language processing (pp ). ACL.
20
Model Selection: How Many Topics to Choose
The author of LDA suggests to select the number of topics from 50 to 150 (Blei 2012); however, the optimal number usually depends on the size of the dataset. Cross validation on perplexity is often used for selecting the number of topics. Specifically, we propose possible numbers of topics first, evaluate the average perplexity using cross validation, and pick the number of topics that has the lowest perplexity. The following plot illustrates the selection of optimal number of topics for 4 datasets.
21
LDA Research Example – Profiling Underground Economy Sellers
The underground economy is the online black market for exchanging products/services that relate to cybercrimes. Cyber crime activities have been mostly commoditized in the underground economy. Sellers impose a growing threat to cyber security. Sellers advertise their products/services by giving details about their resources, payments, contacts, etc. Objective: to profile underground economy sellers to reflect their specialties(characteristics) Li, W., Chen, H., & Nunamaker Jr, J. F. (2016). Identifying and profiling key sellers in cyber carding community: AZSecure text mining system. Journal of Management Information Systems, 33(4),
22
LDA Research Example – Profiling Underground Economy Sellers
Input: Original threads from hacker forums Preprocessing: Thread Retrieval: Identifying threads related to the underground economy by conducting snowball sampling-based keywords search Thread Classification: Identifying advertisement threads using MaxEnt classifier Focusing on malware advertisements and stolen card advertisement Can be generalized to other advertisements.
23
LDA Research Example – Profiling Underground Economy Sellers
To profile the seller, we seek to identify the major topics in its advertisement. Example input: 23 Seller of stolen data: Rescator Description of the stolen data/service Prices of the stolen data Contact: a dedicated shop and ICQ Payment Options
24
LDA Research Example – Profiling Underground Economy Sellers
For LDA model selection, we use perplexity to choose the optimal number of topics for the advertisement corpus. Output: LDA gives the probabilities of each topics associated with the seller. We pick the top-𝐾 topics to profile the seller (𝐾=5 in our example). For each topic, we pick the top-𝐽 keywords to interpret the topic (𝐽=10 in our example). The following table helps us to profile Rescator based on its characteristics in terms of the product, the payment, and the contact. Top Seller Characteristics of Rescator # Top Keywords Interpretation 5 shop, wmz, icq, webmoney, price, dump, Product: CCs, dumps (valid, verified); Payment: wmz, webmoney, bitcoin, lesspay; Contact: shop, register, deposit, , icq, jabber 6 валид(valid), чекер(checker), карты(cards), баланс(balance), карт(cards) 8 shop, good, CCs, bases, update, cards, bitcoin, webmoney, validity, lesspay 11 dollars, dumps, deposit, payment, sell, online, verified 16 , shop, register, icq, account, jabber,
25
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
26
LDA Variants: Relaxing the Assumptions of LDA
Consider the order of the words: words in a document cannot be exchanged Conditioning on the previous word (Wallach 2006) Hidden Markov Model (Griffiths et al. 2005) Consider the order of the documents Dynamic LDA (Blei & Lafferty 2006) Consider previously unseen topics: the number of topics is not fixed Bayesian Nonparametrics (Blei et al. 2010)
27
Dynamic LDA Motivation: In Dynamic LDA, topic evolves over time.
LDA assumes the order of documents does not matter (Not appropriate for sequential corpora) We want to capture language change over time. In Dynamic LDA, topic evolves over time. Dynamic LDA uses a logistic normal distribution to model topics evolving over time. Topics drift through time Example: Blei, D. M., and Lafferty, J. D “Dynamic topic models,” in Proceedings of the 23rd international conference on Machine learning (ICML 2006), pp. 113–120 (doi: / ).
28
Bayesian Nonparametric Topic Modeling: Hierarchical Dirichlet Process
Bayesian nonparametrics: the parameter space has infinite imension Nonparametric topic model: a topic model whose topic space has infinite dimension (i.e., an infinite number of topics) Less vulnerable to model overfitting or underfitting caused by the misspecification of topic number A prominent nonparametric topic model is hierarchical Dirichlet Process (HDP) (Teh et al. 2006) HDP is increasingly chosen for modeling topics over LDA for its reliability and flexibility. Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006). Hierarchical Dirichlet Processes. Journal of the American Statistical Association,
29
Dirichlet Process: the Major Building Block of HDP
Dirichlet process (DP): A probability distribution of discrete distributions over the topic space with probability one. 𝐺 0 ~𝐷𝑃 𝜔,𝐻 𝜔 (concentration parameter): how concentrated the discrete distributions over topics drawn from the DP are; 𝐻 (base distribution): determines the topic space and the expectation of the discrete distributions drawn from the DP. Two nice properties of DP samples that allow for modeling topics: Clustering property: topics previously drawn from the distribution will likely to be drawn again, allowing the words within a document to be clustered under certain topics. Infinity property: once can drawn as many as topics as needed because DP samples are distributions over topics; therefore, the number of topics is unbounded.
30
Example: 𝐺 0 ~𝐷𝑃 𝜔,𝐺𝑎𝑢𝑠𝑠𝑖𝑎𝑛(0,1)
𝜔=1 𝜔=10 𝜔=100 𝜔=1000 Sample 𝐺 0 #1 Sample 𝐺 0 #2 Sample 𝐺 0 #3 𝐺 0 : Discrete distribution with probability one. Can take an infinite number of values as sampled from 𝐺𝑎𝑢𝑠𝑠𝑖𝑎𝑛(0,1). Values sampled from 𝐺𝑎𝑢𝑠𝑠𝑖𝑎𝑛(0,1) are often repeated, thus forming clusters (each spike represents one cluster.)
31
Hierarchical Dirichlet Process (HDP)
For the corpus: 𝐺 0 ~𝐷𝑃 𝜔,𝐻 , For each document: 𝐺 𝑑 ~𝐷𝑃(𝛼, 𝐺 0 ) Corpus Topic Distribution Document Topic Distribution Per-word Topic Observed Word Dirichlet process (DP) allows for modeling an infinite number of topics. 𝐻: base topic distribution (e.g., Dirichlet distribution Dir(𝜶)); 𝜔: corpus topic concentration parameter; 𝛼: document topic concentration parameter
32
HDP Model Specifications:
Variable Distribution Description 𝚽 Laplace distribution (Taddy 2013): 𝜙 𝑤 ~𝐿𝑎𝑝𝑙𝑎𝑐𝑒(𝜆) Coefficient of the response 𝑦 𝑑 on word 𝑤 𝑮 0 Dirichlet Process (Teh et al. 2006): 𝐺 0 ~𝐷𝑃 𝜔,𝐻 , 𝐻: distribution over topics Corpus topic distribution Collection of all possible topics that will be used in the corpus 𝑮 𝑑 𝐺 𝑑 ~𝐷𝑃(𝛼, 𝐺 0 ) Document topic distribution Collection of all possible topics for the document 𝜷 𝑑,𝑛 Multinomial distribution (Blei et al. 2003): 𝜷 𝑑𝑛 ~ 𝐺 𝑑 Topic of the 𝑛-th word in the 𝑑-th document 𝑤 𝑑,𝑛 Multinomial distribution (Rabinovich & Blei 2014): 𝑤 𝑑𝑛 ~𝑀𝑢𝑙𝑡𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝜷 𝑑𝑛 Observed 𝑛-th word in the 𝑑-th document Hierarchical Dirichlet Process hide
33
Latent Dirichlet Allocation vs. Hierarchical Dirichlet Process
Corpus-Level Topics Corpus-Level Topics Topics 1 2 3 4 5 ∞ Topics Document 1 (Topic Prop.) Document 1 (Topic Dist.) Topics 1 2 3 4 5 Prop. .1 .2 .4 Document 2 (Topic Prop.) Topics 1 2 3 4 5 Prop. .3 .2 .1 Document 2 (Topic Dist.) …… …… Document D (Topic Prop.) Topics 1 2 3 4 5 Prop. .3 .2 .1 Document D (Topic Dist.)
34
Hierarchical Dirichlet Process Performance: Document Modeling on Academic Papers
Perplexity of LDA versus HDP Posterior number of topics in HDP (a) (b) (a): HDP performed as well as the best LDA model, doing so without any form of model selection procedure. (b): The posterior over the number of topics obtained under the HDP model is consistent with this range of the best-fitting LDA models. Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006). Hierarchical Dirichlet Processes. Journal of the American Statistical Association,
35
LDA Variants: Incorporating Metadata
Account for metadata of the documents (e.g., author, title, geographic location, links, etc.) Author-topic model (Rosen-Zvi et al. 2004) Assumption: The topic proportions are attached to authors. Allows for inferences about authors, for example, author similarity. Relational topic model (Chang & Blei 2010) Documents are linked (e.g., citation, hyperlink) Assumption: links between documents depend on the distance between their topic proportions. Takes into account node attributes (the words of the document) in modeling the network links. Supervised topic model (Blei & McAuliffe 2007) A general purpose method for incorporating metadata into topic models
36
From Topic Model to Supervised Topic Model (STM)
Documents (e.g., reviews) 𝑲 topics: Responses (e.g., service quality) picture 0.04 video tutorial price delivery 0.04 service model device machine 0.01 cvv vbv ssn pos skimmer 0.02 encrypt 0.01 Descriptive (unsupervised) Predictive (Supervised) 𝒘 𝒚 𝛽 1 𝛽 2 𝛽 3 𝛽 4 𝛽 5 STM can simultaneously Explore the underlying topics Make predictions using the extracted topics More accurate prediction by capturing the underlying topics shared across documents (Blei & Mcauliffe 2008) Existing STMs: Supervised LDA, Dirichlet Multinomial Regression, Discriminative LDA, Dependency-LDA, Inverse Regression Topic Model (IRTM)…
37
Supervised LDA Supervised LDA are topic models of documents and response variables. They are fit to find topics predictive of the response variable. How many topics? rating 10-topic sLDA model on movie reviews (Pang and Lee, 2005): identifying the topics correspond to ratings Blei, D. M., and Mcauliffe, J. D “Supervised Topic Models,” in Advances in neural information processing systems, pp. 121–128 (doi: /asmb.540).
38
Parametric vs. Nonparametric
picture 0.04 video tutorial price delivery 0.04 service model device machine 0.01 cvv vbv ssn pos skimmer 0.02 encrypt 0.01 Documents (e.g., reviews) 𝒘 𝒚 𝛽 1 𝛽 2 𝛽 3 𝛽 4 𝛽 5 𝑲 topics: Responses (e.g., service quality) Proportion of predefined number of topics Parametric Approach: picture 0.04 video tutorial price delivery 0.04 service model device machine 0.01 cvv vbv ssn pos skimmer 0.02 encrypt 0.01 Documents (e.g., reviews) 𝒘 𝒚 ∞ topics: Responses (e.g., service quality) Distribution of unlimited number of topics Nonparametric Approach: non parametric flexibility to determine the number of topics potential number of topics is unlimited
39
Supervised HDP: Nonparametric Supervised Topic Model
Corpus Topic Distribution Document Topic Distribution Per-word Topic Observed Word Response Coefficient Latent variables Observed data
40
Mean Absolute Error (MAE) Root Mean Square Error (RMSE)
At least 17% improvement over closest technique Better Predictive R-squared Failed to converge Failed to converge Mean Absolute Error (MAE) Failed to converge Failed to converge Better Root Mean Square Error (RMSE) Failed to converge Failed to converge Better Li, W., Yin, J., & Chen, H. (2018). Supervised Topic Modeling Using Hierarchical Dirichlet Process-Based Inverse Regression: Experiments on E-Commerce Applications. IEEE Transactions on Knowledge and Data Engineering, 30(6),
41
LDA Variants: Coupling with Deep Learning
Neural Topic Model (Miao et al. 2017) Variational autoencoder (VAE): a deep learning technique for approximating posterior distribution using the autoencoder framework. Building upon deep learning’s improved capability of approximating non- linear relationships, VAE generally provides better model inference than traditional variational inference algorithms. Neural topic model: leveraging VAE for model inference Word embedding-based topic modeling (Das et al. 2015) Word embedding: representing word semantics in a continuous vector space Semantically related words tend to be closer to each other in the vector space. Word embedding-based topic: a distribution over the vector space (instead of a multinomial distribution over words) This encourages the model to group words that are a priori known to be semantically related into topics. Das, R., Zaheer, M., & Dyer, C. (2015). Gaussian LDA for Topic Models with Word Embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (Vol. 1, pp. 795–804). Miao, Y., Grefenstette, E., & Blunsom, P. (2017). Discovering Discrete Latent Topics with Neural Variational Inference. In D. Precup & Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 2410–2419).
42
LDA Variants: Generalizing to Other Kinds of Data
LDA is mixed-membership model of grouped data. Rather than associating each group of data with one topic, each group exhibits multiple topics in different proportions. Hence, LDA can be adapted to other kinds of observations with only small changes to the corresponding inference algorithms. Population genetics Application: finding ancestral populations Intuition: each individual’s genotype descends from one or more of the ancestral populations (topics) Computer vision Application: classifying images, connect images and captions, build image hierarchies, etc. Intuition: each image exhibits a combination of visual patterns and that the same visual patterns recur throughout a collection of images
43
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
44
Future Directions Evaluation and model checking
Held-out accuracy may not correspond to better organization or easier interpretation (Amazon Mechanical Turk experiment; see Chang et al., 2009; perplexity is not strongly correlated to human judgement; sometimes even slightly anti-correlated) Which topic model should I use? How can I decide which of the many modeling assumptions are important for my goals? Visualization and user interfaces How to display the topics? How to best display a document with a topic model? How can we best display document connections?
45
Future Directions Topic models for data discovery Topic interpretation
How can topic models help us form hypothesis about the data? What can we learn about the language based on the topic model posterior? Topic interpretation Topics are distributions over words; therefore, interpreting topics semantics from these distributions becomes important. How to properly interpret a topic? Multilingual topic modeling Whether the same topic can appear in different languages? How to find common topics across different languages?
46
Outline Introduction and Motivation Latent Dirichlet Allocation
Probabilistic Modeling Overview LDA Assumptions Inference Evaluation Research Example: LDA Application in Profiling Underground Economy Sellers LDA Variants Relaxing the Assumptions of LDA Incorporating Metadata Coupling with Deep Learning Generalizing to Other Kinds of Data Future Directions Tools & Implementation Details
47
Topic Modeling - Tools Name Model/Algorithm Language Author Notes
lda-c Latent Dirichlet allocation C D. Blei This implements variational inference for LDA. class-slda Supervised topic models for classification C++ C. Wang Implements supervised topic models with a categorical response. lda R package for Gibbs sampling in many models R J. Chang Implements many models and is fast . Supports LDA, RTMs (for networked documents), MMSB (for network data), and sLDA (with a continuous response). gensim Software Framework for Topic Modelling with Large Corpora Python R. Rehurek, P. Sojka Provides distribution implementation of many models, including LDA, author-topic model, HDP, Dynamic topic model. tmve Topic Model Visualization Engine A. Chaney A package for creating corpus browsers. dtm Dynamic topic models and the influence model S. Gerrish This implements topics that change over time and a model of how individual documents predict that change. ctm-c Correlated topic models This implements variational inference for the CTM. Mallet LDA, Hierarchical LDA Java A. McCallum Implements LDA and Hierarchical LDA Stanford topic modeling toolbox LDA, Labeled LDA, Partially Labeled LDA Stanford NLP Group Implements LDA, Labeled LDA, and PLDA
48
gensim (Python) Example: Topic Modeling News Articles
We use the news article dataset from the Lee corpus. test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data']) lee_train_file = test_data_dir + os.sep + 'lee_background.cor‘ def build_texts(fname): with open(fname) as f: for line in f: yield gensim.utils.simple_preprocess(line, deacc=True, min_len=3) train_texts = list(build_texts(lee_train_file)) bigram = gensim.models.Phrases(train_texts) def process_texts(texts): texts = [[word for word in line if word not in stops] for line in texts] texts = [bigram[line] for line in texts] texts = [[word.split('/')[0] for word in lemmatize(' '.join(line), allowed_tags=re.compile('(NN)'), min_length=3)] for line in texts] return texts train_texts = process_texts(train_texts) dictionary = Dictionary(train_texts) corpus = [dictionary.doc2bow(text) for text in train_texts] Data import & preprocessing Bigram collocation detection Creating a bag-of-words model Tutorial retrieved from:
49
gensim (Python) Example: Topic Modeling News Articles
Specifying the number of topics ldamodel = gensim.models.LdaModel(corpus=corpus, num_topics=10, id2word=dictionary) ldatopics = ldamodel.show_topics(formatted=False) import pyLDAvis.genism pyLDAvis.enable_notebook() pyLDAvis.gensim.prepare(ldamodel, corpus, dictionary) Executing LDA & visualize Example output: Tutorial retrieved from:
50
gensim (Python) Example: Topic Modeling News Articles
Executing HDP & showing learned topics hdpmodel = gensim.models.HdpModel(corpus=corpus, id2word=dictionary) hdptopics = hdpmodel.show_topics(formatted=False) hdpmodel.show_topics() Example output: [u'topic 0: 0.004*collapse *afghanistan *troop *force *government *benefit *operation *taliban *time *today *ypre *tourism *person *help *wayne *fire *peru *day *united_state *hih', u'topic 1: 0.003*group *government *target *palestinian *end *terrorism *cease *memorandum *radio *call *official *path *security *wayne *attack *human_right *four *gunman *sharon *subsidiary', u'topic 2: 0.003*rafter *double *team *reality *manager *cup *australia *abc *nomination *user *freeman *herberton *lung *believe *injury *steve_waugh *fact *statement *mouth *alejandro', … Each topic is returned with its most probable keywords and its associated weights. Tutorial retrieved from:
51
gensim (Python) Example: Topic Modeling News Articles
We now evaluate the HDP model in comparison with the LDA model as measured by topic coherence. hdptopics = [[word for word, prob in topic] for topicid, topic in hdptopics] ldatopics = [[word for word, prob in topic] for topicid, topic in ldatopics] hdp_coherence = gensim.models.CoherenceModel(topics=hdptopics[:10], texts=train_texts, dictionary=dictionary, window_size=10).get_coherence() lda_coherence = gensim.models.CoherenceModel(topics=ldatopics, texts=train_texts, dictionary=dictionary, window_size=10).get_coherence() def evaluate_bar_graph(coherences, indices): assert len(coherences) == len(indices) n = len(coherences) x = np.arange(n) plt.bar(x, coherences, width=0.2, tick_label=indices, align='center') plt.xlabel('Models') plt.ylabel('Coherence Value') evaluate_bar_graph([hdp_coherence, lda_coherence], ['HDP', 'LDA']) Calculating topic coherence Visualize topic coherence comparison The output bar graph is on the next slide. Tutorial retrieved from:
52
gensim (Python) Example: Topic Modeling News Articles
The figure on the right shows the comparison of the performance between HDP and LDA, as measured by topic coherence. As suggested in the figure, HDP generates more coherent topics than LDA. Tutorial retrieved from:
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.