1. Probabilistic Topic Models for Text Mining
ChengXiang Zhai (翟成祥)
Department of Computer Science
Graduate School of Library & Information Science
Institute for Genomic Biology, Statistics
University of Illinois, Urbana-Champaign
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

2. (Slide from Rebecca Hwa’s “Intro to Text Mining”)
What Is Text Mining?
“The objective of Text Mining is to exploit information contained in textual documents in various ways, including …discovery of patterns and trends in data, associations among entities, predictive rules, etc.” (Grobelnik et al., 2001)
“Another way to view text data mining is as a process of exploratory data analysis that leads to heretofore unknown information, or to answers for questions for which the answer is not currently known.” (Hearst, 1999)
(Slide from Rebecca Hwa’s “Intro to Text Mining”)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

3. Two Different Views of Text Mining
Data Mining View: Explore patterns in textual data
Find latent topics
Find topical trends
Find outliers and other hidden patterns
Natural Language Processing View: Make inferences based on partial understanding natural language text
Information extraction
Question answering
Shallow mining
Deep mining
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

4. Applications of Text Mining
Direct applications: Go beyond search to find knowledge
Question-driven (Bioinformatics, Business Intelligence, etc): We have specific questions; how can we exploit data mining to answer the questions?
Data-driven (WWW, literature, , customer reviews, etc): We have a lot of data; what can we do with it?
Indirect applications
Assist information access (e.g., discover latent topics to better summarize search results)
Assist information organization (e.g., discover hidden structures)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

5. Text Mining Methods Topic of this lecture
Data Mining Style: View text as high dimensional data
Frequent pattern finding
Association analysis
Outlier detection
Information Retrieval Style: Fine granularity topical analysis
Topic extraction
Exploit term weighting and text similarity measures
Natural Language Processing Style: Information Extraction
Entity extraction
Relation extraction
Sentiment analysis
Question answering
Machine Learning Style: Unsupervised or semi-supervised learning
Mixture models
Dimension reduction
Topic of this lecture
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

6. Outline The Basic Topic Models: Extensions
Probabilistic Latent Semantic Analysis (PLSA) [Hofmann 99]
Latent Dirichlet Allocation (LDA) [Blei et al. 02]
Extensions
Contextual Probabilistic Latent Semantic Analysis (CPLSA) [Mei & Zhai 06]
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

7. Basic Topic Model: PLSA
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

8. PLSA: Motivation
What did people say in their blog articles about “Hurricane Katrina”?
Query = “Hurricane Katrina”
Results:
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

9. Probabilistic Latent Semantic Analysis/Indexing (PLSA/PLSI) [Hofmann 99]
Mix k multinomial distributions to generate a document
Each document has a potentially different set of mixing weights which captures the topic coverage
When generating words in a document, each word may be generated using a DIFFERENT multinomial distribution (this is in contrast with the document clustering model where, once a multinomial distribution is chosen, all the words in a document would be generated using the same model)
We may add a background distribution to “attract” background words
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

10. in doc d in the collection
PLSA as a Mixture Model
Document d
warning 0.3 system ?
Topic 1
d,1 1
“Generating” word w
in doc d in the collection
aid 0.1 donation 0.05 support 2
Topic 2
d,2
1 - B
d, k W … k
statistics 0.2 loss dead B
Topic k B
is 0.05 the a
Background B
Parameters:
B=noise-level (manually set)
’s and ’s are estimated with Maximum Likelihood
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

11. How to Estimate j: EM Algorithm
the 0.2
a 0.1
we 0.01
to 0.02 …
Known
Background
p(w | B)
Observed Doc(s) ML
Estimator
Suppose,
we know
the identity
of each
word ...
Unknown
topic model
p(w|1)=?
“Text mining” …
text =?
mining =?
association =?
word =? … …
information =?
retrieval =?
query =?
document =?
Unknown
topic model
p(w|2)=?
“information retrieval”
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

12. How the Algorithm Works
c(w,d)p(zd,w = B)
c(w,d)(1 - p(zd,w = B))p(zd,w=j)
πd1,1 ( P(θ1|d1) )
πd1,2 ( P(θ2|d1) )
c(w, d)
aid 7 d1
price 5
Initial value
oil 6
πd2,1 ( P(θ1|d2) )
πd2,2 ( P(θ2|d2) )
aid 8 d2
price 7
oil 5
Initial value
P(w| θ)
Topic 1
Topic 2
Iteration 1: M Step: re-estimate πd, j and P(w| θj) by adding and normalizing the splitted word counts
Initializing πd, j and P(w| θj) with random values
Iteration 1: E Step: split word counts with different topics (by computing z’ s)
Iteration 2: E Step: split word counts with different topics (by computing z’ s)
Iteration 2: M Step: re-estimate πd, j and P(w| θj) by adding and normalizing the splitted word counts
Iteration 3, 4, 5, …
Until converging
aid
price
Initial value 12
oil
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

13. Application of Bayes rule (in multiple collections)
Parameter Estimation
E-Step:
Word w in doc d is generated
from cluster j
from background
Application of Bayes rule
M-Step:
Re-estimate
mixing weights
cluster LM
Fractional counts contributing to
using cluster j in generating d
generating w from cluster j
Sum over all docs
(in multiple collections)
m = 1 if one collection
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

14. PLSA with Prior Knowledge
There are different ways of choosing aspects (topics)
Google = Google News + Google Map + Google scholar, …
Google = Google US + Google France + Google China, …
Users have some domain knowledge in mind, e.g.,
We expect to see “retrieval models” as a topic in IR.
We want to show the aspects of “history” and “statistics” for Youtube
A flexible way to incorporate such knowledge as priors of PLSA model
In Bayesian, it’s your “belief” on the topic distributions 14
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

15. in doc d in the collection
Adding Prior
Most likely 
Document d
warning 0.3 system
Topic 1
d,1 1
“Generating” word w
in doc d in the collection
aid 0.1 donation 0.05 support 2
Topic 2
d,2
1 - B
d, k W … k
statistics 0.2 loss dead B
Topic k B
is 0.05 the a
Background B
Parameters:
B=noise-level (manually set)
’s and ’s are estimated with Maximum Likelihood 15
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

16. Adding Prior as Pseudo Counts
Observed Doc(s)
the 0.2
a 0.1
we 0.01
to 0.02 …
Known
Background
p(w | B)
MAP
Estimator
Suppose,
we know
the identity
of each
word ...
Unknown
topic model
p(w|1)=?
“Text mining” …
text =?
mining =?
association =?
word =?
Pseudo Doc …
Unknown
topic model
p(w|2)=?
“information retrieval” …
information =?
retrieval =?
query =?
document =?
Size = μ
text 16
mining
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

17. Maximum A Posterior (MAP) Estimation
Pseudo counts of w from prior ’
+p(w|’j) +
Sum of all pseudo counts
What if =0? What if =+? 17
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

18. Basic Topic Model: LDA
The following slides about LDA are taken from Michael C. Mozer’s course lecture
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

19. LDA: Motivation “Documents have no generative probabilistic semantics”
i.e., document is just a symbol
Model has many parameters
linear in number of documents
need heuristic methods to prevent overfitting
Cannot generalize to new documents
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

20. Unigram Model
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

21. Mixture of Unigrams
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

22. Topic Model / Probabilistic LSI
d is a localist representation of (trained) documents
LDA provides a distributed representation
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

23. LDA Vocabulary of |V| words
Document is a collection of words from vocabulary.
N words in document
w = (w1, ..., wN)
Latent topics
random variable z, with values 1, ..., k
Like topic model, document is generated by sampling a topic from a mixture and then sampling a word from a mixture.
But topic model assumes a fixed mixture of topics (multinomial distribution) for each document.
LDA assumes a random mixture of topics (Dirichlet distribution) for each topic.
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

24. Generative Model “Plates” indicate looping structure
Outer plate replicated for each document
Inner plate replicated for each word
Same conditional distributions apply for each replicate
Document probability
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

25. Fancier Version
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

26. Inference
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

27. Inference In general, this formula is intractable: Expanded version:
1 if wn is the j'th vocab word
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

28. Variational Approximation
Computing log likelihood and introducing Jensen's inequality: log(E[x]) >= E[log(x)]
Find variational distribution q such that the above equation is computable.
q parameterized by γ and φn
Maximize bound with respect to γ and φn to obtain best approximation to p(w | α, β)
Lead to variational EM algorithm
Sampling algorithms (e.g., Gibbs sampling) are also common
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

29. Data Sets C. Elegans Community abstracts TREC AP corpus (subset)
28,414 unique terms
TREC AP corpus (subset)
16,333 newswire articles
23,075 unique terms
Held-out data – 10%
Removed terms
50 stop words, words appearing once
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

30. Note: fold in hack for pLSI to allow it to handle novel documents.
C. Elegans
Note: fold in hack for pLSI to allow it to handle novel documents.
Involves refitting p(z|dnew) parameters -> sort of a cheat
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

31. AP
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

32. Summary: PLSA vs. LDA
LDA adds a Dirichlet distribution on top of PLSA to regularize the model
Estimation of LDA is more complicated than PLSA
LDA is a generative model, while PLSA isn’t
PLSA is more likely to over-fit the data than LDA
Which one to use?
If you need generalization capacity, LDA
If you want to mine topics from a collection, PLSA may be better (we want overfitting!)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

33. Extension of PLSA: Contextual Probabilistic Latent Semantic Analysis (CPLSA)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

34. A General Introduction to EM
Data: X (observed) + H(hidden) Parameter: 
“Incomplete” likelihood: L( )= log p(X| )
“Complete” likelihood: Lc( )= log p(X,H| )
EM tries to iteratively maximize the incomplete likelihood:
Starting with an initial guess (0),
1. E-step: compute the expectation of the complete likelihood
2. M-step: compute (n) by maximizing the Q-function
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

35. Convergence Guarantee
Goal: maximizing “Incomplete” likelihood: L( )= log p(X| )
I.e., choosing (n), so that L((n))-L((n-1))0
Note that, since p(X,H| ) =p(H|X, ) P(X| ) , L() =Lc() -log p(H|X, )
L((n))-L((n-1)) = Lc((n))-Lc( (n-1))+log [p(H|X,  (n-1) )/p(H|X, (n))]
Taking expectation w.r.t. p(H|X, (n-1)),
L((n))-L((n-1)) = Q((n);  (n-1))-Q( (n-1);  (n-1)) + D(p(H|X,  (n-1))||p(H|X,  (n)))
Doesn’t contain H
EM chooses (n) to maximize Q
KL-divergence, always non-negative
Therefore, L((n))  L((n-1))!
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

36. Another way of looking at EM
Likelihood p(X| )
L((n-1)) + Q(; (n-1)) -Q( (n-1);  (n-1) ) + D(p(H|X,  (n-1) )||p(H|X,  ))
L((n-1)) + Q(; (n-1)) -Q( (n-1);  (n-1) )
next guess
current guess
Lower bound
(Q function) 
E-step = computing the lower bound
M-step = maximizing the lower bound
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

37. Why Contextual PLSA?
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

38. Motivating Example: Comparing Product Reviews
IBM Laptop
Reviews
APPLE Laptop
Reviews
DELL Laptop
Reviews
Common Themes
“IBM” specific
“APPLE” specific
“DELL” specific
Battery Life
Long, 4-3 hrs
Medium, 3-2 hrs
Short, 2-1 hrs
Hard disk
Large, GB
Small, 5-10 GB
Medium, GB
Speed
Slow, Mhz
Very Fast, 3-4 Ghz
Moderate, 1-2 Ghz
Unsupervised discovery of common topics and their variations
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

39. Motivating Example: Comparing News about Similar Topics
Vietnam War
Afghan War
Iraq War
Common Themes
“Vietnam” specific
“Afghan” specific
“Iraq” specific
United nations …
Death of people
Unsupervised discovery of common topics and their variations
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

40. Motivating Example: Discovering Topical Trends in Literature
Theme Strength
Time
Explain the Plots. Temporal theme analysis separate to make the fonts bigger… more explanations.
Title: sample ETP: theme evolutionary graph
1980
1990
1998
2003
TF-IDF Retrieval
Language Model
IR Applications
Text Categorization
Unsupervised discovery of topics and their temporal variations
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

41. Motivating Example: Analyzing Spatial Topic Patterns
How do blog writers in different states respond to topics such as “oil price increase during Hurricane Karina”?
Unsupervised discovery of topics and their variations in different locations
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

42. Motivating Example: Sentiment Summary
Unsupervised/Semi-supervised discovery of topics and
different sentiments of the topics
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

43. Research Questions Can we model all these problems generally?
Can we solve these problems with a unified approach?
How can we bring human into the loop?
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

44. Contextual Text Mining
Given collections of text with contextual information (meta-data)
Discover themes/subtopics/topics (interesting word clusters)
Compute variations of themes over contexts
Applications:
Summarizing search results
Federation of text information
Opinion analysis
Social network analysis
Business intelligence ..
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

45. Context Features of Text (Meta-data)
Weblog Article
communities
Author
Compared with other kinds of data, Weblogs have some interesting special characteristics, which make it interesting to exploit for text mining.
source
Location
Time
Author’s Occupation
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

46. Context = Partitioning of Text
Papers about Web
papers written in 1998
1998
papers written by authors in US
1999 …… ……
2005
2006
WWW
SIGIR
ACL
KDD
SIGMOD
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

47. Themes/Topics Uses of themes:
government 0.3 response
[ Criticism of government response to the hurricane primarily consisted of criticism of its response to the approach of the storm and its aftermath, specifically in the delayed response ] to the [ flooding of New Orleans. … 80% of the 1.3 million residents of the greater New Orleans metropolitan area evacuated ] …[ Over seventy countries pledged monetary donations or other assistance]. …
Theme 1
donate 0.1 relief 0.05 help
Theme 2 …
city 0.2 new orleans
Theme k
Is 0.05 the a
Uses of themes:
Summarize topics/subtopics
Navigate in a document space
Retrieve documents
Segment documents …
Background B
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

48. View of Themes: Context-Specific Version of Views
Context: After 1998 (Language models)
feedback
language
model
smoothing
query
generation
mixture
estimate EM
pseudo
vector
space
TF-IDF
Okapi
LSI
Rocchio
weighting
feedback
term
retrieval
Theme 1:
Retrieval Model
retrieve
model
relevance
document
query
feedback
judge
expansion
pseudo
query
Theme 2:
Feedback
Context: Before 1998 (Traditional models)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

49. Coverage of Themes: Distribution over Themes
Oil Price
Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. …
Government Response
Aid and donation
Background
Context: Texas
Oil Price
Government Response
Theme coverage can depend on context
Aid and donation
Background
Context: Louisiana
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

50. General Tasks of Contextual Text Mining
Theme Extraction: Extract the global salient themes
Common information shared over all contexts
View Comparison: Compare a theme from different views
Analyze the content variation of themes over contexts
Coverage Comparison: Compare the theme coverage of different contexts
Reveal how closely a theme is associated to a context
Others:
Causal analysis
Correlation analysis
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

51. A General Solution: CPLSA
CPLAS = Contextual Probabilistic Latent Semantic Analysis
An extension of PLSA model ([Hofmann 99]) by
Introducing context variables
Modeling views of topics
Modeling coverage variations of topics
Process of contextual text mining
Instantiation of CPLSA (context, views, coverage)
Fit the model to text data (EM algorithm)
Compute probabilistic topic patterns
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

52. “Generation” Process of CPLSA
Choose a theme
View1
View2
View3
Themes
government
donation
New Orleans
Draw a word from i
Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. …
Document context:
Time = July 2005
Location = Texas
Author = xxx
Occup. = Sociologist
Age Group = 45+ …
government 0.3 response
donate 0.1 relief 0.05 help
city 0.2 new orleans
government
response
donate
help
aid
Orleans
new
Texas
July 2005
sociologist
Choose a view
Theme coverages:
Texas
July 2005
document ……
Choose a Coverage
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

53. Probabilistic Model
To generate a document D with context feature set C:
Choose a view vi according to the view distribution
Choose a coverage кj according to the coverage distribution
Choose a theme according to the coverage кj
Generate a word using
The likelihood of the document collection is:
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

54. Parameter Estimation: EM Algorithm
Interesting patterns:
Theme content variation for each view:
Theme strength variation for each context
Prior from a user can be incorporated using MAP estimation
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

55. Regularization of the Model
Why?
Generality high complexity (inefficient, multiple local maxima)
Real applications have domain constraints/knowledge
Two useful simplifications:
Fixed-Coverage: Only analyze the content variation of themes (e.g., author-topic analysis, cross-collection comparative analysis )
Fixed-View: Only analyze the coverage variation of themes (e.g., spatiotemporal theme analysis)
In general
Impose priors on model parameters
Support the whole spectrum from unsupervised to supervised learning
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

56. Interpretation of Topics
Statistical topic models
term relevance weight feedback independence
model
frequent
probabilistic
document …
Multinomial topic models
Collection (Context)
Relevance Score
Re-ranking
Coverage; Discrimination
NLP Chunker
Ngram stat.
database system, clustering algorithm, r tree, functional dependency, iceberg cube, concurrency control, index structure …
Candidate label pool
Ranked List of Labels
clustering algorithm;
distance measure; …
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

57. Relevance: the Zero-Order Score
Intuition: prefer phrases covering high probability words
Clustering
Good Label (l1): “clustering algorithm”
dimensional
algorithm
Latent Topic  …
birch
shape
Bad Label (l2): “body shape” …
p(w|)
body
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

58. Relevance: the First-Order Score
Intuition: prefer phrases with similar context (distribution)
Clustering
Clustering
Clustering
dimension
dimension
dimension
Good Label (l1): “clustering algorithm”
Bad Label (l2): “hash join”
Topic  …
partition
partition
algorithm
algorithm
algorithm
C: SIGMOD Proceedings
join … …
Score (l,  )
hash
hash
hash
P(w|)
P(w|l1)
P(w|l2)
D(||l1) < D(||l2)
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

59. Sample Results Comparative text mining Spatiotemporal pattern mining
Sentiment summary
Event impact analysis
Temporal author-topic analysis
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

60. Comparing News Articles Iraq War (30 articles) vs
Comparing News Articles Iraq War (30 articles) vs. Afghan War (26 articles)
The common theme indicates that “United Nations” is involved in both wars
Cluster 1
Cluster 2
Cluster 3
Common
Theme
united
nations …
killed
month
deaths
Iraq n
Weapons
Inspections
troops
hoon
sanches
Afghan
Northern
alliance
kabul
taleban
aid
taleban
rumsfeld
hotel
front
Collection-specific themes indicate different roles of “United Nations” in the two wars
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

61. Comparing Laptop Reviews
Top words serve as “labels” for common themes
(e.g., [sound, speakers], [battery, hours], [cd,drive])
These word distributions can be used to segment text and
add hyperlinks between documents
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

62. Spatiotemporal Patterns in Blog Articles
Query= “Hurricane Katrina”
Topics in the results:
Spatiotemporal patterns
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

63. Theme Life Cycles for Hurricane Katrina
Oil Price
price oil gas increase product
fuel
company …
New Orleans
city orleans new louisiana flood
evacuate
storm …
The upper figure is the life cycles for different themes in Texas. The red line refers to a theme with the top probability words such as price, oil, gas, increase, etc, from which we know that it is talking about “oil price”. The blue one, on the other hand, talks about events that happened in the city “new orleans”. In the upper figure, we can see that both themes were getting hot during the first two weeks, and became weaker around the mid September. The theme New Orleans got strong again around the last week of September while the other theme dropped monotonically.
In the bottom figure, which is the life cycles for the same theme “New Orleans” in different states. We observe that this theme reaches
the highest probability first in Florida and Louisiana, followed by Washington and Texas, consecutively. During early September, this theme drops significantly in Louisiana while still strong in other states. We suppose this is because of the evacuation in Louisiana. Surprisingly, around late September, a re-arising pattern can be observed in most states, which is most significant in Louisiana. Since this is the time period in which Hurricane Rita arrived, we guess that Hurricane Rita has an impact on the discussion of Hurricane Katrina. This is reasonable since people
are likely to mention the two hurricanes together or make comparisons. We can find more clues to this hypothesis from Hurricane Rita data set.
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

64. Theme Snapshots for Hurricane Katrina
Week4: The theme is again strong along the east coast and the Gulf of Mexico
Week3: The theme distributes more uniformly over the states
Week2: The discussion moves towards the north and west
Week5: The theme fades out in most states
Week1: The theme is the strongest along the Gulf of Mexico
This slide shows the snapshot for theme ``Government Response'' over the first five weeks of Hurricane Katrina. The darker the color is, the hotter the discussion about this theme is. we observe that at the first week of Hurricane Katrina, the theme ``Government Response'‘ is the strongest in the southeast states, especially those along the Gulf of Mexico. In week 2, we can see the pattern that the theme is spreading towards the north and western states because the northern states are getting darker. In week 3, the theme is distributed even more uniformly, which means that it is spreading all over the states. However, in week 4, we observe that the theme converges to east states and southeast coast again. Interestingly, this week happens to overlap with the first week of Hurricane Rita, which may raise the public concern about government response again in those areas. In week 5, the theme becomes weak in most inland states and most of the remaining discussions are along the coasts.
Another interesting observation is that this theme is originally very strong in Louisiana (the one to the right of Texas, ), but dramatically weakened in Louisiana during week 2 and 3, and becomes strong again from the fourth week. Interestingly, Week 2 and 3 are consistent with the time of evacuation in Louisiana.
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

65. Theme Life Cycles: KDD
gene expressions probability microarray …
marketing customer
model business …
rules association
support …
Global Themes life cycles of KDD Abstracts
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

66. Theme Evolution Graph: KDD
1999
2000
2001
2002
2003
2004 T
web classifica –tion features0.006 topic …
SVM criteria classifica – tion linear …
mixture random cluster clustering
variables …
topic mixture LDA semantic …
decision tree classifier class Bayes … …
Classifica - tion text unlabeled document labeled learning …
Informa - tion web social retrieval distance networks 0.004 … … …
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

67. Blog Sentiment Summary (query=“Da Vinci Code”)
Neutral
Positive
Negative
Facet 1: Movie
... Ron Howards selection of Tom Hanks to play Robert Langdon.
Tom Hanks stars in the movie,who can be mad at that?
But the movie might get delayed, and even killed off if he loses.
Directed by: Ron Howard Writing credits: Akiva Goldsman ...
Tom Hanks, who is my favorite movie star act the leading role.
protesting ... will lose your faith by ... watching the movie.
After watching the movie I went online and some research on ...
Anybody is interested in it?
... so sick of people making such a big deal about a FICTION book and movie.
Facet 2: Book
I remembered when i first read the book, I finished the book in two days.
Awesome book.
I’m reading “Da Vinci Code” now. …
So still a good book to past time.
This controversy book cause lots conflict in west society.
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

68. Results: Sentiment Dynamics
Facet: the book “ the da vinci code”. ( Bursts during the movie, Pos > Neg )
Facet: religious beliefs ( Bursts during the movie, Neg > Pos )
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

69. Event Impact Analysis: IR Research
xml model collect
judgment
rank
subtopic …
vector concept extend model space
boolean
function
feedback …
Publication of the paper “A language modeling approach to information retrieval”
Starting of the TREC conferences
year
1992
term relevance weight feedback independence
model
frequent
probabilistic
document …
Theme: retrieval models
SIGIR papers
1998
model language estimate parameter distribution
probable
smooth
markov
likelihood …
probabilist model logic ir boolean
algebra
estimate
weight …
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

70. Temporal-Author-Topic Analysis
close pattern sequential
min_support
threshold
top-k
fp-tree …
index graph web
gspan substructure
gindex
bide
xml …
project itemset intertransaction
support associate
frequent
closet
prefixspan …
Author
Jiawei Han
Rakesh Agrawal
Author A
Global theme: frequent patterns
time
2000
Author B
pattern frequent frequent-pattern
sequential
pattern-growth
constraint
push …
research next transaction
panel technical
article
revolution
innovate …
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

71. Modeling Topical Communities (Mei et al. 08)
Community 1:
Information Retrieval
Community 2:
Data Mining
Community 3:
Machine Learning 71
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

72. Other Extensions (LDA Extensions)
Many extensions of LDA, mostly done by David Blei, Andrew McCallum and their co-authors
Some examples:
Hierarchical topic models [Blei et al. 03]
Modeling annotated data [Blei & Jordan 03]
Dynamic topic models [Blei & Lafferty 06]
Pachinko allocation [Li & McCallum 06])
Also, some specific context extension of PLSA, e.g., author-topic model [Steyvers et al. 04]
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

73. Future Research Directions
Topic models for text mining
Evaluation of topic models
Improve the efficiency of estimation and inferences
Incorporate linguistic knowledge
Applications in new domains and for new tasks
Text mining in general
Combination of NLP-style and DM-style mining algorithms
Integrated mining of text (unstructured) and unstructured data (e.g., Text OLAP)
Interactive mining:
Incorporate user constraints and support iterative mining
Design and implement mining languages
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

74. Lecture 5: Key Points
Topic models coupled with topic labeling are quite useful for extracting and modeling subtopics in text
Adding context variables significantly increases a topic model’s capacity of performing text mining
Enable interpretation of topics in context
Accommodate variation analysis and correlation analysis of topics over context
User’s preferences and domain knowledge can be added as prior or soft constraint
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

75. Readings PLSA: LDA: CPLSA:
LDA:
Many recent extensions, mostly done by David Blei and Andrew McCallums
CPLSA:
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

76. Discussion Topic models for mining multimedia data
Simultaneous modeling of text and images
Cross-media analysis
Text provides context to analyze images and vice versa
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

77. Looking forward to collaborations…
Course Summary
Integrated Multimedia Data Analysis
Mutual reinforcement (e.g., text images)
Simultaneous mining of text + images +video…
Scope of the course
Information Retrieval
Text Data
Retrieval models/framework
Evaluation
Feedback
Contextual topic models
Evaluation
User modeling
Ranking
Learning with little supervision
Multimedia Data
Computer Vision
Natural Language Processing
Machine Learning
Looking forward to collaborations…
Statistics
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008

78. Thank You!
2008 © ChengXiang Zhai China-US-France Summer School, Lotus Hill Inst., 2008