0. Generative (Bayesian) modeling
04/04/2016

1. B. Freeman, Tomasz Malisiewicz, Tom Landauer and Peter Foltz,
Slides by (credit to):
David M. Blei
Andrew Y. Ng,
Michael I. Jordan,
Ido Abramovich,
L. Fei-Fei,
P. Perona, J. Sivic,
B. Russell,
A. Efros,
A. Zisserman,
B. Freeman, Tomasz Malisiewicz,
Thomas Huffman,
Tom Landauer and Peter Foltz,
Melanie Martin,
Hsuan-Sheng Chiu,
Haiyan Qiao,
Jonathan Huang
Thank you!

2. Generative modeling unsupervised learning … beyond clustering
How can we describe/model the world for the computer?
Bayesian networks!

3. Bayesian networks
Directed acyclic graphs (DAG) whose nodes represent random variables
Arcs represent (directed) dependence between random variables

4. Bayesian networks Filled nodes: observable variables
Empty nodes: hidden (not observable) variables Zi
wi1
w2i
w3i
w4i

5. Collapsed notation of Bayesian networks
Frames indicates multiplications
E.g. N features and M instances: Zi
wi1
w2i
w3i
w4i

6. Generative (Bayesian) modeling
Find the parameters of the given model which explains/reconstruct the observed data
Model
„Generative story”
DATA

7. Model = Bayesian network
„Generative story”
Model = Bayesian network
The structure of the network is given by the human engineer
The form of the nodes’ distribution (conditioned on their parents) is given as well
The parameters of the distributions have to be estimated from data

8. Parameter estimation in Bayesian network – only observable variables
Bayesian network assumes that the variables only (directly) dependent from their parents → parameter estimation at each node can be carried out separetly
Maximum Likelihood (or Bayesian estimation)

9. Expectation-Maximisation (EM)
The extension of Maximum Likelihood parameter estimation if hidden variables are present
We search for the parameter vector Φ which maximises the likelihood of the joint of observable variables X and hidden ones Z

10. Expectation-Maximisation (EM)
Iterative algorithm. Step l:
(E)xpectation step:
estimate the values of Z (calculate
expected values) using Φl
(M)aximization step:
Maximum likelihood estimetion by using Z

11. EM example
There are two coins. We drop them together but we can observe only the sum of the heads:
h(0)=4
h(1)=9
h(2)=2
What is the bias of the coins?
Φ1=P1(H), Φ2=P2(H) ?

12. EM example
a single z hidden variable: what is the proportion of the first coin out of h(1)=9 init Φ10=0.2 Φ20=0.5 E-step

13. EM example
M-step

14. Text classification/clustering
E.g. recognition of documents’ topic or clustering images based on their content
„Bag-of-words model”
The term-document matrix:

15. Image
Bag-of-”words”

16. The dictionary consists of M words
N documents:
D={d1, … ,dN}
The dictionary consists of M words
W={w 1 , … ,w M}
The size of the term-document matrix is N * M and it contains the number of occurances of a certain word in a certain document

17. Drawbacks of the bag-of-words model
Word order is ignored
Synonyms: We refer to a concept (object) by multiple words,
e.g: tired-sleepy
→ low recall
Polysemy: most of words have multiple senses, pl: bank, chips
→ low precision

19. Document clustering – unigram model
Let’s assign a „topic” to each document
The topics are hidden variables

20. Generative story of „unigram model”
How documents generate?
„Drop” a topic (and a size)
For each word position drop a word according to the topic’s distribution
TOPIC
Word
...
Word

21. Unigram model Each M documents, Drop a topic z.Zi
wi1
w2i
w3i
w4i
not clear what the plates are and what N and M are, and how you relate this to a naïve bayes model/
between this slide and the previous one, add a slide introducing the naïve bayes model.
add a slide explaining why naïve bayes is not good (you are picking a single class for each document).
you need to say more precisely what the plate means, i.e., parameter sharing in the cpts
Each M documents,
Drop a topic z.
Drop a word (independently from the others) from a multinomial distribution conditiond on z

22. EM for clustering
E-step
M-step

23. pLSA The distributions found are interpretable
Probabilistic Latent Semantic Analysis
We assign a distribution of topics to each of the document
Topics still have a distrbution over words
The distributions found are interpretable

24. Relation to clustering…
A document can belong to multiple clusters
We’re interested in the distribution of topics rather than pushing each doc into a cluster
→ more flexible

25. Generative story of pLSA
How documents generate?
Generate the document’s topic distribution
For each word position drop a topic from the doc’s topic distribution
drop a word according to the topic’s distribution
TOPIC distribution
...
TOPIC
TOPIC
word
...
word

27. Example
money
money
loan
bank
DOCUMENT 1: money1 bank1 bank1 loan1 river2 stream2 bank1 money1 river2 bank1 money1 bank1 loan1 money1 stream2 bank1 money1 bank1 bank1 loan1 river2 stream2 bank1 money1 river2 bank1 money1 bank1 loan1 bank1 money1 stream2 .8
loan
bank
bank
loan .2
TOPIC 1 .3
DOCUMENT 2: river2 stream2 bank2 stream2 bank2 money1 loan1 river2 stream2 loan1 bank2 river2 bank2 bank1 stream2 river2 loan1 bank2 stream2 bank2 money1 loan1 river2 stream2 bank2 stream2 bank2 money1 river2 stream2 loan1 bank2 river2 bank2 money1 bank1 stream2 river2 bank2 stream2 bank2 money1
river
bank
river .7
stream
river
bank
stream
TOPIC 2

28. Parameter estimation (model fitting, training)?
DOCUMENT 1: money? bank? bank? loan? river? stream? bank? money? river? bank? money? bank? loan? money? stream? bank? money? bank? bank? loan? river? stream? bank? money? river? bank? money? bank? loan? bank? money? stream?
TOPIC 1 ?
DOCUMENT 2: river? stream? bank? stream? bank? money? loan? river? stream? loan? bank? river? bank? bank? stream? river? loan? bank? stream? bank? money? loan? river? stream? bank? stream? bank? money? river? stream? loan? bank? river? bank? money? bank? stream? river? bank? stream? bank? money? ?
TOPIC 2

29. pLSA Observable data Term distributions Topic distributions
over topics
Topic distributions
For documents
For that we will use a method called probabilistic latent semantic analysis.
pLSA can be thought of as a matrix decomposition.
Here is our term-document matrix (documents, words) and we want to
find topic vectors common to all documents and mixture coefficients P(z|d) specific to each document.
Note that these are all probabilites which sum to one.
So a column here is here expressed as a convex combination of the topic vectors.
What we would like to see is that the topics correspond to objects.
So an image will be expressed as a mixture of different objects and backgrounds.
Slide credit: Josef Sivic

30. Latent semantic analysis (LSA)

31. Generative story of pLSA
How documents generate?
Generate the document’s topic distribution
For each word position drop a topic from the doc’s topic distribution
drop a word according to the topic’s distribution
TOPIC distribution
...
TOPIC
TOPIC
word
...
word

32. pLSA modell For each document d and word position:
For each word position drop a topic from the doc’s topic distribution
drop a word according to the topic’s distribution d
zd1
zd2
zd3
zd4
between this slide and the previous one, add a motivating example where you have a latent variable in naïve bayse
show an unrolled bn for this model using your running example (text classification)
wd1
wd2
wd3
wd4

33. pLSA for images
(example) w N d z D
“eye”
Sivic et al. ICCV 2005

34. pLSA – parameter estimation

35. pLSA – E-step
What is the expected value of hidden variables (topics z) if the parameter values are fixed

36. pLSA – M-step
We use the values of hidden hidden variables p(z|d,w)

37. EM algorithm
It can converge to local optimum
Stopping condition?

38. Approximate inference
The E-step in huge networks is unfeasible
There are approaches which are fast but do only approximate inference (=E-step) rather than exact one
The most popular approximate inferece method is:
Drop samples according to Bayesian network
The average of the samples can be used as expected values of hidden variables

39. Markov Chain Monte Carlo method (MCMC)
MCMC is an approximate inference method
The samples are not independent from each other but they are generated one by one based on the previous sample (form a chain)
Gibbs sampling is the most famous MCMC method:
The next sample is generated by fixing all but variables and drop a value for the non-fixed one conditioned on the other ones

40. Outlook

41. Drawbacks of pLSA
It can be recalculated from scratch if a new document arrives
The number of parameters increases as the function of the number of instances
d is just an index, it doesn’t fit well into the generative story

42. 1990
1999
2003

43. 2010

44. An even more complex task
Recognise objects inside images without any supervision
After training the model can be applied for unknown images as well

45. Summary
Generative (Bayesian) modeling enables to define any description/model of the world with any complexity
(clustering is only an example for it)
EM algorithm is a general tool for solving parameter estimation problems where latent variables is incorporated