1. Lecture 5
Unsupervised Learning in fully Observed Directed and Undirected Graphical Models

2. Last Time
Last time we saw how to learn the parameters in a Maximum Likelihood setting for labeled data: fully observed supervised learning.
We looked at both the generative (Naive Bayes) and the discriminative (linear & logistic regression) models.
Now we turn to unsupervised learning of general Bayes nets and Markov Random field models.
Directed: very easy and intuitive.
Undirected: very easy for decomposable models.
hard for non-decomposable models: IPF.

3. Directed Models Parameterize by full probability tables.
The log-likelihood is a sum of independent tables if there are no hidden variables.
Use counts m(x) as the relevant statistics of the data.
Constraints enforced using Lagrange multipliers
ML estimates for parameters are simple the appropriate counts

4. Undirected GM
The normalization constant spoils the factorization property.
There is still an easy characterization in terms of counts, however this does not automatically provide the ML estimates of the potentials
When models are decomposable however, we can still write down the solution by inspection.
In all other cases we compute the solution iteratively by means of the “iterative proportional fitting procedure”.
IPF leaves Z invariant, and sets the new marginal on a clique equal to the empirical marginal.
IPF is coordinate ascent on Log-Likelihood!
IPF is guaranteed to converge.