1. Learning with Missing Data
Eran Segal
Weizmann Institute

2. Incomplete Data Hidden variables Missing values Challenges
Foundational – is the learning task well defined?
Computational – how can we learn with missing data?

3. We need to consider the data missing mechanism
Treating Missing Data
How Should we treat missing data?
Case I: A coin is tossed on a table, occasionally it drops and measurements are not taken
Sample sequence: H,T,?,?,T,?,H
Treat missing data by ignoring it
Case II: A coin is tossed, but only heads are reported
Sample sequence: H,?,?,?,H,?,H
Treat missing data by filling it with Tails
We need to consider the data missing mechanism

4. Modeling Data Missing Mechanism
X = {X1,...,Xn} are random variables
OX = {OX1,...,OXn} are observability variables
Always observed
Y = {Y1,...,Yn} new random variables
Val(Yi) = Val(Yi)  {?}
Yi is a deterministic function of Xi and OX1:

5. Modeling Missing Data Mechanism
Case I (random missing values)
Case II (deliberate missing values)     X X OX OX Y Y

6. Modeling Missing Data Mechanism
Case I (random missing values)
Case II (deliberate missing values)     X X OX OX Y Y

7. Treating Missing Data
When can we ignore the missing data mechanism and focus only on the likelihood?
For every Xi, Ind(Xi;OXi)
Missing at Random (MAR) is sufficient
The probability that the value of Xi is missing is independent of its actual value given other observed values
In both cases, the likelihood decomposes

8. Hidden (Latent) Variables
Attempt to learn a model with hidden variables
In this case, MAR always holds (variable is always missing)
Why should we care about unobserved variables? X1 X2 X3 X1 X2 X3 H Y1 Y2 Y3 Y1 Y2 Y3
17 parameters
59 parameters

9. Hidden (Latent) Variables
Hidden variables also appear in clustering
Naïve Bayes model:
Class variable is hidden
Observed attributes are independent given the class
Hidden
Cluster X1
... X2 Xn
Observed
possible missing values

10. Likelihood for Complete Data
P(X) x0 x1
x0
x1
Input Data: X Y x0 y0 y1 x1 X
Likelihood: Y
P(Y|X) X y0 y1 x0
y0|x0
y1|x0 x1
y0|x1
y1|x1
Likelihood decomposes by variables
Likelihood decomposes within CPDs

11. Likelihood for Incomplete Data
P(X) x0 x1
x0
x1
Input Data: X Y ? y0 x0 y1 X
Likelihood: Y
P(Y|X) X y0 y1 x0
y0|x0
y1|x0 x1
y0|x1
y1|x1
Likelihood does not decompose by variables
Likelihood does not decompose within CPDs
Computing likelihood per instance requires inference!

12. Bayesian Estimation … … Bayesian network for parameter estimationX X …
X[1]
X[2]
X[M] … Y
Y[1]
Y[2]
Y[M]
Y|X=0
Y|X=1
Posteriors are not independent

13. Identifiability Likelihood can have multiple global maxima Example:
We can rename the values of the hidden variable H
If H has two values, likelihood has two global maxima
With many hidden variables, there can be an exponential number of global maxima
Multiple local and global maxima can also occur with missing data (not only hidden variables) Y

14. MLE from Incomplete Data
Nonlinear optimization problem
Gradient Ascent:
Follow gradient of likelihood w.r.t. to parameters
Add line search and conjugate gradient methods to get fast convergence
L(D|) 

15. MLE from Incomplete Data
Nonlinear optimization problem
Expectation Maximization (EM):
Use “current point” to construct alternative function (which is “nice”)
Guaranty: maximum of new function has better score than current point
L(D|) 

16. MLE from Incomplete Data
Nonlinear optimization problem
L(D|) 
Gradient Ascent and EM
Find local maxima
Require multiple restarts to find approx. to the global maximum
Require computations in each iteration

17. Gradient Ascent
Theorem:
Proof:
How do we compute ?

18. Gradient Ascent

19. Gradient Ascent Requires computation: P(xi,pai|o[m],) for all i, m
Can be done with clique-tree algorithm, since Xi,Pai are in the same clique

20. Gradient Ascent Summary
Pros
Flexible, can be extended to non table CPDs
Cons
Need to project gradient onto space of legal parameters
For reasonable convergence, need to combine with advanced methods (conjugate gradient, line search)

21. Expectation Maximization (EM)
Tailored algorithm for optimizing likelihood functions
Intuition
Parameter estimation is easy given complete data
Computing probability of missing data is “easy” (=inference) given parameters
Strategy
Pick a starting point for parameters
“Complete” the data using current parameters
Estimate parameters relative to data completion
Iterate
Procedure guaranteed to improve at each iteration

22. Expectation Maximization (EM)
Initialize parameters to 0
Expectation (E-step):
For each data case o[m] and each family X,U compute P(X,U | o[m], i)
Compute the expected sufficient statistics for each x,u
Maximization (M-step):
Treat the expected sufficient statistics as observed and set the parameters to the MLE with respect to the ESS

23. Expectation Maximization (EM)
Initial network
M-Step (reparameterize) X Y
Updated network X
Expected counts
N(X)
N(X,Y) Y
E-Step (inference) +
Training data
Iterate X Y ? y0 x0 y1

24. Expectation Maximization (EM)
Formal Guarantees:
L(D:i+1)  L(D:i)
Each iteration improves the likelihood
If i+1=i , then i is a stationary point of L(D:)
Usually, this means a local maximum
Main cost:
Computations of expected counts in E-Step
Requires inference for each instance in training set
Exactly the same as in gradient ascent!

25. EM – Practical Considerations
Initial parameters
Highly sensitive to starting parameters
Choose randomly
Choose by guessing from another source
Stopping criteria
Small change in data likelihood
Small change in parameters
Avoiding bad local maxima
Multiple restarts
Early pruning of unpromising starting points

26. EM in Practice – Alarm Network
Data sampled from true network
20% of data randomly deleted
PCWP CO
HRBP
HREKG
HRSAT
ERRCAUTER HR
HISTORY
CATECHOL
SAO2
EXPCO2
ARTCO2
VENTALV
VENTLUNG
VENITUBE
DISCONNECT
MINVOLSET
VENTMACH
KINKEDTUBE
INTUBATION
PULMEMBOLUS
PAP
SHUNT
ANAPHYLAXIS
MINOVL
PVSAT
FIO2
PRESS
INSUFFANESTH
TPR
LVFAILURE
ERRBLOWOUTPUT
STROEVOLUME
LVEDVOLUME
HYPOVOLEMIA
CVP BP

27. EM in Practice – Alarm Network
Training error
Test error

28. EM in Practice – Alarm Network

29. Partial Data: Parameter Estimation
Non-linear optimization problem
Methods for learning: EM and Gradient Ascent
Exploit inference for learning
Challenges
Exploration of a complex likelihood/posterior
More missing data  many more local maxima
Cannot represent posterior  must resort to approximations
Inference
Main computational bottleneck for learning
Learning large networks  exact inference is infeasible  resort to approximate inference

30. Structure Learning w. Missing Data
Distinguish two learning problems
Learning structure for a given set of random variables
Introduce new hidden variables
How do we recognize the need for a new variable?
Where do we introduce a newly added hidden variable within G?
Open ended and less understood…

31. Structure Learning w. Missing Data
Theoretically, there is no problem
Define score, and search for structure that maximizes it
Likelihood term will require gradient ascent or EM
Practically infeasible
Typically we have O(n2) candidates at each search step
Requires EM for evaluating each candidate
Requires inference for each data instance of each candidate
Total running time per search step: O(n2  M #EM iteration  cost of BN inference)

32. Typical Search Requires EM A B C A B D C Requires EM D A B A B C C D D
Add B D A B D C
Requires EM
Reverse CB
Delete BC D A B A B C C D D
Requires EM

33. Structural EM
Basic idea: use expected sufficient statistics to learn structure, not just parameters
Use current network to complete the data using EM
Treat the completed data as “real” to score candidates
Pick the candidate network with the best score
Use the previous completed counts to evaluate networks in the next step
After several steps, compute a new data completion from the current network

34. Structural EM Conceptually In practice
Algorithm maintains an actual distribution Q over completed datasets as well as current structure G and parameters G
At each step we do one of the following
Use <G,G> to compute a new completion Q and redefine G as the MLE relative to Q
Evaluate candidate successors G’ relative to Q and pick best
In practice
Maintain Q implicitly as a model <G,G>
Use the model to compute sufficient statistics MQ[x,u] when these are needed to evaluate new structures
Use sufficient statistics to compute MLE estimates of candidate structures

35. Structural EM Benefits
Many fewer EM runs
Score relative to completed data is decomposable!
Utilize same benefits as structure learning w. complete data
Each candidate network requires few recomputations
Here savings is large since each sufficient statistics computation requires inference
As in EM, we optimize a simpler score
Can show improvements and convergence
An SEM step that improves in D+ space, improves real score