Learning with Missing Data Eran Segal Weizmann Institute

Incomplete Data Hidden variables Missing values Challenges Foundational – is the learning task well defined? Computational – how can we learn with missing data?

We need to consider the data missing mechanism Treating Missing Data How Should we treat missing data? Case I: A coin is tossed on a table, occasionally it drops and measurements are not taken Sample sequence: H,T,?,?,T,?,H Treat missing data by ignoring it Case II: A coin is tossed, but only heads are reported Sample sequence: H,?,?,?,H,?,H Treat missing data by filling it with Tails We need to consider the data missing mechanism

Modeling Data Missing Mechanism X = {X1,...,Xn} are random variables OX = {OX1,...,OXn} are observability variables Always observed Y = {Y1,...,Yn} new random variables Val(Yi) = Val(Yi)  {?} Yi is a deterministic function of Xi and OX1:

Modeling Missing Data Mechanism Case I (random missing values) Case II (deliberate missing values)     X X OX OX Y Y

Modeling Missing Data Mechanism Case I (random missing values) Case II (deliberate missing values)     X X OX OX Y Y

Treating Missing Data When can we ignore the missing data mechanism and focus only on the likelihood? For every Xi, Ind(Xi;OXi) Missing at Random (MAR) is sufficient The probability that the value of Xi is missing is independent of its actual value given other observed values In both cases, the likelihood decomposes

Hidden (Latent) Variables Attempt to learn a model with hidden variables In this case, MAR always holds (variable is always missing) Why should we care about unobserved variables? X1 X2 X3 X1 X2 X3 H Y1 Y2 Y3 Y1 Y2 Y3 17 parameters 59 parameters

Hidden (Latent) Variables Hidden variables also appear in clustering Naïve Bayes model: Class variable is hidden Observed attributes are independent given the class Hidden Cluster X1 ... X2 Xn Observed possible missing values

Likelihood for Complete Data P(X) x0 x1 x0 x1 Input Data: X Y x0 y0 y1 x1 X Likelihood: Y P(Y|X) X y0 y1 x0 y0|x0 y1|x0 x1 y0|x1 y1|x1 Likelihood decomposes by variables Likelihood decomposes within CPDs

Likelihood for Incomplete Data P(X) x0 x1 x0 x1 Input Data: X Y ? y0 x0 y1 X Likelihood: Y P(Y|X) X y0 y1 x0 y0|x0 y1|x0 x1 y0|x1 y1|x1 Likelihood does not decompose by variables Likelihood does not decompose within CPDs Computing likelihood per instance requires inference!

Bayesian Estimation … … Bayesian network for parameter estimation X X … X[1] X[2] X[M] … Y Y[1] Y[2] Y[M] Y|X=0 Y|X=1 Posteriors are not independent

Identifiability Likelihood can have multiple global maxima Example: We can rename the values of the hidden variable H If H has two values, likelihood has two global maxima With many hidden variables, there can be an exponential number of global maxima Multiple local and global maxima can also occur with missing data (not only hidden variables) Y

MLE from Incomplete Data Nonlinear optimization problem Gradient Ascent: Follow gradient of likelihood w.r.t. to parameters Add line search and conjugate gradient methods to get fast convergence L(D|) 

MLE from Incomplete Data Nonlinear optimization problem Expectation Maximization (EM): Use “current point” to construct alternative function (which is “nice”) Guaranty: maximum of new function has better score than current point L(D|) 

MLE from Incomplete Data Nonlinear optimization problem L(D|)  Gradient Ascent and EM Find local maxima Require multiple restarts to find approx. to the global maximum Require computations in each iteration

Gradient Ascent Theorem: Proof: How do we compute ?

Gradient Ascent

Gradient Ascent Requires computation: P(xi,pai|o[m],) for all i, m Can be done with clique-tree algorithm, since Xi,Pai are in the same clique

Gradient Ascent Summary Pros Flexible, can be extended to non table CPDs Cons Need to project gradient onto space of legal parameters For reasonable convergence, need to combine with advanced methods (conjugate gradient, line search)

Expectation Maximization (EM) Tailored algorithm for optimizing likelihood functions Intuition Parameter estimation is easy given complete data Computing probability of missing data is “easy” (=inference) given parameters Strategy Pick a starting point for parameters “Complete” the data using current parameters Estimate parameters relative to data completion Iterate Procedure guaranteed to improve at each iteration

Expectation Maximization (EM) Initialize parameters to 0 Expectation (E-step): For each data case o[m] and each family X,U compute P(X,U | o[m], i) Compute the expected sufficient statistics for each x,u Maximization (M-step): Treat the expected sufficient statistics as observed and set the parameters to the MLE with respect to the ESS

Expectation Maximization (EM) Initial network M-Step (reparameterize) X Y Updated network X Expected counts N(X) N(X,Y) Y E-Step (inference) + Training data Iterate X Y ? y0 x0 y1

Expectation Maximization (EM) Formal Guarantees: L(D:i+1)  L(D:i) Each iteration improves the likelihood If i+1=i , then i is a stationary point of L(D:) Usually, this means a local maximum Main cost: Computations of expected counts in E-Step Requires inference for each instance in training set Exactly the same as in gradient ascent!

EM – Practical Considerations Initial parameters Highly sensitive to starting parameters Choose randomly Choose by guessing from another source Stopping criteria Small change in data likelihood Small change in parameters Avoiding bad local maxima Multiple restarts Early pruning of unpromising starting points

EM in Practice – Alarm Network Data sampled from true network 20% of data randomly deleted PCWP CO HRBP HREKG HRSAT ERRCAUTER HR HISTORY CATECHOL SAO2 EXPCO2 ARTCO2 VENTALV VENTLUNG VENITUBE DISCONNECT MINVOLSET VENTMACH KINKEDTUBE INTUBATION PULMEMBOLUS PAP SHUNT ANAPHYLAXIS MINOVL PVSAT FIO2 PRESS INSUFFANESTH TPR LVFAILURE ERRBLOWOUTPUT STROEVOLUME LVEDVOLUME HYPOVOLEMIA CVP BP

EM in Practice – Alarm Network Training error Test error

EM in Practice – Alarm Network

Partial Data: Parameter Estimation Non-linear optimization problem Methods for learning: EM and Gradient Ascent Exploit inference for learning Challenges Exploration of a complex likelihood/posterior More missing data  many more local maxima Cannot represent posterior  must resort to approximations Inference Main computational bottleneck for learning Learning large networks  exact inference is infeasible  resort to approximate inference

Structure Learning w. Missing Data Distinguish two learning problems Learning structure for a given set of random variables Introduce new hidden variables How do we recognize the need for a new variable? Where do we introduce a newly added hidden variable within G? Open ended and less understood…

Structure Learning w. Missing Data Theoretically, there is no problem Define score, and search for structure that maximizes it Likelihood term will require gradient ascent or EM Practically infeasible Typically we have O(n2) candidates at each search step Requires EM for evaluating each candidate Requires inference for each data instance of each candidate Total running time per search step: O(n2  M #EM iteration  cost of BN inference)

Typical Search Requires EM A B C A B D C Requires EM D A B A B C C D D Add B D A B D C Requires EM Reverse CB Delete BC D A B A B C C D D Requires EM

Structural EM Basic idea: use expected sufficient statistics to learn structure, not just parameters Use current network to complete the data using EM Treat the completed data as “real” to score candidates Pick the candidate network with the best score Use the previous completed counts to evaluate networks in the next step After several steps, compute a new data completion from the current network

Structural EM Conceptually In practice Algorithm maintains an actual distribution Q over completed datasets as well as current structure G and parameters G At each step we do one of the following Use <G,G> to compute a new completion Q and redefine G as the MLE relative to Q Evaluate candidate successors G’ relative to Q and pick best In practice Maintain Q implicitly as a model <G,G> Use the model to compute sufficient statistics MQ[x,u] when these are needed to evaluate new structures Use sufficient statistics to compute MLE estimates of candidate structures

Structural EM Benefits Many fewer EM runs Score relative to completed data is decomposable! Utilize same benefits as structure learning w. complete data Each candidate network requires few recomputations Here savings is large since each sufficient statistics computation requires inference As in EM, we optimize a simpler score Can show improvements and convergence An SEM step that improves in D+ space, improves real score