Presentation is loading. Please wait.

Presentation is loading. Please wait.

Learning with Missing Data Eran Segal Weizmann Institute.

Similar presentations

Presentation on theme: "Learning with Missing Data Eran Segal Weizmann Institute."— Presentation transcript:

1 Learning with Missing Data Eran Segal Weizmann Institute

2 Incomplete Data Hidden variables Missing values Challenges Foundational – is the learning task well defined? Computational – how can we learn with missing data?

3 Treating Missing Data How Should we treat missing data? Case I: A coin is tossed on a table, occasionally it drops and measurements are not taken Sample sequence: H,T,?,?,T,?,H Treat missing data by ignoring it Case II: A coin is tossed, but only heads are reported Sample sequence: H,?,?,?,H,?,H Treat missing data by filling it with Tails We need to consider the data missing mechanism

4 Modeling Data Missing Mechanism X = {X 1,...,X n } are random variables O X = {O X 1,...,O X n } are observability variables Always observed Y = {Y 1,...,Y n } new random variables Val(Y i ) = Val(Y i )  {?} Y i is a deterministic function of X i and O X 1 :

5 Modeling Missing Data Mechanism  X  OXOX Case I (random missing values)  X  OXOX Case II (deliberate missing values) Y Y

6 Modeling Missing Data Mechanism  X  OXOX Case I (random missing values)  X  OXOX Case II (deliberate missing values) Y Y

7 Treating Missing Data When can we ignore the missing data mechanism and focus only on the likelihood? For every X i, Ind(X i ;O X i ) Missing at Random (MAR) is sufficient The probability that the value of X i is missing is independent of its actual value given other observed values In both cases, the likelihood decomposes

8 Hidden (Latent) Variables Attempt to learn a model with hidden variables In this case, MAR always holds (variable is always missing) Why should we care about unobserved variables? X1X1 X2X2 X3X3 H Y1Y1 Y2Y2 Y3Y3 X1X1 X2X2 X3X3 Y1Y1 Y2Y2 Y3Y3 17 parameters 59 parameters

9 Hidden (Latent) Variables Hidden variables also appear in clustering Naïve Bayes model: Class variable is hidden Observed attributes are independent given the class Cluster X1X1... X2X2 XnXn Hidden Observed possible missing values

10 Likelihood for Complete Data P(Y|X) Xy0y0 y1y1 x0x0  y 0 |x 0  y 1 |x 0 x1x1  y 0 |x 1  y 1 |x 1 X Y XY x0x0 y0y0 x0x0 y1y1 x1x1 y0y0 P(X) x0x0 x1x1 x0x0  x1 Input Data: Likelihood:  Likelihood decomposes by variables  Likelihood decomposes within CPDs

11 Likelihood for Incomplete Data P(Y|X) Xy0y0 y1y1 x0x0  y 0 |x 0  y 1 |x 0 x1x1  y 0 |x 1  y 1 |x 1 X Y XY ?y0y0 x0x0 y1y1 ?y0y0 P(X) x0x0 x1x1 x0x0  x1 Input Data: Likelihood:  Likelihood does not decompose by variables  Likelihood does not decompose within CPDs  Computing likelihood per instance requires inference!

12 Bayesian Estimation XX X[1] X[M]X[2] … X Y Bayesian network Bayesian network for parameter estimation Y[1] Y[M]Y[2] …  Y|X=0  Y|X=1 Posteriors are not independent

13 Identifiability Likelihood can have multiple global maxima Example: We can rename the values of the hidden variable H If H has two values, likelihood has two global maxima With many hidden variables, there can be an exponential number of global maxima Multiple local and global maxima can also occur with missing data (not only hidden variables) H Y

14 Gradient Ascent:  Follow gradient of likelihood w.r.t. to parameters  Add line search and conjugate gradient methods to get fast convergence L(D|  ) MLE from Incomplete Data Nonlinear optimization problem 

15 Expectation Maximization (EM):  Use “current point” to construct alternative function (which is “nice”)  Guaranty: maximum of new function has better score than current point MLE from Incomplete Data Nonlinear optimization problem  L(D|  )

16 Gradient Ascent and EM  Find local maxima  Require multiple restarts to find approx. to the global maximum  Require computations in each iteration MLE from Incomplete Data Nonlinear optimization problem  L(D|  )

17 Gradient Ascent Theorem: Proof: How do we compute ?

18 Gradient Ascent

19 Requires computation: P(x i,pa i |o[m],  ) for all i, m Can be done with clique-tree algorithm, since X i,Pa i are in the same clique

20 Gradient Ascent Summary Pros Flexible, can be extended to non table CPDs Cons Need to project gradient onto space of legal parameters For reasonable convergence, need to combine with advanced methods (conjugate gradient, line search)

21 Expectation Maximization (EM) Tailored algorithm for optimizing likelihood functions Intuition Parameter estimation is easy given complete data Computing probability of missing data is “easy” (=inference) given parameters Strategy Pick a starting point for parameters “Complete” the data using current parameters Estimate parameters relative to data completion Iterate Procedure guaranteed to improve at each iteration

22 Expectation Maximization (EM) Initialize parameters to  0 Expectation (E-step): For each data case o[m] and each family X,U compute P(X,U | o[m],  i ) Compute the expected sufficient statistics for each x,u Maximization (M-step): Treat the expected sufficient statistics as observed and set the parameters to the MLE with respect to the ESS

23 Expectation Maximization (EM) X Y XY ?y0y0 x0x0 y1y1 ?y0y0 Training data Initial network + E-Step (inference) Expected counts N(X) N(X,Y) M-Step (reparameterize) X Y Updated network Iterate

24 Expectation Maximization (EM) Formal Guarantees: L(D:  i+1 )  L(D:  i ) Each iteration improves the likelihood If  i+1 =  i, then  i is a stationary point of L(D:  ) Usually, this means a local maximum Main cost: Computations of expected counts in E-Step Requires inference for each instance in training set Exactly the same as in gradient ascent!

25 EM – Practical Considerations Initial parameters Highly sensitive to starting parameters Choose randomly Choose by guessing from another source Stopping criteria Small change in data likelihood Small change in parameters Avoiding bad local maxima Multiple restarts Early pruning of unpromising starting points


27 EM in Practice – Alarm Network Training errorTest error

28 EM in Practice – Alarm Network

29 Partial Data: Parameter Estimation Non-linear optimization problem Methods for learning: EM and Gradient Ascent Exploit inference for learning Challenges Exploration of a complex likelihood/posterior More missing data  many more local maxima Cannot represent posterior  must resort to approximations Inference Main computational bottleneck for learning Learning large networks  exact inference is infeasible  resort to approximate inference

30 Structure Learning w. Missing Data Distinguish two learning problems Learning structure for a given set of random variables Introduce new hidden variables How do we recognize the need for a new variable? Where do we introduce a newly added hidden variable within G? Open ended and less understood…

31 Structure Learning w. Missing Data Theoretically, there is no problem Define score, and search for structure that maximizes it Likelihood term will require gradient ascent or EM Practically infeasible Typically we have O(n 2 ) candidates at each search step Requires EM for evaluating each candidate Requires inference for each data instance of each candidate Total running time per search step: O(n 2  M  #EM iteration  cost of BN inference)

32 Reverse C  B Delete B  C Add B  D Typical Search B C A D B C A D B C A D B C A D Requires EM

33 Structural EM Basic idea: use expected sufficient statistics to learn structure, not just parameters Use current network to complete the data using EM Treat the completed data as “real” to score candidates Pick the candidate network with the best score Use the previous completed counts to evaluate networks in the next step After several steps, compute a new data completion from the current network

34 Structural EM Conceptually Algorithm maintains an actual distribution Q over completed datasets as well as current structure G and parameters  G At each step we do one of the following Use to compute a new completion Q and redefine  G as the MLE relative to Q Evaluate candidate successors G’ relative to Q and pick best In practice Maintain Q implicitly as a model Use the model to compute sufficient statistics M Q [x,u] when these are needed to evaluate new structures Use sufficient statistics to compute MLE estimates of candidate structures

35 Structural EM Benefits Many fewer EM runs Score relative to completed data is decomposable! Utilize same benefits as structure learning w. complete data Each candidate network requires few recomputations Here savings is large since each sufficient statistics computation requires inference As in EM, we optimize a simpler score Can show improvements and convergence An SEM step that improves in D + space, improves real score

Download ppt "Learning with Missing Data Eran Segal Weizmann Institute."

Similar presentations

Ads by Google