White Parts from: Technical overview for machine-learning researcher – slides from UAI 1999 tutorial

Important Distinctions in Learning BNs Complete data versus incomplete data Observed variables versus hidden variables Learning parameters versus learning structure Scoring methods versus conditional independence tests methods Exact scores versus asymptotic scores Search strategies versus Optimal learning of trees/polytrees/TANs

Overview of today’s lecture Introduction to Bayesian statistics Learning parameters of Bayesian networks The lecture today assumes: complete data, observable variables, no hidden variables.

P(hhth…ttth)= #h(1- )#t = A(1- )B Maximum Likelihood approach: maximize probability of data with respect to the unknown parameter(s). Likelihood function: P(hhth…ttth)= #h(1- )#t = A(1- )B Equivalent method is to maximize the log-likelihood function. The maximum for such sampling data is obtained when  = A/ (A + B)

What if #heads is zero ? The maximum for such sampling data is obtained when  = A / (A+B) = 0 A “regularized maximum” likelihood solution is needed When the data is “small”. = A+ε / (A+B+2ε) where ε is some small number. What is the “meaning” of ε aside of a needed correction?

What if we have some idea a priori? If our a priori idea is equivalent to seeing a heads, and b tails and now we see data of A heads and B tails, then what is a natural estimate for  ? If we use the estimate  = a/( a+b ) as a prior guess, then after seeing the data our natural estimate changes to: = ( A+a )/(N+N’) where N = a+b, N’=A+B. So what is a good choice for {a, b} ? For a random coin, maybe (100,100)? For a random thumbtack maybe (7,3)? a and b are imaginary counts, N= a+b is the equivalent sample size while A, B are the data counts and A+B is the data size. What is the meaning of ε in the previous slide ?

 =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’)) Generalizing to a Dice with 6 sides If our a priori idea is equivalent to seeing a1,…,a6 counts from each state and we now see data A1 …A6 counts from each state, then what is a natural estimate for  ? If we use a point estimate  = (a1/N,…,an/N) then after seeing the data our point estimate changes to  =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’)) Note that posterior estimate can be updated after each data point (namely in sequel or “online”) and that the posterior can serve as prior for future data points.

Another view of the update We just saw that from  = (a1/N,…,an/N) we move on after seeing the data to i = ( ai+Ai)/(N+N’) for each i. This update can be viewed as a weighted mixture of prior knowledge with data estimates: i = N/(N+N’) (ai/N) + N’ /(N+N’) (Ai/N’)

The Full Bayesian Approach Use a priori knowledge about the parameters. Encode uncertainty about the parameters via a distribution. Update the distribution after data is seen. Choose expected value as estimate.

p(|data) =  p() p(data|) (As before) p(|data) =  p() p(data|)

p(|data) = p(| hhth …ttth) =  p() p( hhth …ttth|) If we had more 100 heads the peak would move much more to the right. If we had 50 heads and 50 tails the peak would just sharpen considerably. p(|data) = p(| hhth …ttth) =  p() p( hhth …ttth|) p(| data) =  p() #h (1-) #t = p(| #h, #t) (#h, #t) are sufficient statistics for binomial sampling

The Beta distribution as a prior for =f







The Beta prior distribution for  Example: ht … htthh

From Prior to Posterior Observation 1: If the prior is Beta(;a,b) and we have seen A heads and B tails, then the posterior is Beta(;A+a,B+b). Consequence: If the prior is Beta(;a,b) and we use a point estimate  = a/N, then after seeing the data our point estimate changes to = (A+a)/(N+N’) where N’=A+B. So what is a good choice for the hyper-parameters {a, b} ? For a random coin, maybe (100,100)? For a random thumbtack maybe (7,3)? a and b are imaginary counts, N=a+b is the equivalent sample size while A, B are the data counts and A+B is the data size.

From Prior to Posterior in Blocks Observation 2: If the prior is Dir(a1,…,an) and we have seen A1 …An counts from each state, then the posterior is Dir(a1+A1,…, an+An). Consequence: If the prior is Dir(a1,…,an) and we use a point estimate  = (a1/N,…,an/N) then after seeing the data our point estimate changes to  =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’)) Note that posterior distribution can be updated after each data point (namely in sequel or “online”) and that the posterior can serve as prior for the future data points.

Another view of the update Recall the Consequence that from  = (a1/N,…,an/N) we move on after seeing the data to i = ( ai+Ai)/(N+N’) for each i. This update can be viewed as mixture of prior and data estimates: i = N/(N+N’) (ai/N) + N’(N+N’) (Ai/N’)

Learning Bayes Net parameters with complete data – no hidden variables

Learning Bayes Net parameters p(), p(h| ) =  and p(t| ) = 1- 

Learning Bayes Net parameters

P(X=x | x) = x P(Y=y |X=x, y|x, y|~x)= y|x P(Y=y |X=~x, y|x, y|~x)= y|~x Global and Local parameter independence  three separate independent thumbtack estimation tasks, assuming complete data.