1. White Parts from: Technical overview for machine-learning researcher – slides from UAI 1999 tutorial

2. Important Distinctions in Learning BNs
Complete data versus incomplete data
Observed variables versus hidden variables
Learning parameters versus learning structure
Scoring methods versus conditional independence tests methods
Exact scores versus asymptotic scores
Search strategies versus Optimal learning of trees/polytrees/TANs

3. Overview of today’s lecture
Introduction to Bayesian statistics
Learning parameters of Bayesian networks
The lecture today assumes: complete data, observable variables, no hidden variables.

4. P(hhth…ttth)= #h(1- )#t = A(1- )B
Maximum Likelihood approach: maximize probability of data with respect to the unknown parameter(s). Likelihood function:
P(hhth…ttth)= #h(1- )#t = A(1- )B
Equivalent method is to maximize the log-likelihood function.
The maximum for such sampling data is obtained when
 = A/ (A + B)

5. What if #heads is zero ?
The maximum for such sampling data is obtained when
 = A / (A+B) = 0
A “regularized maximum” likelihood solution is needed
When the data is “small”.
= A+ε / (A+B+2ε)
where ε is some small number.
What is the “meaning” of ε aside of a needed correction?

6. What if we have some idea a priori?
If our a priori idea is equivalent to seeing a heads, and b tails and now we see data of A heads and B tails, then what is a natural estimate for  ?
If we use the estimate  = a/( a+b ) as a prior guess, then after seeing the data our natural estimate changes to:
= ( A+a )/(N+N’) where N = a+b, N’=A+B.
So what is a good choice for {a, b} ?
For a random coin, maybe (100,100)?
For a random thumbtack maybe (7,3)?
a and b are imaginary counts, N= a+b is the equivalent sample size while A, B are the data counts and A+B is the data size.
What is the meaning of ε in the previous slide ?

7.  =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’))
Generalizing to a Dice with 6 sides
If our a priori idea is equivalent to seeing a1,…,a6 counts from each state and we now see data A1 …A6 counts from each state, then what is a natural estimate for  ?
If we use a point estimate  = (a1/N,…,an/N) then after seeing the data our point estimate changes to
 =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’))
Note that posterior estimate can be updated after each data point (namely in sequel or “online”) and that the posterior can serve as prior for future data points.

8. Another view of the update
We just saw that from  = (a1/N,…,an/N) we move on after seeing the data to i = ( ai+Ai)/(N+N’) for each i.
This update can be viewed as a weighted mixture of prior knowledge with data estimates:
i = N/(N+N’) (ai/N) + N’ /(N+N’) (Ai/N’)

9. The Full Bayesian Approach
Use a priori knowledge about the parameters.
Encode uncertainty about the parameters via a distribution.
Update the distribution after data is seen.
Choose expected value as estimate.

10. p(|data) =  p() p(data|)
(As before)
p(|data) =  p() p(data|)

11. p(|data) = p(| hhth …ttth) =  p() p( hhth …ttth|)
If we had more 100 heads the peak would move much more to the right.
If we had 50 heads and 50 tails the peak would just sharpen considerably.
p(|data) = p(| hhth …ttth) =  p() p( hhth …ttth|)
p(| data) =  p() #h (1-) #t = p(| #h, #t)
(#h, #t) are sufficient statistics for binomial sampling

12. The Beta distribution as a prior for =f

13. 

14. 

15. 

16. The Beta prior distribution for 
Example: ht … htthh

19. From Prior to Posterior
Observation 1: If the prior is Beta(;a,b) and we have seen A heads and B tails, then the posterior is Beta(;A+a,B+b).
Consequence: If the prior is Beta(;a,b) and we use a point estimate  = a/N, then after seeing the data our point estimate changes to = (A+a)/(N+N’) where N’=A+B.
So what is a good choice for the hyper-parameters {a, b} ?
For a random coin, maybe (100,100)?
For a random thumbtack maybe (7,3)?
a and b are imaginary counts, N=a+b is the equivalent sample size while A, B are the data counts and A+B is the data size.

24. From Prior to Posterior in Blocks
Observation 2: If the prior is Dir(a1,…,an) and we have seen A1 …An counts from each state, then the posterior is Dir(a1+A1,…, an+An).
Consequence: If the prior is Dir(a1,…,an) and we use a point estimate  = (a1/N,…,an/N) then after seeing the data our point estimate changes to
 =( ( a1+A1)/(N+N’), … ,(an+An)/(N+N’))
Note that posterior distribution can be updated after each data point (namely in sequel or “online”) and that the posterior can serve as prior for the future data points.

25. Another view of the update
Recall the Consequence that from  = (a1/N,…,an/N) we move on after seeing the data to i = ( ai+Ai)/(N+N’) for each i.
This update can be viewed as mixture of prior and data estimates:
i = N/(N+N’) (ai/N) + N’(N+N’) (Ai/N’)

27. Learning Bayes Net parameters with complete data – no hidden variables

28. Learning Bayes Net parameters
p(), p(h| ) =  and p(t| ) = 1- 

29. Learning Bayes Net parameters

30. P(X=x | x) = x
P(Y=y |X=x, y|x, y|~x)= y|x
P(Y=y |X=~x, y|x, y|~x)= y|~x
Global and Local parameter independence  three separate independent thumbtack estimation tasks, assuming complete data.