1. Required Sample size for Bayesian network Structure learning
Samee Ullah Khan
and
Kwan Wai Bong Peter

2. Outline Motivation Introduction Sample Complexity Summary Conclusion
Sanjoy Dasgupta
Russell Greiner
Nir Friedman
David Haussler
Summary
Conclusion

3. Motivation John Works at a Pharmaceutical Company.
Optimal Sample Size of a Clinical Trial?
It’s a function of Both Statistical Significance of the Difference and the Magnitude of Apparent difference between Performances.
Purpose: A tool (measure) for Public and Commercial vendors to plan clinical trials.
Looking For: Gain acceptance from potential users.
Statistically Significance Evidence

4. Motivation: Solution
Optimize the difference between the performances of both treatments.
 Let C= diff (expected cost of new treatment –expected cost of old treatment)

5. Motivation
C=0, m= users,  is the difference in performance

6. Motivation
C>0

7. Motivation
C<0

8. Motivation: Conclusion
Actual improvement in performance is known
It may be extended to the uncertainty about the amount of improvement.
It is also possible to shift the functions 1` or 2`to right.
Where ` is standard deviation of the posterior distribution of unknown parameter .

9. Motivation: Model Paired Observations (X1,Y1),(X2,Y2)……..
Xi is new clinical outcome
Yi is old clinical outcome
Let Z be the objective function
Zi=Xi-Yi (i=1,2,3……….)
Assume that has normal density N(,2)
Formulating our previous knowledge about  assume a prior density N(,2).
Under the assumptions is a sufficient statistics for the parameter . 

10. Introduction Efficient learning -- more accurate models with less data
Compare: P(A) and P(B) vs joint P(A,B) former requires less data!
Discover structural properties of the domain
Identifying independencies in the domain helps to
Order events that occur sequentially
Sensitivity analysis and inference
Predict effect of actions
Involves learning causal relationship among variables

11. Introduction
Why Struggle for Accurate Structure

12. Introduction Adding an Arc
Increases the number of parameters to be fitted
Wrong assumptions about causality and domain structure

13. Introduction Deleting an Arc
Cannot be compensated by accurate fitting of parameters
Also misses causality and domain structure

14. Introduction Approaches to Learning Structure Constraint based
Perform tests of conditional independence
Search for a network that is consistent with the observed dependencies and independencies
Score based
Define a score that evaluates how well the (in)dependencies in a structure match the observations
Search for a structure that maximizes the score

15. Introduction Constraints versus Scores Constraint based Score based
Intuitive, follows closely the definition of BNs
Separates structure construction from the form of the independence tests
Sensitive to errors in individual tests
Score based
Statistically motivated
Can make compromises
Both
Consistent---with sufficient amounts of data and computation, they learn the correct structure

16. Dasgupta’s model Haussler’s extension of the PAC framework
Situation: fixed network structure
Goal: To learn the conditional probability functions accurately

17. Dasgupta’s model A learning algorithm A: Given:
An approximation parameter  > 0
A confidence parameter 0 <  < 1
Variables drawn from a instance space X, x1, x2, …, xn
An oracle which generates randomly instances of X according to some unknown distribution P that we are going to learn
Some hypothesis class H

18. Dasgupta’s model
Output: hypothesis h  H such that with probability > 1-
where
d(.,.) is a distance measure
hopt is the concept h’  H that minimizes d(P, h’)

19. Dasgupta’s model: Distance measure
Most intuitive: L1 norm
Most popular: Kullback-Leibler divergence (relative entropy)
Minimizing dKL with respect to the empirically observed distribution is equivalent to solving the maximum likelihood problem

20. Dasgupta’s model: Distance measure
Disadvantage of dKL: unbounded
So, the measure adopted in this model is relative entropy by replacing log with ln.

21. Dasgupta’s model
The algorithm, given m samples drawn from some distribution P, finds the best fitting hypothesis by evaluating each h(,)H(,) by computing the empirical log loss E(-ln h(,)) and returning the hypothesis with the smallest value, where H(,)H, called an (,)-bounded approximation of H.

22. Dasgupta’s model
By using Hoeffding and Chernoff bounds, the number of samples needed is bounded by
Lower bound:

23. Rusell Greiner’s claim
Many learning algorithms that determine which Bayesian network is optimal usually based on some measures such as log-likelihood, MDL, BIC. These typical measures are independent of the queries that will be posed.
Learning algorithms should consider the distribution of queries as well as the underlying distribution of events, and seek the BN with the best performance over the query distribution rather than the one that appears closest to the underlying event distribution.

24. Russell Greiner’s model
Let
V: set of the N variables
SQ: set of all possible legal statistical queries
sq(x; y): a distribution over SQ
Suppose we fixed a network B over V, and let B(x|y) be the real-value probability that B returns for this assignment. Given distribution sq(.,.) over SQ, the “score” of B is
err(B)=errsq,p(B) if sq, p are clear from context

25. Russell Greiner’s model
Observation:
Any Bayesian network B* that encodes the underlying distribution p(.), will in fact produce the optimal performance; i.e. err(B*) will be optimal
This means that if we have a learning algorithm that produces better approximations to p(.) as it sees more training examples, then in the limit the sq(.) distribution becomes irrelevant.

26. Russell Greiner’s model
Given a set of labeled statistical queries Q={<xi;yi;pi>}i, let
be the empirical score of the Bayesian net.

27. Russell Greiner’s model
Compute err(B):
#P-hard to compute the estimate of err(B) from general statistical queries
If we know that all queries encountered sq(x;y), satisfy p(y) for some >0, then we only need
complete event examples, with
example queries to obtain an -close estimate, with probability at least 1-.

28. Nir Friedman’s model Review BN is composed of two parts. Setup DAG
Parameters encoding
Setup
Let B* be a BN that describe the target distributions from training samples.
Entropy Distance (Kullback-Leibler)
Learn from Random Variables, decrease with N.

29. Nir Friedman’s model: Learning
Criteria:
Error Threshold 
Confidence Threshold 
N(,) sample size
If the sample size is larger than N(,) then Pr(D(PLrn()||P)>)< where Lrn() represents the learning routine.
If N(,) is MINIMAL the it is called sample complexity.

30. Nir Friedman’s model:Notations
Vector Valued U={X1, X2,……Xn}
X,Y,Z Variables
x,y,z  values
So B=<G,>
G is DAG
 are number of parameters
xi|xi =P(xi|xi)
BN is minimal

31. Nir Friedman’s model:Learning
Given a training set wN={u1,……..un} of U find B that best matches D.
The loglikelihood of B:
Decomposing loglikelihood according to structure:

32. Nir Friedman’s model:Learning
So we can derive
Assume G has fixed structure, optimize 
Argument is large networks not desirable

33. Nir Friedman’s model: PSM
Penalized weighting function:
MDL principle:
Total description length of data
AIC
BIC

34. Nir Friedman’s model: Sample Complexity
Log-likelihood and penality term
Random noise
Entropy distance

35. Nir Friedman’s model: Sample Complexity
Idealized case

36. Nir Friedman’s model: Sample Complexity
Sub-sampling strategies in learning

37. Nir Friedman’s model: Summary
It can be shown on the sample complexity of BN using MDL
Bound is loose
To search for an optimal structure is NP-hard

38. David Haussler’s model
The model is based on prediction. The learner attempts to infer an unknown target concept f chosen from a concept class F of {0, 1} valued function.
For any given instance i, the learner predicts value of f(xi).
After the prediction, the learner is to the correct answer. It improves on the result.

39. David Haussler’s model
Criteria for sample bounds:
Probability of f(xm+1) over (x1, f(x1)), …,(xm,f(xm))
Cumulative mistakes made over m trials
The model uses VC dimension

40. VC General condition for uniform convergence: Definition:
Shattered set. Let X be the instance space and C the concept class
SX, shattered by C
S’ S, c C which contains all S’ and none of S-S’
SX, C(S)  S

41. David Haussler’s model
Information Gain
At instance m, the learner has observed f(x1),…,f(xm) labels predict f(xm+1)

42. David Haussler’s model