1. Evaluating Hypotheses
CS 9633 Machine Learning
Evaluating Hypotheses
Adapted from notes by Tom Mitchell
Computer Science Department
CS 9633 Machine Learning

2. Computer Science Department
Three big questions
Given the observed accuracy of a hypothesis over a limited sample of data, how well does this estimate its accuracy over additional data?
Given that one hypothesis outperforms another over some sample of data, how probable is it that this hypothesis is more accurate in general?
When data is limited, what is the best way to use this data to both learn a hypothesis and estimate its accuracy?
Computer Science Department
CS 9633 Machine Learning

3. Difficulties in estimating hypotheses with small data sets
Bias in the estimate
Performance over training data provides an over-optimistic estimate of performance
More likely with rich hypothesis space
Typically use test set to estimate accuracy
Variance in the estimate
Measured accuracy over test set can still differ from true accuracy
The smaller the test sample, the greater the expected variance
Computer Science Department
CS 9633 Machine Learning

4. Estimating Hypothesis Accuracy
We want to know the accuracy of the hypothesis in classifying future instances
We also want to know the probably error of the accuracy estimate
Computer Science Department
CS 9633 Machine Learning

5. Two Questions More Precisely
Given a hypothesis h and a data sample containing n examples drawn at random according to the distribution D, what is the best estimate of the accuracy of h over future instances drawn from the same distribution?
What is the probably error of the accuracy estimate?
Computer Science Department
CS 9633 Machine Learning

6. Sample Error and True Error
Sample error: error rate of the hypothesis over the sample of data that is available
True Error: the error rate of the hypothesis over the entire unknown distribution D of examples
Computer Science Department
CS 9633 Machine Learning

7. Computer Science Department
Sample Error
Definition: The sample error (denoted errors(h)) of hypothesis h with respect to target function f and data sample S is
Where n is the number of examples in S, and the quantity d(f(x),h(x)) is 1 if f(x)h(x) and 0 otherwise.
Computer Science Department
CS 9633 Machine Learning

8. Computer Science Department
True Error
Definition: The true error (denoted errorD(h)) of hypothesis h with respect to target function f and distribution D , is the probability that h will misclassify an instance drawn at random according to D.
Computer Science Department
CS 9633 Machine Learning

9. Computer Science Department
The problem
We want to know errorD(h) but we can only obtain errors(h)
So we must determine how well the sample error estimates the true error.
Computer Science Department
CS 9633 Machine Learning

10. Confidence Intervals for Discrete-Valued Hypotheses
We wish to estimate the true error for some discrete-valued hypothesis h, based on its observed sample error over a sample S where
The sample S contains n examples drawn independent of one another, and independent of h, according to the probability distribution D
n 30
Hypothesis h commits r errors over these n examples
Errors(h)  r/n
Computer Science Department
CS 9633 Machine Learning

11. Computer Science Department
Important Result
Given no other information, the most probable value of errorD(h) is errors(h)
With approximately 95% probability, the true errorD(h) lies in the interval
Computer Science Department
CS 9633 Machine Learning

12. Computer Science Department
More General Result
Confidence Level N%
50%
68%
80%
90%
95%
98%
99%
Constant zN
0.67
1.00
1.28
1.64
1.96
2.33
2.58
Computer Science Department
CS 9633 Machine Learning

13. Where the Binomial Distribution Applies
Notation
Let n be the number of trials
Let r be the number of successes
Let p be the probability of success
Conditions for Application
Two possible outcomes (success and failure)
The probability of success is the same on each trial
There are n trials, where n is constant
The n trials are independent
Computer Science Department
CS 9633 Machine Learning

14. The Binomial Distribution
Computer Science Department
CS 9633 Machine Learning

15. Mean (Expected Value) and Variance
Computer Science Department
CS 9633 Machine Learning

16. Difference in sampling error and true error
errors(h) is an estimator of errorD(h)
Is it an unbiased estimator?
The estimation bias of an estimator Y for an arbitrary parameter p is
E[Y]-p
If the estimation bias is zero, Y is an unbiased estimator of P
Computer Science Department
CS 9633 Machine Learning

17. Answer to previous question
What is expected value of r?
E(r) =
What is expected value of r/n?
E(r/n) =
Is r/n unbiased estimate of p?
Computer Science Department
CS 9633 Machine Learning

18. Important distinction
Estimation bias is a number
Inductive bias is a set of assertions
Computer Science Department
CS 9633 Machine Learning

19. Variance of an estimator
Problem: r = n = 50
What is variance of errors(h)?
Computer Science Department
CS 9633 Machine Learning

20. General Expression for Variance
Computer Science Department
CS 9633 Machine Learning

21. Computer Science Department
Confidence Intervals
Definition: An N% confidence interval for some parameter p is an interval that is expected with probability N% to contain p.
Computer Science Department
CS 9633 Machine Learning

22. Finding Confidence Intervals
Question: For a given value of N how can we find the size of the interval that contains N% of the probability mass?
Answer: Approximate binomial distribution with normal distribution
Rule of thumb: ok for n>30
Computer Science Department
CS 9633 Machine Learning

23. Computer Science Department
Results
If a random variable Y obeys a normal distribution with mean  and standard deviation , then the measured value y of Y will fall in the following interval N% of the time
  zN
Equivalently, the mean  will fall in the following interval N% of the time
y  zN
Computer Science Department
CS 9633 Machine Learning

24. N% confidence interval
2 approximations
When estimating standard deviation used errors(h) as approximation of errorD(h)
Used normal distribution to approximate binomial
Computer Science Department
CS 9633 Machine Learning