1. CS 2750: Machine Learning Line Fitting + Bias-Variance Trade-off
Prof. Adriana Kovashka University of Pittsburgh
January 26, 2017

2. Test set (labels unknown)
Generalization
Training set (labels known)
Test set (labels unknown)
How well does a learned model generalize from the data it was trained on to a new test set?
Slide credit: L. Lazebnik

3. Generalization Components of expected loss
Noise in our observations: unavoidable
Bias: how much the average model over all training sets differs from the true model
Error due to inaccurate assumptions/simplifications made by the model
Variance: how much models estimated from different training sets differ from each other
Underfitting: model is too “simple” to represent all the relevant class characteristics
High bias and low variance
High training error and high test error
Overfitting: model is too “complex” and fits irrelevant characteristics (noise) in the data
Low bias and high variance
Low training error and high test error
Adapted from L. Lazebnik

4. Bias-Variance Trade-off
Models with too few parameters are inaccurate because of a large bias (not enough flexibility).
Models with too many parameters are inaccurate because of a large variance (too much sensitivity to the sample).
Purple dots = possible test points
Red dots = training data (all that we see before we ship off our model!)
Green curve = true underlying model Blue curve = our predicted model/fit
Adapted from D. Hoiem

5. Polynomial Curve Fitting
Slide credit: Chris Bishop

6. Sum-of-Squares Error Function
Slide credit: Chris Bishop

7. 0th Order Polynomial
Slide credit: Chris Bishop

8. 1st Order Polynomial
Slide credit: Chris Bishop

9. 3rd Order Polynomial
Slide credit: Chris Bishop

10. 9th Order Polynomial
Slide credit: Chris Bishop

11. Over-fitting
Root-Mean-Square (RMS) Error:
Slide credit: Chris Bishop

12. Data Set Size:
9th Order Polynomial
Slide credit: Chris Bishop

13. Data Set Size:
9th Order Polynomial
Slide credit: Chris Bishop

14. Regularization
Penalize large coefficient values (Remember: We want to minimize this expression.)
Adapted from Chris Bishop

15. Regularization:
Slide credit: Chris Bishop

16. Regularization:
Slide credit: Chris Bishop

17. Polynomial Coefficients
Slide credit: Chris Bishop

18. Polynomial Coefficients
No regularization Huge regularization
Adapted from Chris Bishop

19. Regularization: vs.
Slide credit: Chris Bishop

20. Training vs test error Underfitting Overfitting Complexity Error
Low Bias
High Variance
High Bias
Low Variance
Error
Test error
Training error
Slide credit: D. Hoiem

21. The effect of training set size
Complexity
Low Bias
High Variance
High Bias
Low Variance
Test Error
Few training examples
Note: these figures don’t work in pdf
Many training examples
Slide credit: D. Hoiem

22. The effect of training set size
Fixed prediction model
Number of Training Examples
Error
Testing
Generalization Error
Training
Adapted from D. Hoiem

23. Choosing the trade-off between bias and variance
Need validation set (separate from the test set)
Complexity
Low Bias
High Variance
High Bias
Low Variance
Error
Validation error
Training error
Slide credit: D. Hoiem

24. Bias-variance (Bishop Sec. 3.2)
Figure from Chris Bishop

25. How to reduce variance? Get more training data
Regularize the parameters
Choose a simpler classifier
Slide credit: D. Hoiem

26. Remember… Three kinds of error Try simple classifiers first
Inherent: unavoidable
Bias: due to over-simplifications
Variance: due to inability to perfectly estimate parameters from limited data
Try simple classifiers first
Use increasingly powerful classifiers with more training data (bias-variance trade-off)
Adapted from D. Hoiem