1. Over-fitting and Regularization Chapter 4 textbook Lectures 11 and 12 on amlbook.com

2. Over-fitting is easy to recognize in 1D Parabolic target function 4 th order hypothesis 5 data points -> E in = 0

3. The origin of over-fitting can be analyzed in 1D: Bias/variance dilemma. How does this apply to case on previous slide?

4. Shape of fit very sensitive to noise in data Out-of-sample error will vary greatly from one dataset to another.

5. Over-fitting is easy to avoid in 1D: Results from HW1 Sum of squared deviations Degree of polynomial E val E in

6. Using E val to avoid over-fitting works in all dimensions but computation grows rapidly for large d EE E in E cv-1 E val Digit recognition one vs not one d = 2 (intensity and symmetry) Terms in  5 (x) added successively 500 pts in training set Validation set needs to be large; 8798 this case

7. What if we want to add higher order terms to a linear model but don’t have enough data for a validation set? Solution: Augment the error function used to optimize weights Example Penalizes choices with large |w|. Called “weight decay”

8. Normal equations with weight decay essentially unchanged (Z T Z + I) w reg =Z T y

9. Best value is subjective In this case = 0.0001 is large enough to suppress swings but data still important in determining optimum weights.

11. Review for Quiz 2 Topics: linear models extending linear models by transformation dimensionality reduction over fitting and regularization 2 classes are distinguished by a threshold values of a linear combination of d attributes. Explain how h(w|x) = sign(w T x) becomes a hypothesis set for linear binary classification

12. More Review for Quiz 2 Topics: linear models extending linear models by transformation dimensionality reduction over fitting and regularization We have used 1-step optimization in 4 ways: polynomial regression in 1D (curve fitting) multivariate linear regression extending linear models by transformation regularization by weight decay 2 of these are equivalent; which ones

13. More Review for Quiz 2 Topics: linear models extending linear models by transformation dimensionality reduction over fitting and regularization 1-step optimization requires the in-sample error to be the sum of squared residuals. Define the in-sample error for following multivariate linear regression, extending linear models by transformation regularization by weight decay

14. For multivariate linear regression Derive the normal equations for extended linear regression with weight decay

15. Interpret the “learning curve” for multivariate linear regression when training data has normally distributed noise Why does E out approach  2 from above? Why does E in approach  2 from below? Why is E in not defined for N<d+1?

16. What do these learning curves say about simple vs complex models Still larger than bound set by noise

17. How do we estimate a good level of complexity without sacrificing training data?

18. Why chose 3 rather than 4?

19. Review:Maximum Likelihood Estimation Estimate parameters  of a probability distribution given a sample X drawn from that distribution 19 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

20. Form the likelihood function Likelihood of  given the sample X l (θ| X ) = p ( X |θ) = ∏ t p(x t |θ) Log likelihood L (θ| X ) = log( l (θ| X )) = ∑ t log p(x t |θ) Maximum likelihood estimator (MLE) θ * = argmax θ L (θ| X ) the value of θ that maximizes L (θ| X ) 20 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

21. How was MLE used in logistic regression to derive an expression for in-sample error?

22. In logistic regression, parameters are the weights Likelihood of w given the sample X l (w| X ) = p ( X |w) = ∏ t p(x t |w) Log likelihood L (w| X ) = log( l (w| X )) = ∑ t log p(x t |w) In logistic regression, p(x t |w) =  (y n w T x n )

23. Since Log is a monotone increasing function, maximizing log(likelihood) is equivalent to minimizing -log(likelihood) Text also normalizes by dividing by N; hence error function becomes How?

24. Derive the log-likelihood function for a 1D Gaussian distribution

25. Stochastic gradient decent: correct weights by error in each data point

26. I want to perform PCA on a dataset. What must I assume about the noise in data? PCA

27. Correlation coefficients of normally distributed attributes x are zero. What can we say about the covariance of x More PCA

28. Attributes x are normally distributed with mean  and covariance . z = Mx is a linear transformation to feature space defined by matrix M. What are the mean and covariance of these features? More PCA

29. z k is the a feature defined by projection of attributes in the direction of the eigenvector w k of the covariance matrix. Prove that eigenvalue k is the variance of z k 29 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) More PCA

30. How do we find values of x 1 and x 2 that minimize f(x 1, x 2 ) subject to the constraint g(x 1, x 2 ) = c? Constrained optimization Find stationary points of f(x 1, x 2 ) = 1 - x 1 2 – x 2 2 subject to constraint g(x 1, x 2 ) = x 1 + x 2 = 1