Presentation is loading. Please wait.

Presentation is loading. Please wait.

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 3: LINEAR MODELS FOR REGRESSION.

Similar presentations


Presentation on theme: "PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 3: LINEAR MODELS FOR REGRESSION."— Presentation transcript:

1 PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 3: LINEAR MODELS FOR REGRESSION

2 Linear Basis Function Models (1) Example: Polynomial Curve Fitting

3 Sum-of-Squares Error Function

4 0 th Order Polynomial

5 1 st Order Polynomial

6 3 rd Order Polynomial

7 9 th Order Polynomial

8 Linear Basis Function Models (2) Generally where Á j (x) are known as basis functions. Typically, Á 0 (x) = 1, so that w 0 acts as a bias. In the simplest case, we use linear basis functions : Á d (x) = x d.

9 Linear Basis Function Models (3) Polynomial basis functions: These are global; a small change in x affect all basis functions.

10 Linear Basis Function Models (4) Gaussian basis functions: These are local; a small change in x only affect nearby basis functions. ¹ j and s control location and scale (width).

11 Linear Basis Function Models (5) Sigmoidal basis functions: where Also these are local; a small change in x only affect nearby basis functions. ¹ j and s control location and scale (slope).

12 Maximum Likelihood and Least Squares (1) Assume observations from a deterministic function with added Gaussian noise: which is the same as saying, Given observed inputs,, and targets,, we obtain the likelihood function where

13 Maximum Likelihood and Least Squares (2) Taking the logarithm, we get where is the sum-of-squares error.

14 Computing the gradient and setting it to zero yields Solving for w, we get where Maximum Likelihood and Least Squares (3) The Moore-Penrose pseudo-inverse,.

15 Maximum Likelihood and Least Squares (4) Maximizing with respect to the bias, w 0, alone, we see that We can also maximize with respect to ¯, giving

16 Geometry of Least Squares Consider S is spanned by. w ML minimizes the distance between t and its orthogonal projection on S, i.e. y. N -dimensional M -dimensional

17 Sequential Learning Data items considered one at a time (a.k.a. online learning); use stochastic (sequential) gradient descent: This is known as the least-mean-squares (LMS) algorithm. Issue: how to choose ´ ?

18 1 st Order Polynomial

19 3 rd Order Polynomial

20 9 th Order Polynomial

21 Over-fitting Root-Mean-Square (RMS) Error:

22 Polynomial Coefficients

23 Regularized Least Squares (1) Consider the error function: With the sum-of-squares error function and a quadratic regularizer, we get which is minimized by Data term + Regularization term ¸ is called the regularization coefficient.

24 Regularized Least Squares (1) Is it true that or

25 Regularized Least Squares (2) With a more general regularizer, we have LassoQuadratic

26 Regularized Least Squares (3) Lasso tends to generate sparser solutions than a quadratic regularizer.

27 Understanding Lasso Regularizer Lasso regularizer plays the role of thresholding

28 Multiple Outputs (1) Analogously to the single output case we have: Given observed inputs,, and targets,, we obtain the log likelihood function

29 Multiple Outputs (2) Maximizing with respect to W, we obtain If we consider a single target variable, t k, we see that where, which is identical with the single output case.

30 The Bias-Variance Decomposition (1) Recall the expected squared loss, where The second term of E [L] corresponds to the noise inherent in the random variable t. What about the first term?

31 The Bias-Variance Decomposition (2) Suppose we were given multiple data sets, each of size N. Any particular data set, D, will give a particular function y(x; D ). We then have

32 The Bias-Variance Decomposition (3) Taking the expectation over D yields

33 The Bias-Variance Decomposition (4) Thus we can write where

34 Bias-Variance Tradeoff

35 The Bias-Variance Decomposition (5) Example: 25 data sets from the sinusoidal, varying the degree of regularization, ¸.

36 The Bias-Variance Decomposition (6) Example: 25 data sets from the sinusoidal, varying the degree of regularization, ¸.

37 The Bias-Variance Decomposition (7) Example: 25 data sets from the sinusoidal, varying the degree of regularization, ¸.

38 The Bias-Variance Trade-off From these plots, we note that an over-regularized model (large ¸ ) will have a high bias, while an under- regularized model (small ¸ ) will have a high variance.

39 Bayesian Linear Regression (1) Define a conjugate prior over w Combining this with the likelihood function and using results for marginal and conditional Gaussian distributions, gives the posterior where

40 Bayesian Linear Regression (2) A common choice for the prior is for which Next we consider an example …

41 Bayesian Linear Regression (3) 0 data points observed Prior Data Space

42 Bayesian Linear Regression (4) 1 data point observed Likelihood Posterior Data Space

43 Bayesian Linear Regression (5) 2 data points observed Likelihood Posterior Data Space

44 Bayesian Linear Regression (6) 20 data points observed Likelihood Posterior Data Space

45 Predictive Distribution (1) Predict t for new values of x by integrating over w : where

46 Predictive Distribution (1) How to compute the predictive distribution ?

47 Predictive Distribution (2) Example: Sinusoidal data, 9 Gaussian basis functions, 1 data point

48 Predictive Distribution (3) Example: Sinusoidal data, 9 Gaussian basis functions, 2 data points

49 Predictive Distribution (4) Example: Sinusoidal data, 9 Gaussian basis functions, 4 data points

50 Predictive Distribution (5) Example: Sinusoidal data, 9 Gaussian basis functions, 25 data points

51 Salesman’s Problem Given a consumer a described by a vector x a product b to sell with base cost c estimated price distribution of b in the mind of a is Pr(t|x, w) = N(x T w,  2 ) What is price should we offer to a ?

52 Salesman’s Problem

53 Bayesian Model Comparison (1) How do we choose the ‘right’ model?

54 Bayesian Model Comparison (1) How do we choose the ‘right’ model? Assume we want to compare models M i, i=1, …,L, using data D ; this requires computing PosteriorPrior Model evidence or marginal likelihood

55 Bayesian Model Comparison (1) How do we choose the ‘right’ model? PosteriorPrior Model evidence or marginal likelihood

56 Bayesian Model Comparison (3) For a model with parameters w, we get the model evidence by marginalizing over w

57 Bayesian Model Comparison (4) For a given model with a single parameter, w, con- sider the approximation where the posterior is assumed to be sharply peaked.

58 Bayesian Model Comparison (4) For a given model with a single parameter, w, con- sider the approximation where the posterior is assumed to be sharply peaked.

59 Bayesian Model Comparison (5) Taking logarithms, we obtain With M parameters, all assumed to have the same ratio, we get Negative Negative and linear in M.

60 Bayesian Model Comparison (5) Data matching Model Complexity

61 Bayesian Model Comparison (6) Matching data and model complexity

62 What You Should Know Least square and its maximum likelihood interpretation Sequential learning for regression Regularization and its effect Bayesian regression Predictive distribution Bias variance tradeoff Bayesian model selection


Download ppt "PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 3: LINEAR MODELS FOR REGRESSION."

Similar presentations


Ads by Google