Learning Theory Reza Shadmehr Distribution of the ML estimates of model parameters Signal dependent noise models

Review: maximum likelihood estimate of parameters and noise The “true” underlying process What we measured Our model of the process Our ML estimate, given X : Log-likelihood function to maximize:

Variance of scalar and vector random variables cov of vector random variables produce symmetric positive definite matrices

We say that vector x has a multivariate Gaussian distribution with mean vector  and variance-covariance matrix , iff each element has Normal distribution. Multivariate Normal distribution d: dimension of x Variance-covariance matrix

Bias of the parameter estimates for a given X Suppose the outputs y were actually produced by the process: The “true” underlying process What we measured Our model of the process Given a constant X, the underlying process would give us different y every time that we run it. If on each run we find an ML w and , how would w and  vary with respect to w* ? If there were no noise: ML estimate: In the absence of noise, ML estimate would recover w* exactly.

Bias of the parameter estimates for a given X How does the ML estimate behave in the presence of noise in y? The “true” underlying process What we measured Our model of the process nx1 vector ML estimate: Because  is normally distributed: In other words:

Bias and variance of an estimator Parameters of a distribution can be estimated (e.g., via ML). Here we assess the “goodness” of that estimate by quantifying its bias and variance. Given some data samples: Bias of the estimator is the expectation of the deviation from the true value of the parameter. Variance of the estimator is the anticipated uncertainty in the estimate due to the particular selection of the samples. Note that bias of an estimator is similar to structural errors. Variance of the estimator is similar to approximation errors.

… so is unbiased. But what about its variance? Bias of the ML parameter estimates for a given X

Variance of the parameter estimates for a given X For a given X, the ML (or least square) estimate of our parameter has this normal distribution: Matrix of constants vector of random variables Assume: mxm

Variance of the parameter estimates for a given X More formally:

Example m: dimension of w determinant probability density

Example m: dimension of w determinant

Bias of the ML estimate of  Preliminaries: a. compute expected value of the residuals Preliminaries: b. compute variance of the residuals nxn matrix n:number of data points

Bias of the ML estimate of  Preliminaries: b. compute variance of the residuals

Bias of the ML estimate of  Preliminaries: C. useful properties of trace operator

Bias of the ML estimate of  Remember that residuals are mean zero.

Bias of the ML estimate of  Number of parameters in w Number of data points nxn nxm So the ML estimate of  is biased, it tends to underestimate the actual variance of the noise. The bias becomes smaller as number of data points increase with respect to the number of unknown parameters.

When noise in each data sample has an independent variance (Midterm 2005)

When noise in each data sample has an independent variance This is the weighted least squares solution where each data point is weighted by the inverse of the noise.

Example: sensory noise may be proportional to the “signal”.

Summary n=Number of data points m=Number of parameters in w