# Ordinary Least-Squares

## Presentation on theme: "Ordinary Least-Squares"— Presentation transcript:

Ordinary Least-Squares

Outline Linear regression Geometry of least-squares
Discussion of the Gauss-Markov theorem Ordinary Least-Squares

One-dimensional regression
Ordinary Least-Squares

One-dimensional regression
Find a line that represent the ”best” linear relationship: Ordinary Least-Squares

One-dimensional regression
Problem: the data does not go through a line Ordinary Least-Squares

One-dimensional regression
Problem: the data does not go through a line Find the line that minimizes the sum: Ordinary Least-Squares

One-dimensional regression
Problem: the data does not go through a line Find the line that minimizes the sum: We are looking for that minimizes Ordinary Least-Squares

Matrix notation and Using the following notations
Ordinary Least-Squares

Matrix notation and Using the following notations
we can rewrite the error function using linear algebra as: Ordinary Least-Squares

Matrix notation and Using the following notations
we can rewrite the error function using linear algebra as: Ordinary Least-Squares

Multidimentional linear regression
Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression
Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression
Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression
Using a model with m parameters and n measurements Ordinary Least-Squares

Multidimentional linear regression
Using a model with m parameters and n measurements Ordinary Least-Squares

Ordinary Least-Squares

Ordinary Least-Squares

parameter 1 Ordinary Least-Squares

parameter 1 measurement n Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing is flat at Ordinary Least-Squares

Minimizing is flat at Ordinary Least-Squares

Minimizing is flat at does not go down around Ordinary Least-Squares

Minimizing is flat at does not go down around Ordinary Least-Squares

Positive semi-definite
In 1-D In 2-D Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Always true Ordinary Least-Squares

Minimizing The normal equation Always true Ordinary Least-Squares

Geometric interpretation
Ordinary Least-Squares

Geometric interpretation
b is a vector in Rn Ordinary Least-Squares

Geometric interpretation
b is a vector in Rn The columns of A define a vector space range(A) Ordinary Least-Squares

Geometric interpretation
b is a vector in Rn The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A) Ordinary Least-Squares

Geometric interpretation
b is a vector in Rn The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A) Ordinary Least-Squares

Geometric interpretation
is the orthogonal projection of b onto range(A) Ordinary Least-Squares

The normal equation: Ordinary Least-Squares

The normal equation: Existence: has always a solution
Ordinary Least-Squares

The normal equation: Existence: has always a solution
Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

The normal equation: Existence: has always a solution
Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

Under-constrained problem
Ordinary Least-Squares

Under-constrained problem
Ordinary Least-Squares

Under-constrained problem
Ordinary Least-Squares

Under-constrained problem
Poorly selected data One or more of the parameters are redundant Ordinary Least-Squares

Under-constrained problem
Poorly selected data One or more of the parameters are redundant Add constraints Ordinary Least-Squares

How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem Ordinary Least-Squares

How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: Ordinary Least-Squares

How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Ordinary Least-Squares

How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Then is the best unbiased linear estimator Ordinary Least-Squares

b ei a no errors in ai Ordinary Least-Squares

b b ei ei a a no errors in ai errors in ai Ordinary Least-Squares

b a homogeneous errors Ordinary Least-Squares

b b a a homogeneous errors non-homogeneous errors
Ordinary Least-Squares

b a no outliers Ordinary Least-Squares

outliers b b a a no outliers outliers Ordinary Least-Squares