Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem.

Similar presentations


Presentation on theme: "Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem."— Presentation transcript:

1 Ordinary Least-Squares

2 Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem

3 Ordinary Least-Squares One-dimensional regression

4 Ordinary Least-Squares One-dimensional regression Find a line that represent the ”best” linear relationship:

5 Ordinary Least-Squares One-dimensional regression Problem: the data does not go through a line

6 Ordinary Least-Squares One-dimensional regression Problem: the data does not go through a line Find the line that minimizes the sum:

7 Ordinary Least-Squares One-dimensional regression Problem: the data does not go through a line Find the line that minimizes the sum: We are looking for that minimizes

8 Ordinary Least-Squares Matrix notation Using the following notations and

9 Ordinary Least-Squares Matrix notation Using the following notations and we can rewrite the error function using linear algebra as:

10 Ordinary Least-Squares Matrix notation Using the following notations and we can rewrite the error function using linear algebra as:

11 Ordinary Least-Squares Multidimentional linear regression Using a model with m parameters

12 Ordinary Least-Squares Multidimentional linear regression Using a model with m parameters

13 Ordinary Least-Squares Multidimentional linear regression Using a model with m parameters

14 Ordinary Least-Squares Multidimentional linear regression Using a model with m parameters and n measurements

15 Ordinary Least-Squares Multidimentional linear regression Using a model with m parameters and n measurements

16 Ordinary Least-Squares

17

18 parameter 1

19 Ordinary Least-Squares measurement n parameter 1

20 Ordinary Least-Squares Minimizing

21 Ordinary Least-Squares Minimizing

22 Ordinary Least-Squares Minimizing is flat at

23 Ordinary Least-Squares Minimizing is flat at

24 Ordinary Least-Squares Minimizing is flat at does not go down around

25 Ordinary Least-Squares Minimizing is flat at does not go down around

26 Ordinary Least-Squares Positive semi-definite In 1-DIn 2-D

27 Ordinary Least-Squares Minimizing

28 Ordinary Least-Squares Minimizing

29 Ordinary Least-Squares Minimizing

30 Ordinary Least-Squares Minimizing Always true

31 Ordinary Least-Squares Minimizing Always true The normal equation

32 Ordinary Least-Squares Geometric interpretation

33 Ordinary Least-Squares Geometric interpretation b is a vector in R n

34 Ordinary Least-Squares Geometric interpretation b is a vector in R n The columns of A define a vector space range(A)

35 Ordinary Least-Squares Geometric interpretation b is a vector in R n The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A)

36 Ordinary Least-Squares Geometric interpretation b is a vector in R n The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A)

37 Ordinary Least-Squares Geometric interpretation is the orthogonal projection of b onto range(A)

38 Ordinary Least-Squares The normal equation:

39 Ordinary Least-Squares The normal equation: Existence: has always a solution

40 Ordinary Least-Squares The normal equation: Existence: has always a solution Uniqueness: the solution is unique if the columns of A are linearly independent

41 Ordinary Least-Squares The normal equation: Existence: has always a solution Uniqueness: the solution is unique if the columns of A are linearly independent

42 Ordinary Least-Squares Under-constrained problem

43 Ordinary Least-Squares Under-constrained problem

44 Ordinary Least-Squares Under-constrained problem

45 Ordinary Least-Squares Under-constrained problem Poorly selected data One or more of the parameters are redundant

46 Ordinary Least-Squares Under-constrained problem Poorly selected data One or more of the parameters are redundant Add constraints

47 Ordinary Least-Squares How good is the least-squares criteria? Optimality: the Gauss-Markov theorem

48 Ordinary Least-Squares How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define:

49 Ordinary Least-Squares How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If

50 Ordinary Least-Squares How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Then is the best unbiased linear estimator

51 Ordinary Least-Squares a b eiei no errors in a i

52 Ordinary Least-Squares a b eiei a b eiei no errors in a i errors in a i

53 Ordinary Least-Squares a b homogeneous errors

54 Ordinary Least-Squares a b a b homogeneous errors non-homogeneous errors

55 Ordinary Least-Squares a b no outliers

56 Ordinary Least-Squares a b a b no outliers outliers


Download ppt "Ordinary Least-Squares. Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem."

Similar presentations


Ads by Google