Linear Least Squares QR Factorization

Systems of linear equations Problem to solve: M x = b Given M x = b : Is there a solution? Is the solution unique?

Systems of linear equations Find a set of weights x so that the weighted sum of the columns of the matrix M is equal to the right hand side b

Systems of linear equations - Existence A solution exists when b is in the span of the columns of M A solution exists if: There exist weights, x 1, …., x N, such that:

Systems of linear equations - Uniqueness A solution is unique only if the columns of M are linearly independent. Then: Mx = b  Mx + My= b  M(x+y) = b Suppose there exist weights, y 1, …., y N, not all zero, such that:

QR factorization 1 A matrix Q is said to be orthogonal if its columns are orthonormal, i.e. Q T ·Q=I. Orthogonal transformations preserve the Euclidean norm since Orthogonal matrices can transform vectors in various ways, such as rotation or reflections but they do not change the Euclidean length of the vector. Hence, they preserve the solution to a linear least squares problem.

QR factorization 2 Any matrix A (m·n) can be represented as A = Q·R,where Q (m·n) is orthonormal and R (n·n) is upper triangular:

QR factorization 2 Given A, let its QR decomposition be given as A=Q·R, where Q is an (m x n) orthonormal matrix and R is upper triangular. QR factorization transform the linear least square problem into a triangular least squares. Q·R·x = b R·x = Q T ·b x=R -1 ·Q T ·b Matlab Code:

Normal Equations Consider the system It can be a result of some physical measurements, which usually incorporate some errors. Since, we can not solve it exactly, we would like to minimize the error: r=b-Ax r 2 =r T r=(b-Ax) T (b-Ax)=b T b-2x T A T b+x T A T Ax (r 2 ) x =0 - zero derivative is a (necessary) minimum condition -2A T b+2A T Ax=0; A T Ax = A T b; – Normal Equations

Normal Equations 2 A T Ax = A T b – Normal Equations

Least squares via A=QR decomposition A (m,n) =Q (m,n) R (n,n), Q is orthogonal, therefore Q T Q=I. QRx=b R (n,n) x=Q T (n,m) b (m,1) -well defined linear system x=R -1 Q T b Q is found by Gram=Schmidt orthogonalization of A. How to find R? QR=A Q T QR=Q T A, but Q is orthogonal, therefore Q T Q=I: R=Q T A R is upper triangular, since in orthogonalization procedure only a 1,..a k (without a k+1,…) are used to produce q k

Least squares via A=QR decomposition 2 Let us check the correctness: QRx=b Rx=Q T b x=R -1 Q T b

Last lecture reminder QR Factorization – By picture

For i = 1 to N “For each Target Column” For j = 1 to i-1 “For each Source Column left of target” end Normalize Orthogonalize Search Direction QR Factorization – Minimization View Minimization Algorithm