Direct Methods for Linear Systems Lecture 3 Alessandra Nardi Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

Last lecture review Formulation of circuit equations –Conservation laws (KCL, KVL) –Branch constitutive equations (BCE) –KCL, KVL, BCE combined in different ways: STA MNA

Outline Systems of linear equations –Existence and uniqueness review –Gaussian Elimination basics LU factorization Pivoting

Systems of linear equations Problem to solve: M x = b Given M x = b : –Is there a solution? –Is the solution unique?

Systems of linear equations Find a set of weights x so that the weighted sum of the columns of the matrix M is equal to the right hand side b

Systems of linear equations - Existence A solution exists when b is in the span of the columns of M A solution exists if: There exist weights, x 1, …., x N, such that:

Systems of linear equations - Uniqueness A solution is unique only if the columns of M are linearly independent. Then: Mx = b  Mx + My= b  M(x+y) = b Suppose there exist weights, y 1, …., y N, not all zero, such that:

Systems of linear equations Square matrices Given Mx = b, where M is square – If a solution exists for any b, then the solution for a specific b is unique. For a solution to exist for any b, the columns of M must span all N-length vectors. Since there are only N columns of the matrix M to span this space, these vectors must be linearly independent. A square matrix with linearly independent columns is said to be nonsingular.

Application Problems Matrix is n x n Often symmetric and diagonally dominant Nonsingular of real numbers

Methods for solving linear equations Direct methods: find the exact solution in a finite number of steps Iterative methods: produce a sequence a sequence of approximate solutions hopefully converging to the exact solution

Gaussian Elimination Basics Gaussian Elimination Method for Solving M x = b A “Direct” Method Finite Termination for exact result (ignoring roundoff) Produces accurate results for a broad range of matrices Computationally Expensive

Gaussian Elimination Basics Reminder by 3x3 example

Gaussian Elimination Basics – Key idea Use Eqn 1 to Eliminate x 1 from Eqn 2 and 3

GE Basics – Key idea in the matrix MULTIPLIERSPivot Remove x1 from eqn 2 and eqn 3

GE Basics – Key idea in the matrix Pivot Multiplier Remove x2 from eqn 3

GE Basics – Simplify the notation Remove x1 from eqn 2 and eqn 3

Pivot Multiplier GE Basics – Simplify the notation Remove x2 from eqn 3

GE Basics – GE yields triangular system Altered During GE ~ ~

GE Basics – Backward substitution

GE Basics – RHS updates

GE basics: summary (1) M x = b U x = yEquivalent system U: upper trg (2)Noticed that: Ly = bL: unit lower trg (3)U x = y LU x = b  M x = b GE  Efficient way of implementing GE: LU factorization

Solve M x = b Step 1 Step 2 Forward Elimination Solve L y = b Step 3 Backward Substitution Solve U x = y = M = L U Gaussian Elimination Basics Note: Changing RHS does not imply to recompute LU factorization

GE Basics – Fitting the pieces together

LU factorization Basics – Picture

LU Basics Source-row oriented approach algorithm For i = 1 to n-1 { “For each source row” For j = i+1 to n { “For each target row below the source” For k = i+1 to n { “For each row element beyond Pivot” } Pivot Multiplier

LU Basics Target-row oriented approach algorithm For i = 2 to n { “For each target row” For j = 1 to i-1 { “For each source row above the target” For k = j+1 to n { “For each row element beyond Pivot” } Pivot Multiplier

LU – Source-row and Target-row Multipliers Factored Portion Active Set k k Factored Portion Mult Active Set k k Source-Row oriented approach Target-Row oriented approach

For i = 1 to n-1 { “For each Row” For j = i+1 to n { “For each target Row below the source” For k = i+1 to n { “For each Row element beyond Pivot” } Pivot Multiplier multipliers Multiply-adds LU Basics – Computational Complexity

LU Basics – Limitations of the naïve approach Zero Pivots Small Pivots (Round-off error)  both can be solved with partial pivoting

At Step i Multipliers Factored Portion (L) Row i Row j What if Cannot form Simple Fix (Partial Pivoting) If Find Swap Row j with i LU Basics – Partial pivoting for zero pivots

Two Important Theorems 1) Partial pivoting (swapping rows) always succeeds if M is non singular 2) LU factorization applied to a diagonally dominant matrix will never produce a zero pivot LU Basics – Partial pivoting for zero pivots

LU Basics – Partial pivoting for small pivots GE

LU Basics – Partial pivoting for small pivots GE Rounded to 3 digits

64 bits 52 bits 11 bits sign Double precision number Basic Problem Avoid sum and subtraction of large and tiny numbers  Avoid big multipliers An Aside on Floating Point Arithmetic LU Basics – Partial pivoting for small pivots

Partial Pivoting for Roundoff reduction LU Basics – Partial pivoting for small pivots Small Multipliers

LU Basics – Partial pivoting for small pivots GE Rounded to 3 digits swap

Pivoting strategies Partial Pivoting: –Only row interchange Complete Pivoting –Row and Column interchange Threshold Pivoting –Only if prospective pivot is found to be smaller than a certain threshold k k k k

Summary Existence and uniqueness review Gaussian elimination basics –GE basics –LU factorization –Pivoting