1. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 31 Linear Algebraic Equations Part 3 An equation of the form ax+by+c=0 or equivalently ax+by=- c is called a linear equation in x and y variables. ax+by+cz=d is a linear equation in three variables, x, y, and z. Thus, a linear equation in n variables is a 1 x 1 +a 2 x 2 + … +a n x n = b A solution of such an equation consists of real numbers c 1, c 2, c 3, …, c n. If you need to work more than one linear equations, a system of linear equations must be solved simultaneously.

2. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 32 Noncomputer Methods for Solving Systems of Equations For small number of equations (n ≤ 3) linear equations can be solved readily by simple techniques such as “method of elimination.” Linear algebra provides the tools to solve such systems of linear equations. Nowadays, easy access to computers makes the solution of large sets of linear algebraic equations possible and practical.

3. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 33 Gauss Elimination Chapter 9 Solving Small Numbers of Equations There are many ways to solve a system of linear equations: –Graphical method –Cramer’s rule –Method of elimination –Computer methods For n ≤ 3

4. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 34 Graphical Method For two equations: Solve both equations for x 2:

5. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 35 Plot x 2 vs. x 1 on rectilinear paper, the intersection of the lines present the solution. Fig. 9.1

6. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 36 Figure 9.2

7. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 37 Determinants and Cramer’s Rule Determinant can be illustrated for a set of three equations: Where [A] is the coefficient matrix:

8. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 38 Assuming all matrices are square matrices, there is a number associated with each square matrix [A] called the determinant, D, of [A]. If [A] is order 1, then [A] has one element: [A]=[a 11 ] D=a 11 For a square matrix of order 3, the minor of an element a ij is the determinant of the matrix of order 2 by deleting row i and column j of [A].

9. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 39

10. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 310 Cramer’s rule expresses the solution of a systems of linear equations in terms of ratios of determinants of the array of coefficients of the equations. For example, x 1 would be computed as:

11. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 311 Method of Elimination The basic strategy is to successively solve one of the equations of the set for one of the unknowns and to eliminate that variable from the remaining equations by substitution. The elimination of unknowns can be extended to systems with more than two or three equations; however, the method becomes extremely tedious to solve by hand.

12. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 312 Naive Gauss Elimination Extension of method of elimination to large sets of equations by developing a systematic scheme or algorithm to eliminate unknowns and to back substitute. As in the case of the solution of two equations, the technique for n equations consists of two phases: –Forward elimination of unknowns –Back substitution

13. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 313 Fig. 9.3

14. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 314 Pitfalls of Elimination Methods Division by zero. It is possible that during both elimination and back-substitution phases a division by zero can occur. Round-off errors. Ill-conditioned systems. Systems where small changes in coefficients result in large changes in the solution. Alternatively, it happens when two or more equations are nearly identical, resulting a wide ranges of answers to approximately satisfy the equations. Since round off errors can induce small changes in the coefficients, these changes can lead to large solution errors.

15. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 315 Singular systems. When two equations are identical, we would loose one degree of freedom and be dealing with the impossible case of n-1 equations for n unknowns. For large sets of equations, it may not be obvious however. The fact that the determinant of a singular system is zero can be used and tested by computer algorithm after the elimination stage. If a zero diagonal element is created, calculation is terminated.

16. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 316 Techniques for Improving Solutions Use of more significant figures. Pivoting. If a pivot element is zero, normalization step leads to division by zero. The same problem may arise, when the pivot element is close to zero. Problem can be avoided: –Partial pivoting. Switching the rows so that the largest element is the pivot element. –Complete pivoting. Searching for the largest element in all rows and columns then switching.

17. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 317 Gauss-Jordan It is a variation of Gauss elimination. The major differences are: –When an unknown is eliminated, it is eliminated from all other equations rather than just the subsequent ones. –All rows are normalized by dividing them by their pivot elements. –Elimination step results in an identity matrix. –Consequently, it is not necessary to employ back substitution to obtain solution.

18. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 318 LU Decomposition and Matrix Inversion Chapter 10 Provides an efficient way to compute matrix inverse by separating the time consuming elimination of the Matrix [A] from manipulations of the right-hand side {B}. Gauss elimination, in which the forward elimination comprises the bulk of the computational effort, can be implemented as an LU decomposition.

19. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 319 If L- lower triangular matrix U- upper triangular matrix Then, [A]{X}={B} can be decomposed into two matrices [L] and [U] such that [L][U]=[A] [L][U]{X}={B} Similar to first phase of Gauss elimination, consider [U]{X}={D} [L]{D}={B} –[L]{D}={B} is used to generate an intermediate vector {D} by forward substitution –Then, [U]{X}={D} is used to get {X} by back substitution.

20. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 320 Fig.10.1

21. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 321 LU decomposition requires the same total FLOPS as for Gauss elimination. Saves computing time by separating time- consuming elimination step from the manipulations of the right hand side. Provides efficient means to compute the matrix inverse

22. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 322 Error Analysis and System Condition Inverse of a matrix provides a means to test whether systems are ill-conditioned. Vector and Matrix Norms Norm is a real-valued function that provides a measure of size or “length” of vectors and matrices. Norms are useful in studying the error behavior of algorithms.

23. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 323

24. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 324 Figure 10.6

25. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 325 The length of this vector can be simply computed as Length or Euclidean norm of [F] For an n dimensional vector Frobenius norm

26. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 326 Frobenius norm provides a single value to quantify the “size” of [A]. Matrix Condition Number Defined as For a matrix [A], this number will be greater than or equal to 1.

27. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 327 That is, the relative error of the norm of the computed solution can be as large as the relative error of the norm of the coefficients of [A] multiplied by the condition number. For example, if the coefficients of [A] are known to t-digit precision (rounding errors~10 -t ) and Cond [A]=10 c, the solution [X] may be valid to only t-c digits (rounding errors~10 c-t ).

28. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 328 Special Matrices and Gauss-Seidel Chapter 11 Certain matrices have particular structures that can be exploited to develop efficient solution schemes. –A banded matrix is a square matrix that has all elements equal to zero, with the exception of a band centered on the main diagonal. These matrices typically occur in solution of differential equations. –The dimensions of a banded system can be quantified by two parameters: the band width BW and half-bandwidth HBW. These two values are related by BW=2HBW+1. Gauss elimination or conventional LU decomposition methods are inefficient in solving banded equations because pivoting becomes unnecessary.

29. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 329 Figure 11.1

30. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 330 Tridiagonal Systems A tridiagonal system has a bandwidth of 3: An efficient LU decomposition method, called Thomas algorithm, can be used to solve such an equation. The algorithm consists of three steps: decomposition, forward and back substitution, and has all the advantages of LU decomposition.

31. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 331 Gauss-Seidel Iterative or approximate methods provide an alternative to the elimination methods. The Gauss-Seidel method is the most commonly used iterative method. The system [A]{X}={B} is reshaped by solving the first equation for x 1, the second equation for x 2, and the third for x 3, …and n th equation for x n. For conciseness, we will limit ourselves to a 3x3 set of equations.

32. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 332 Now we can start the solution process by choosing guesses for the x’s. A simple way to obtain initial guesses is to assume that they are zero. These zeros can be substituted into x 1 equation to calculate a new x 1 =b 1 /a 11.

33. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 333 New x 1 is substituted to calculate x 2 and x 3. The procedure is repeated until the convergence criterion is satisfied: For all i, where j and j-1 are the present and previous iterations.

34. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 334 Fig. 11.4

35. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 335 Convergence Criterion for Gauss- Seidel Method The Gauss-Seidel method has two fundamental problems as any iterative method: –It is sometimes nonconvergent, and –If it converges, converges very slowly. Recalling that sufficient conditions for convergence of two linear equations, u(x,y) and v(x,y) are

36. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 336 Similarly, in case of two simultaneous equations, the Gauss-Seidel algorithm can be expressed as

37. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 337 Substitution into convergence criterion of two linear equations yield: In other words, the absolute values of the slopes must be less than unity for convergence:

38. Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. by Lale Yurttas, Texas A&M University Part 338 Figure 11.5