1. Rutgers University School of Engineering Spring :440:127 - Introduction to Computers for Engineers Sophocles J. Orfanidis ECE Department
week 10

2. Weekly Topics
Week Basics – variables, arrays, matrices, plotting (ch. 2 & 3)
Week Basics – operators, functions, strings, cells (ch. 2,3,11)
Week Matrices (ch. 4)
Week Plotting – 2D and 3D plots (ch. 5)
Week User-defined functions (ch. 6)
Week Input-output formatting – fprintf, fscanf (ch. 7) – Exam 1
Week Relational & logical operators, if, switch statements (ch. 8)
Week For-loops, while-loops (ch. 9)
Week Review for weeks 5-8
Week 10 - Matrix algebra – solving linear equations (ch. 10) – Exam 2
Week 11 - Numerical methods I (ch. 13)
Week 12 - Numerical methods II (ch. 13)
Week 13 - Strings, structures, cell arrays, symbolic math (ch. 11,12)
Week 14 – Exam 3
References: H. Moore, MATLAB for Engineers, 4/e, Prentice Hall, 2014
G. Recktenwald, Numerical Methods with MATLAB, Prentice Hall, 2000
A. Gilat, MATLAB, An Introduction with Applications, 4/e, Wiley, 2011

3. Matrix Algebra dot product matrix-vector multiplication
matrix-matrix multiplication
matrix inverse
solving linear systems
least-squares solutions
determinant, rank, condition number
vector & matrix norms
iterative solutions of linear systems
examples
electric circuits
temperature distributions

4. Operators and Expressions
operation element-wise matrix-wise
addition
subtraction
multiplication * *
division / /
left division \ \
exponentiation ^ ^
transpose w/o complex conjugation .'
transpose with complex conjugation '
>> help /
>> help precedence
used in matrix
algebra operations

5. >> [A, A.^2; A^2, A*A] % form sub-blocks ans = 1 2 1 4 3 4 9 16
>> [A, A.^2; A^2, A*A] % form sub-blocks
ans =
% note A^2 = A*A
>> B = 10.^A;
>> [B, log10(B)]

6. >> size(A) % [N,M] = size(A), NxM matrix
matrix indexing
convention
>> A = [1 2 3; 2 0 4; 0 8 5]
A =
>> size(A) % [N,M] = size(A), NxM matrix
ans =

7. dot product The dot product is the basic operation in
matrix-vector and matrix-matrix multiplications
dot product
a, b must have the same
dimension
math
notations
MATLAB
notation

8. for complex-valued vectors
dot product
for complex-valued vectors
complex-conjugate transpose,
or, hermitian conjugate of a
for real-valued vectors, the
operations ' and .'
are equivalent
math
notations
MATLAB
notation

9. >> a = [1; 2; -3]; b = [4; -5; 2];
ans =
-12
>> dot(a,b) % built-in function
ans = % same as sum(a.*b)

10. matrix-vector multiplication
combine three dot product
operations into a single
matrix-vector multiplication

11. A x = b matrix-vector multiplication combine three dot product
operations into a single
matrix-vector multiplication
A x = b

12. matrix-matrix multiplication
combine three matrix-vector multiplications into a single matrix-matrix multiplication

13. >> A = [4 1 2; ; 2 1 1]
A =
>> B = [ ; ; ]
B =
>> C = A*B
C =

14. C(i,j) is the dot product of i-th row of A with j-th column of B
note:
A*B ¹ B*A

15. (NxK)x(KxM) --> NxM
Rule of thumb:
(NxK)x(KxM) --> NxM
A is NxK
B is KxM
then, A*B is NxM

16. vector-vector multiplication
(1x3)x(3x1) --> 1x1 = scalar
row * column = scalar
(3x1)x(1x3) --> 3x3
column * row = matrix

17. vector-vector multiplication
>> [1, 2, 3] * [ ]'
ans = -7
>> [1, 2, 3]' * [ ]
row x column
= scalar
column x row
= matrix

18. solving linear systems
A x = b
solving linear systems
Linear equations have a very large number of applications
in engineering, science, social sciences, and economics
Linear Programming – Management Science
Computer Aided Design – aerodynamics of cars, planes
Signal Processing, Communications, Control, Radar,
Sonar, Electromagnetics, Oil Exploration,
Computer Vision, Pattern & Face Recognition
Chip Design – millions of transistors on a chip
Economic Models, Finance, Statistical Models,
Data Mining, Social Models, Financial Engineering
Markov Models – Speech, Biology, Google Pagerank
Scientific Computing – solving very large problems
the only practical way to solve very large systems is iteratively

19. solving linear systems
matrix
inverse
always use the backslash operator to
solve a linear system, instead of inv(A)

20. solving linear systems (using backslash)
>> x = A\b % solution of A*x = b
x = % x = A^-1 * b
% x = inv(A) * b 2 3
>> norm(A*x-b) % test - should be zero
ans = % or, of the order of eps

21. solving linear systems (using inv)
>> b = [4 8 9]';
>> inv(A) % same as A^(-1)
ans =
>> x = inv(A) * b % but prefer backslash
x = % same as x = A^-1 * b
1.0000
2.0000
3.0000
>> norm(A*x-b)
1.8310e-015
>> inv(sym(A))
ans =
[ 18/31, -4/31, -1/31]
[ -5/31, 8/31, 2/31]
[ -7/31, 5/31, 9/31]

22. A of size NxN and invertible X of size NxK B of size NxK
solving linear systems – back-slash and forward-slash
A of size NxN and invertible
X of size NxK
B of size NxK
AX = B --> X = A\B = inv(A)*B
equivalent
A of size NxN and invertible
X of size KxN
B of size KxN
XA = B --> X = B/A = B*inv(A)
equivalent
A X = B X A = B

23. x may or may not be unique
solving linear systems – least-squares solutions
A of size NxM
x of size Mx1 column
b of size Nx1 column
x = A\b
will be used in
in weeks 11 & 12
x=A\b is a solution of Ax=b
in a least-squares sense,
i.e., x minimizes the norm squared
of the error e = b – A*x:
(b-Ax)'*(b-Ax) = min
x may or may not be unique
depending on whether the linear
system Ax=b is over-determined,
under-determined, or whether A has
full rank or not
pseudo-inverse
x = pinv(A)*b;
>> help \
>> help pinv

24. A = NxM matrix A' = MxN matrix x = Mx1 column A'*A = MxM matrix
Fundamental Theorem of
Linear Algebra – what is it?
least-squares solutions - summary
A = NxM matrix A' = MxN matrix
x = Mx1 column A'*A = MxM matrix
b = Nx1 column A'*b = Mx1 column
Assuming full rank for A, we have the following cases:
1. N>M, overdetermined case, (most common in practice)
x = A\b = unique least-squares solution, same as
x = pinv(A)*b, and
x = (A'*A)^(-1) * (A'*b)
2. N<M, underdetermined case, (there are many solutions)
x=A\b, x=pinv(A)*b, are two possible solutions
3. N=M, square invertible case, x is unique
x = A\b is equivalent to x = A^(-1)*b
x = A\b is numerically
the most accurate method

25. least-squares solutions - example
% overdetermined
% full-rank example
A = [1 2; 3 4; 5 6]
b = [4, 3, 8]';
x = A\b
% x = pinv(A)*b
% x = (A'*A)\(A'*b)
x = -1 2

26. least-squares solutions - example
inverse exists because A was
assumed to have full rank
minimized value of J
achieved at x = x_opt
strictly positive-definite quadratic
form because of full rank of A,
vanishes only at x = x_opt

27. least-squares solutions - example
b = [4, 3, 8]';
x_opt = (A'*A)\(A'*b)
J_min = b'*b - ...
b'*A*inv(A'*A)*A'*b
x_opt = -1 2
J_min = 6

28. least-squares solutions - example

29. least-squares solutions - example
% we can also minimize J with fminsearch,
% i.e., the multivariable version of fminbnd
J 35*(x(1)+1).^2 +...
88*(x(1)+1).*(x(2)-2)+...
56*(x(2)-2).^2 + 6;
x0 = [0,0]'; % arbitrary initial search point
[xmin,Jmin] = fminsearch(J,x0)
% xmin = % Jmin = 6 % %

30. Invertibility, rank, determinants, condition number
The inverse inv(A) of an NxN square matrix A exists if its determinant is
non-zero, or, equivalently if it has full rank, i.e., when its rank is equal to the row or column dimension N
>> doc inv
>> doc det
>> doc rank
>> doc cond
a = [1 2 3]'; b = [4 5 6]';
A = [a, a+b, b]
A =
>> det(A)
ans =
>> rank(A) 2
det(A) = 0

31. Invertibility, rank, determinants, condition number
The larger the cond(A) the more ill-conditioned the linear system, and the less reliable the solution.
A = [1, 5,
2, e-8, 5
3, 9, ];
>> cond(A)
ans =
3.3227e+009
A\[1; 2; 3]
ans = 1
A\[1.001; ; ]
ans =
det(A) = e-008

32. Determinant and inverse of a 2x2 matrix
Example:

33. Matrix Exponential Used widely in solving linear dynamic systems
>> expm(A) % matrix exponential
ans =
>> exp(A) % element-wise exponential
>> doc expm
>> doc exp

34. L1, L2, and L¥ norms of a vector
Vector & Matrix Norms
>> doc norm
L1, L2, and L¥ norms of a vector
used as distance measure between two vectors or matrices
L1 norm
Euclidean, L2 norm
L¥ norm

35. useful for comparing two vectors or matrices
x = [1, -4, 5, 3]; p = inf;
switch p
case 1
N = sum(abs(x)); % N = norm(x,1);
case 2
N = sqrt(sum(abs(x).^2)); % N = norm(x,2);
case inf
N = max(abs(x)); % N = norm(x,inf);
otherwise
end
useful for comparing two vectors or matrices
>> norm(a-b) % a,b vectors of same size
>> norm(A-B) % A,B matrices of same size
equivalent calculation using
the built-in function norm :

36. Euclidean distance
dot product

37. Electric Circuits
Kirchhoff’s
Voltage Law

38. Electric Circuits

39. A x = b

40. A = [25, -15, -10; -15, 30, -15; -10, -15, 30]
b = [-7.5; 15; -5]
A =
b =
x = A\b
x =
0.5000
1.0000

41. inv(A)
ans =
inv(sym(A)) --> (1/105) * [ ]

42. Iterative solutions of linear systems Ax=b
the only practical way to solve very large linear systems is iteratively
Methods:
Jacobi method
Gauss-Seidel method
Relaxation methods
Conjugate Gradient method
Others
G. H. Golub and C. F. Van Loan, Matrix Computations, 3/e, JHU Press, 1996.
D. S. Watkins, Fundamentals of Matrix Computations, 2/e, Wiley, 2002.
L. N. Trefethen and D. Bau, Numerical Linear Algebra, SIAM, 1997.
A. Bjork, Numerical Methods for Least Squares Problems, SIAM, 1996.

43. scalar example illustrating the Jacobi method rearrange rearrange
turn it into a recursion

44. x(1)=0; % arbitrary for k=1:19 x(k+1) = -0.5*x(k) + 6; end
x(1)=0; % arbitrary
for k=1:19
x(k+1) = -0.5*x(k) + 6;
end
k=1:20; plot(k,x,'b.-');

45. if abs(xnew-x)<=tol break; end x = xnew; k = k+1;
tol=1e-10; x0=0;
x=x0; k=1;
while 1
xnew = -0.5*x + 6;
if abs(xnew-x)<=tol
break;
end
x = xnew;
k = k+1;
k, abs(x-4)
k = 37
ans =
5.8208e-011
tol=1e-10; x0=0;
x=x0; k=1;
xnew = -0.5*x+6;
while abs(xnew-x)>tol
x = xnew;
k = k+1;
xnew = -0.5*x + 6;
end
k, abs(x-4)
k = 37
ans =
5.8208e-011
forever
while loop
conventional
while loop

46. More general version of ax=b, where a,x,b are scalars
choose a splitting a = d – r, such that | r/d | < 1
iterative algorithm:
it has solution that converges to b/a:
key assumption that guarantees convergence
this is the scalar version of the
Jacobi and Gauss-Seidel methods

47. Jacobi’s method for A x = b
Define the following ‘Jacobi iteration parameters’
Jacobi iteration:

48. Convergence requires the condition that the so-called ‘spectral radius’ of B be strictly less than unity:
This is hard to calculate for large matrices,
but a sufficient condition for convergence is that
a matrix norm of B, such as the L1, L2, L¥, or Frobenius norms, be less than unity because of the inequality:
Note: if the method fails for
a system A,b, then, try it on
the rearranged system
A --> flipud(A)
b --> flipud(b)
which sometimes works
so that

49. Jacobi iteration:
Exact solution demonstrates the convergence:

50. Jacobi iteration – alternative form:
Jacobi iteration – relaxation form:
relaxation parameter

51. MATLAB implementation
% given A = NxN matrix, b = Nx1 vector
% construct the Jacobi parameters
I = eye(size(A)); % identity matrix
D = diag(diag(A)); % diagonal part of A
B = I - D\A; % iteration matrix
c = D\b; % Nx1 vector
rho = max(abs(eig(B))); % check if rho < 1
Next, implement the Jacobi iteration using a forever while-loop with a stopping condition,

52. MATLAB implementation – v.1
tol = 1e-10; % error tolerance, our choice
x = x0; k = 1; % x0 = arbitrary Nx1 vector
while % forever loop
xnew = B*x + c;
if norm(xnew-x) < tol
break;
end
x = xnew;
k = k+1;
k, x, norm(A*x-b) % print & verify solution
norm() measures the
difference between the
vectors xnew and x,
could also use L¥ norm
max(abs(xnew-x))
Note that it breaks out of the loop just before the tolerance is reached, one more iteration after the loop would meet or exceed the tolerance.

53. MATLAB implementation – v.2
% in order to save individual iterates x(k),
% use a matrix X whose columns are x(k)
tol = 1e-10; % error tolerance
X(:,1)=x0; k=1; % x0 = arbitrary Nx1 vector
while 1
X(:,k+1) = B*X(:,k) + c;
if norm(X(:,k+1)-X(:,k) ) < tol
break;
end
k = k+1;
k, x = X(:,k) % x = converged solution
norm(A*x-b) % verify solution

54. MATLAB implementation – v.3
% another version that saves the iterates
tol=1e-10; % error tolerance
x=x0; k=1; % x0 = arbitrary Nx1 vector
X=x; % X columns are the iterates x(k)
while 1
xnew = B*x + c;
if norm(xnew-x) < tol
break;
end
x = xnew;
k = k+1;
X = [X,x]; % append new column to X
k, x, norm(A*x-b) % print & verify solution

55. Example A = [2 1 0; 1 5 -1; 1 -2 4]; b = [4 8 9]'; xsol = A\b;
I = eye(size(A)); % 3x3 identity matrix
D = diag(diag(A)); % diagonal part of A
B = I - D\A; % 3x3 iteration matrix
c = D\b; % 3x1 vector
rho = max(abs(eig(B)));

56. >> D, B >> c, rho, norm(B,inf)
B =
c =
2.00
1.60
2.25
rho =
0.5000
ans =
0.7500

57. tol=1e-12; % error tolerance
x=[0 1 2]'; k=1; % x0 = [0 1 2]' = arbitrary
X=x; % X = save all iterates
while 1
xnew = B*x + c;
if norm(xnew-x)<tol % norm() measures
break; % the distance
end % between x,xnew
x = xnew;
k = k+1;
X = [X,x]; % append new column
end
k, x, norm(A*x-b), norm(x-xsol), size(X)

58. >> k, x, norm(A*x-b), norm(x-xsol), size(X)40
x =
1.0000
2.0000
3.0000
ans =
2.6657e-012
1.3634e-012
3 40
>> K=1:k;
>> plot(K,X');

59. Temperature Distribution
follows from discretizing
the Laplace equation

60. A x = b

61. A x = b >> x = A\b x = >> A = [ 4 -1 -1 0 20.0000
>> A = [ ];
>> b = [30; 60; 40; 70];

62. display solution T in a rectangular pattern
nodes were numbered in
column order
T = zeros(2,2); % shape of T
T(:) = x
T =

63. A x = b Rules for constructing A and b: main diagonal is 4
if nodes i,j are connected, then, -1, otherwise, 0
3) b(i) is sum of boundary values connected to node i

64. Iterative Solution convenient for large number of subdivisions
used also to solve 2D
electrostatics problems

65. iterate over internal nodes only
N=4; M=4;
left=20; right=30; up=10; dn=40;
T(1,:) = repmat(up,1,M); T(N,:) = repmat(dn,1,M);
T(:,1) = repmat(left,N,1); T(:,M) = repmat(right,N,1);
Tnew = T;
tol = 1e-4; K = 100;
for k=1:K,
for i=2:N-1,
for j=2:M-1,
Tnew(i,j) = (T(i-1,j) + T(i+1,j) +...
T(i,j-1) + T(i,j+1))/4;
end
if norm(Tnew-T) < tol, break; end
T = Tnew;
T(1,[1,end]) = nan; T(end,[1,end]) = nan;
boundary values
iterate over internal nodes only

66. T = % start-up
NaN NaN
NaN NaN
% converged after k = 19 iterations
% to within the specified tol = 1e-4
T =
NaN NaN
NaN NaN

67. T = % after k=1 iteration
NaN NaN
NaN NaN
T = % after k=2 iterations
T = % after k=3 iterations

68. N=30; M=30;
left=0; right=0; up=0; dn=60;
tol = 1e-6; K = 5000;
% breaks out at k = 2475
[X,Y] = meshgrid(2:M-1, 2:N-1);
Z = T(2:M-1, 2:N-1);
surf(X,Y,Z);