1. Matrix Differential Calculus By Dr. Md. Nurul Haque Mollah, Professor, Dept. of Statistics, University of Rajshahi, Bangladesh 01-10-111Dr. M. N. H. MOLLAH

2. Outline  Differentiable Functions  Classification of Functions and Variables for Derivatives  Derivatives of Scalar Functions w. r. to Vector Variable  Derivative of Scalar Functions w. r. to a Matrix Variable  Derivatives of Vector Function w. r. to a Scalar Variable  Derivatives of Vector Function w. r. to a Vector Variable  Derivatives of Vector Function w. r. to a Matrix Variable  Derivatives of Matrix Function w. r. to a Scalar Variable  Derivatives of Matrix Function w. r. to a Vector Variable  Derivatives of Matrix Function w. r. to a Matrix Variable  Some Applications of Matrix Differential Calcul us 01-10-112Dr. M. N. H. MOLLAH

3. 1. Differentiable Functions  A real-valued function where is an open set is said to be continuously differentiable if the partial derivatives exist for each and are continuous functions of x over X. In this case we write over X.  Generally, we write over X if all partial derivatives of order p exist and are continuous as functions of x over X.  If over R n, we simply write  If on X, the gradient of at a point is defined as Dr. M. N. H. MOLLAH01-10-11 3

4.  If over X, the Hessian of at x is defined to be the symmetric matrix having as the ijth element  If where then is represented by the column vector of its component functions as  If X is open, we can write on X if on X. Then the derivative of the vector function with respect to the vector variable x is defined by Dr. M. N. H. MOLLAH01-10-114

5.  If is real-valued functions of (x,y) where we write Dr. M. N. H. MOLLAH01-10-115

6.  If then  For and consider the function defined by Then if the chain rule of differentiation is stated as Dr. M. N. H. MOLLAH01-10-116

7. 2. Classification of Functions and Variables for Derivatives Let us consider scalar functions g, vector functions ƒ and matrix functions F. Each of these may depend on one real variable x, a vector of real variables x, or a matrix of real variables X. We thus obtain the classification of function and variables shown in the following Table. TableScalar Variable Vector variable Matrix variable Scalar function Vector function Matrix function Dr. M. N. H. MOLLAH01-10-117

8. Some Examples of Scalar, Vector and Matrix Functions Dr. M. N. H. MOLLAH01-10-118

9. Consider a scalar valued function ‘g’ of ‘m’ variables g = g(x 1, x 2,…, x m ) = g(x), where x = (x 1, x 2,…, x m ) /. Assuming function g is differentiable, then its vector gradient with respect to x is the m-dimensional column vector of partial derivatives as follows 3. Derivatives of Scalar Functions w. r. to Vector Variable 3.1 Definition Dr. M. N. H. MOLLAH01-10-119

10. Consider the simple linear functional of x as where is a constant vector. Then the gradient of g is w. r. to x is given by Also we can write it as Because the gradient is constant (independent of x), the Hessian matrix of is zero. 3.2 Example 1 Dr. M. N. H. MOLLAH01-10-1110

11. Example 2 Consider the quadratic form where A=(a ij ) is a m x m square matrix. Then the gradient of g(x) w.r. to x is given by Dr. M. N. H. MOLLAH01-10-1111

12. Then the second- order gradient or Hessian matrix of g(x)=x / Ax w. r. to x becomes Dr. M. N. H. MOLLAH01-10-1112

13. For computing the gradients of products and quotients of functions, as well as of composite functions, the same rules apply as for ordinary functions of one variable. Thus The gradient of the composite function f(g(x)) can be generalized to any number of nested functions, giving the same chain rule of differentiation that is valid for functions if one variable. 3.3 Some useful rules for derivative of scalar functions w. r. to vectors Dr. M. N. H. MOLLAH01-10-1113

14. 3.3 Fundamental Rules for Matrix Differential Calculus Dr. M. N. H. MOLLAH01-10-1114

15. 3.4 Some useful derivatives of scalar functions w. r. to a vector variable Dr. M. N. H. MOLLAH01-10-1115

16. Consider a scalar-valued function f of the elements of a matrix X=(x ij ) as f = f(X) = f(x 11, x 12,… x ij,..., x mn ) Assuming that function f is differentiable, then its matrix gradient with respect to X is the m×n matrix of partial derivatives as follows 4. Derivative of Scalar Functions w. r. to a Matrix Variable 4.1 Definition Dr. M. N. H. MOLLAH01-10-1116

17. The trace of a matrix is a scalar function of the matrix elements. Let X=(x ij ) is an m x m square matrix whose trace is denoted by tr (X). Then Proof: The trace of X is defined by Taking the partial derivatives of tr (X) with respect to one of the elements, say x ij, gives 4.2 Example 1 Dr. M. N. H. MOLLAH01-10-1117

18. Thus we get, Dr. M. N. H. MOLLAH01-10-1118

19. The determinant of a matrix is a scalar function of the matrix elements. Let X=(x ij ) is an m x m invertible square matrix whose determinant is denoted |X|. Then Proof: The inverse of a matrix X is obtained as where adj(X) is known as the adjoint matrix of X. It is defined by where C ij =(-1) i+j M ij is the cofactor w. r. to x ij and M ij is the minor w. r. to x ij. 4.2 Example 2 Dr. M. N. H. MOLLAH01-10-1119

20. The minor M ij is obtained by first taking the (n-1) x (n-1) sub-matrix of X that remains when the i-th row and j-th column of X are removed, then computing the determinant of this sub-matrix. Thus the determinant |X| can be expressed in terms of the cofactors as follows Row i can be any row and the result is always the same. In the cofactors C ik none of the matrix elements of the i-th row appear, so the determinant is a linear function of these elements. Taking now a partial derivatives of |X| with respect to one of the elements, say x ij, gives Dr. M. N. H. MOLLAH01-10-1120

21. Thus we get, This also implies that Dr. M. N. H. MOLLAH01-10-1121

22. 4.3 Some useful derivatives of scalar functions w.r.to matrix Dr. M. N. H. MOLLAH01-10-1122

23. Derivatives of trace w.r.to matrix Dr. M. N. H. MOLLAH01-10-1123

24. Dr. M. N. H. MOLLAH01-10-1124

25. Derivatives of determinants w.r.to matrix Dr. M. N. H. MOLLAH01-10-1125

26. Consider the vector valued function ‘f’ of a scalar variable x as f(x)=[f 1 (x), f 2 (x),…, f n (x) ] / Assuming function f is differentiable, then its scalar gradient with respect to x is the n-dimensional row vector of partial derivatives as follows 5. Derivatives of Vector Function w. r. to a Scalar Variable 5.1 Definition Dr. M. N. H. MOLLAH01-10-1126

27. Let Then the gradient of f with respect to x is given by Also we can write it as 5.2 Example Dr. M. N. H. MOLLAH01-10-1127

28. Consider the vector valued function ‘f’ of a vector variable x=(x 1, x 2, …, x m ) / as f(x)= y =[y 1 = f 1 (x), y 2 = f 2 (x),…, y n = f n (x) ] / Assuming function f is differentiable, then its vector gradient with respect to x is the m×n matrix of partial derivatives as follows 6. Derivatives of Vector Function w. r. to a Vector Variable 6.1 Definition Dr. M. N. H. MOLLAH01-10-1128

29. Let Then the gradient of f (x) with respect to x is given by 6.2 Example Dr. M. N. H. MOLLAH01-10-1129

30. Consider the vector valued function ‘f’ of a matrix variable X=(x ij ) of order m×n as f(X)= y =[y 1 = f 1 (X), y 2 = f 2 (X),…, y q = f q (X) ] / Assuming that function f is differentiable, then its matrix gradient with respect to X is the mn×q matrix of partial derivatives as follows 7. Derivatives of Vector Function w. r. to a Matrix Variable 7.1 Definition Dr. M. N. H. MOLLAH01-10-1130

31. Let Then the gradient of f (X) w. r. to matrix variable X is given by 7.2 Example Dr. M. N. H. MOLLAH01-10-1131

32. Consider the matrix valued function ‘F’ of a scalar variable x as F(x)= Y =[y ij = f ij (x)] m×n Assuming that function F is differentiable, then its scalar gradient with respect to the scalar x is the m×n order matrix of partial derivatives as follows 8. Derivatives of Matrix Function w. r. to a Scalar Variable 8.1 Definition Dr. M. N. H. MOLLAH01-10-1132

33. Let Then the gradient of F (x) w. r. to scalar variable x is given by 8.2 Example Dr. M. N. H. MOLLAH01-10-1133

34. Consider the matrix valued function ‘F’ of a vector variable x=(x 1,x 2,…,x m ) as F(x)= Y =[y ij = f ij (x)] n×q Assuming that function F is differentiable, then its vector gradient with respect to the vector x is the m×nq order matrix of partial derivatives as follows 9. Derivatives of Matrix Function w. r. to a Vector Variable 9.1 Definition Dr. M. N. H. MOLLAH01-10-1134

35. Let Then the gradient of F (x) w. r. to scalar variable x is given by 9.2 Example Dr. M. N. H. MOLLAH01-10-1135

36. Consider the matrix valued function ‘F’ of a matrix variable X=(x ij ) m×p as F(X)= Y =[y ij = f ij (X)] n×q Assuming that function F is differentiable, then its matrix gradient with respect to the matrix X is the mp×nq order matrix of partial derivatives as follows 10.Derivatives of Matrix Function w. r. to a Matrix Variable 10.1 Definition Dr. M. N. H. MOLLAH01-10-1136

37. Let Then the gradient of F (X) w. r. to scalar variable X is given by 10.2 Example Dr. M. N. H. MOLLAH01-10-1137

38. Dr. M. N. H. MOLLAH01-10-1138

39. Some important rules for matrix differentiation Dr. M. N. H. MOLLAH01-10-1139

40. Homework's Dr. M. N. H. MOLLAH01-10-1140

41. 11. Some Applications of Matrix Differential Calculus 1. Test of independence between functions 2. Expansion of Tailor series 3. Transformations of Multivariate Density functions 4. Multiple integrations 5. And so on. Dr. M. N. H. MOLLAH01-10-1141

42. Test of Independence A set of functions are said to be correlated of each other if their Jacobian is zero. That is Example: Show that the functions are not independent of one another. Show that Proof: So the functions are not independent. Dr. M. N. H. MOLLAH01-10-1142

43. In deriving some of the gradient type learning algorithms, we have to resort to Taylor series expansion of a function g(x) of a scalar variable x, (3.19) We can do a similar expansion for a function g(x)=g(x 1, x 2,…, x m ) of m variables. We have (3.20) Where the derivatives are evaluated at the point x. The second term is the inner product of the gradient vector with the vector x-x, and the third term is a quadratic form with the symmetric Hassian matrix (∂2g / ∂x 2 ).The truncation error depends on the distance |x-x|; the distance has to be small, if g(x)is approximated using only the first and second-order terms. Taylor series expansions of multivariate functions Dr. M. N. H. MOLLAH01-10-1143

44. The same expansion can be made for a scalar function of a matrix variable. The second order term already becomes complicated because the second order gradient is a four- dimension tensor. But we can easily extend the first order term in (3.20), the inner product of the gradient with the vector x-x to the matrix case. Remember that the vector inner product is define as For the matrix case, this must become the sum. Taylor series expansions of multivariate functions Dr. M. N. H. MOLLAH01-10-1144

45. This is the sum of the products of corresponding elements, just like in the vectorial inner product. This can be nicely presented in matrix form when we remember that for any two matrices, say A and B. With obvious notation. So, we have (3.21) for the first two terms in the Taylor series of a function g of a matrix variable. Taylor series expansions of multivariate functions Dr. M. N. H. MOLLAH01-10-1145