Matrix Differentiation

Abstract

It is natural and convenient to use matrix notation and terminology in defining and discussing certain functions of one or more variables. Earlier (in Chapter 14), matrix notation and terminology were used in defining and discussing linear, bilinear, and quadratic forms. In some cases, the use of matrix notation and terminology is essentially unavoidable—consider, for example, a case where the determinant of an m × m matrix is regarded as a function of its m² elements.

Matrix differentiation is the derivation of the first-, second-, or higher-order partial derivatives of a function or functions that have been expressed in terms of matrices. In deriving, presenting, and discussing the partial derivatives of such functions, it is natural, convenient, and in some cases necessary to employ matrix notation and terminology. Not only may the functions be expressed in terms of matrices, but the functions may comprise the elements of a vector or matrix, as would be the case if the elements of the inverse of an m×m (nonsingular) matrix A were regarded as functions of the elements of A.

Matrix differentiation is of considerable importance in statistics. It is especially useful in connection with the maximum likelihood estimation of the parameters in a statistical model. The maximum likelihood estimates of the model’s parameters satisfy the equations (known as the likelihood equations) obtained by equating to zero the first-order partial derivatives (with respect to the model’s parameters) of the logarithm of the so-called likelihood function—in many important cases, the likelihood function involves the determinant and/or inverse of a matrix. Further, an approximation (suitable for large samples) to the variance-covariance matrix of the maximum likelihood estimators can be obtained by inverting the matrix (known as the information matrix) whose ij th element is–1 times the second-order partial derivative (with respect to the i th and j th parameters) of the logarithm of the likelihood function.