B.1 Matrix Calculus

Getting the chain rule right means making sure that the dimensions of the vectors and matrices generated by taking derivatives line up properly, which motivates the rule:

1. 2.
   ​

   In the elements generated by the chain rule, all the numerators on the RHS must have the same orientation as the numerator on the LHS, and likewise for the denominators.

Rules 1 and 2, along with the requirement that inner matrix dimensions agree, ensure that the chain rule for a row-vector derivative is:

This chain rule works just as well if $z$ or $y$ are scalars:

We could write down the column-vector version by applying rule 2 while ensuring agreement between the inner matrix dimensions. Alternatively, we can apply rule 1 to the chain rule just derived for the row-vector derivative:

This is perhaps the less intuitive of the two chain rules, since it reverses the order in which the factors are usually written in scalar calculus.

This motivates no additional matrix-calculus rules, but maintaining agreement among inner matrix dimensions does enforce a particular order. For example, let $y=\bm{u}(\bm{x})\cdot\bm{v}(\bm{x})$, the dot product of two vector-valued functions. Then the product rule must read:

The column-vector equivalent is easily derived by transposing the RHS. Neither, unfortunately, can be read as “the derivative of the first times the second, plus the first times the derivative of the second,” as it is often taught in scalar calculus. It is easily remembered, nevertheless, by applying our rules 1 and 2, and checking inner matrix dimensions for agreement.

In the special case of a quadratic form, $y=\bm{x}^{\text{T}}\mathbf{A}\bm{x}$, this reduces to:

In the even more special case where $A$ is symmetric, $\mathbf{A}=\mathbf{A}^{\text{T}}$, this yields $2\bm{x}^{\text{T}}\mathbf{A}$. Evidently, the column-vector equivalent is $2\mathbf{A}\bm{x}$.

Given a matrix,

and a scalar-valued function $y$, we define:

This definition can be more easily applied if we translate it into the derivatives with respect to vectors introduced in the previous section. Giving names to the rows ($\bm{\bar{x}}_{i}^{\text{T}}$) and columns ($\bm{x}_{i}$) of $\mathbf{X}$:

we can write:

This lets us more easily derive some common special cases. Consider the bilinear form $y=\bm{a}^{\text{T}}\mathbf{X}\bm{b}$. The derivative with respect to the first row of $\mathbf{X}$ is:

Stacking all $m$ of these rows vertically as in Eq. B.1, we see that:

Alternatively, we might have used the column-gradient formulation:

and then stacked these columns horizontally as in Eq. B.1:

Or again, consider a case where $\mathbf{X}$ shows up in the other part of the bilinear form (in this case, a quadratic form):

Then defining $\bm{z}\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\mathbf{X}\bm{a}+\bm{b}$, and considering again just the first row of $\mathbf{X}$, we find:

where $\bm{v}\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=(\mathbf{W}+\mathbf{W}^{\text{T}})\bm{z}$, and $v_{1}$ is its first element. Stacking these rows vertically, as in Eq. B.1, yields:

A common application of this derivative occurs when working with Gaussian functions, which can be written $e^{-y/2}$ for the $y$ defined in Eq. B.2. In this case, the matrix $\mathbf{W}$ is symmetric, and the result simplifies further. More generally, Eq. B.2 occurs in quadratic penalties on the state in control problems, in which case $X$ would be the state-transition matrix.

The derivative of a matrix-valued function with respect to a matrix is a tensor. These are cumbersome, so in a way our discussion of them is merely preliminary to what follows. Let $x_{ij}$ be the $(i,j)^{\text{th}}$ entry of $X$. We consider a few simple matrix functions of $X$:

where $\bm{a}_{i}$ is the $i^{\text{th}}$ column of $\mathbf{A}$. Transposes and derivatives commute, as usual, so the derivative of $\mathbf{X}^{\text{T}}\mathbf{A}^{\text{T}}$ (e.g.) is just the transpose of the above. That means that

with $\bm{\tilde{a}}_{j}^{\text{T}}$ the $j^{\text{th}}$ row of $A$. From the first we can also compute the slightly more complicated, but elegant:

with $\bm{b}_{j}$ the $j^{\text{th}}$ column of $\mathbf{B}$. And for a square matrix $\mathbf{A}$, we consider the even more complicated:

Finally, consider the vector-valued function $\bm{f}(\mathbf{X}\bm{a}+\bm{b})$, where $\bm{f}$ has Jacobian $\mathbf{J}$ but is otherwise unspecified. Its derivative with respect to an element of $\mathbf{X}$ is

For all of the above, the derivative with respect to the entire matrix $\mathbf{X}$ is just the collection of these matrices for all $i$ and $j$.

Now suppose we want to take a derivative with respect to $\mathbf{X}$ of the scalar-valued function $y(\mathbf{F}(\mathbf{X}))$, for various matrix-valued functions $\mathbf{F}(\cdot)$. We shall consider in particular those lately worked out. The chain rule here says that the $(i,j)^{\text{th}}$ element of this matrix is:

with $\bm{1}$ a vector of ones and $\circ$ the entry-wise (Hadamard) product. We now apply this equation to the results above:

For the vector-valued function $\bm{f}(\mathbf{X}\bm{a}+\bm{b})=\mathrel{\vbox{\hbox{\scriptsize.}\hbox{$ considered above,

A few special cases are interesting. Since the trace and the derivative are linear operators, they commute, and in particular

Therefore, letting $y(\mathbf{X})\stackrel{{\scriptstyle\text{set}}}{{=}}\text{tr}\mathopen{}$ in the above equations, we have

The trace and determinant of a matrix are related by a useful formula, derived through their respective relationships with the matrix’s spectrum. Recall:

For each eigenvalue $\lambda$ and associated eigenvector $\bm{v}$, $\mathbf{M}\bm{v}=\lambda\bm{v}$, so:

Therefore, the eigenvectors of $\exp\{\mathbf{M}\}$ are the eigenvectors $\mathbf{M}$, and the eigenvalues of $\exp\{\mathbf{M}\}$ are the exponentiated eigenvalues of $\mathbf{M}$. Hence:

Therefore:

So far we have made use only of results from scalar calculus. (The derivative of the log of $\mathbf{A}(x)$ can be derived easily in terms of the Maclaurin series for the natural logarithm.)

Cf. the “trace trick,” in e.g. IPGM. When $f=\log|\mathbf{A}|$ (i.e. the log of the determinant of $\mathbf{A}$),

i.e. the inverse transpose. (This can be derived from the “interesting scalar case” below.) From this it follows easily that