Signature
linkedinemailgithubgoogleScholar

Vector and Matrix Derivatives

On this page

Vector and Matrix Derivatives: Essential Tools for Optimization

Before differentiating an objective function, we identify which vector or matrix can vary and which quantities stay fixed. This reference first defines the gradient convention, then groups formulas by the variable we differentiate. Each formula applies only when its dimensions and domain conditions are satisfied.

Dimensions and Notation

For the vector and trace identities below, we use real column vectors and square matrices:

a, b, x∈Rn,A,B,X∈Rn×n.\mathbf{a},\,\mathbf{b},\,\mathbf{x} \in \mathbb{R}^{n}, \qquad A,B,X \in \mathbb{R}^{n\times n}.

For a function that returns a scalar, the gradient has the same dimensions as the variable we differentiate. We use df=(∇af)⊤dadf=(\nabla_{\mathbf{a}}f)^\top d\mathbf{a} for a vector and df=Tr⁡((∇Af)⊤dA)df=\operatorname{Tr}((\nabla_A f)^\top dA) for a matrix. In the matrix case, we treat the entries as independent coordinates. When a function returns a vector, we instead use its Jacobian: rows correspond to output coordinates and columns to input coordinates. All other quantities stay fixed during differentiation.

Vector Derivatives

In the first group, the vector is the variable and the other factors are fixed. Inner-product derivatives give the linear terms; quadratic expressions also require accounting for both occurrences of the vector.

∂a⊤b∂a=b,\frac{\partial \mathbf{a}^\top \mathbf{b}}{\partial \mathbf{a}} = \mathbf{b},
∂b⊤a∂a=b,\frac{\partial \mathbf{b}^\top \mathbf{a}}{\partial \mathbf{a}} = \mathbf{b},
∂a⊤Aa∂a=(A+A⊤) a,\frac{\partial \mathbf{a}^\top A \mathbf{a}}{\partial \mathbf{a}} = (A + A^\top)\,\mathbf{a},
∂(A a)∂a=A,\frac{\partial (A\,\mathbf{a})}{\partial \mathbf{a}} = A,

Matrix Derivatives

We now let the matrix vary. The gradient must have the same dimensions as that matrix; this requirement helps check the factor order and transposes in the formulas below.

∂a⊤Ab∂A=a b⊤,\frac{\partial \mathbf{a}^\top A \mathbf{b}}{\partial A} = \mathbf{a}\,\mathbf{b}^\top,
∂a⊤A⊤b∂A=b a⊤,\frac{\partial \mathbf{a}^\top A^\top \mathbf{b}}{\partial A} = \mathbf{b}\,\mathbf{a}^\top,
∂a⊤A⊤A a∂A=2A a a⊤,\frac{\partial \mathbf{a}^\top A^\top A\,\mathbf{a}}{\partial A} = 2A\,\mathbf{a}\,\mathbf{a}^\top,
∂Tr⁡(A)∂A=I,\frac{\partial \operatorname{Tr}(A)}{\partial A} = I,
∂Tr⁡(A⊤X)∂X=∂Tr⁡(X⊤A)∂X=A,\frac{\partial \operatorname{Tr}(A^\top X)}{\partial X} = \frac{\partial \operatorname{Tr}(X^\top A)}{\partial X} = A,
∂Tr⁡(AXB)∂X=A⊤B⊤,\frac{\partial \operatorname{Tr}(A X B)}{\partial X} = A^\top B^\top,
∂Tr⁡(B⊤X⊤A)∂X=A B⊤.\frac{\partial \operatorname{Tr}(B^\top X^\top A)}{\partial X} = A\,B^\top.

L2L^2 Norm Derivatives

Squared-error objectives lead to the following derivatives. For matrix residuals, we use the Frobenius norm, whose square is the sum of the squared entries; for a single-column residual, it is the Euclidean norm. In the first formula, let Y∈Rm×rY\in\mathbb{R}^{m\times r}, W∈Rm×nW\in\mathbb{R}^{m\times n}, and X∈Rn×rX\in\mathbb{R}^{n\times r}:

∂∥Y−W X∥F2∂W=−2 (Y−W X) X⊤.\frac{\partial \|Y - W\,X\|_F^2}{\partial W} = -2\,(Y - W\,X)\,X^\top.

In the second formula, let Y∈Rm×rY\in\mathbb{R}^{m\times r}, X∈Rm×nX\in\mathbb{R}^{m\times n}, and W∈Rn×rW\in\mathbb{R}^{n\times r}:

∂∥Y−X W∥F2∂W=−2 X⊤ (Y−X W).\frac{\partial \|Y - X\,W\|_F^2}{\partial W} = -2\,X^\top\,(Y - X\,W).

Other Common Derivatives

The logarithmic formulas require x⊤a>0\mathbf{x}^\top\mathbf{a}>0 and det⁡(A)>0\det(A)>0, respectively. The determinant condition also ensures that AA is invertible.

∂ln⁡(x⊤a)∂x=ax⊤a,\frac{\partial \ln(\mathbf{x}^\top \mathbf{a})}{\partial \mathbf{x}} = \frac{\mathbf{a}}{\mathbf{x}^\top \mathbf{a}},
∂ln⁡(det⁡(A))∂A=(A−1)⊤,\frac{\partial \ln(\det(A))}{\partial A} = (A^{-1})^\top,
∂∥A∥F2∂A=2A,\frac{\partial \|A\|_F^2}{\partial A} = 2A,

where ∥A∥F\|A\|_F denotes the Frobenius norm of AA.

Useful Resources

For a broader collection of matrix identities and derivatives, see the Matrix Cookbook. For additional background, see: