Multi Variable Derivatives

Multi-variable derivative

To the people that are not used to matrix derivatives (like me) it could be useful to see how

$$\frac{\partial u^{T}Su}{\partial u}=2Su$$

First, we note that if you derive with respect to some matrix, the output will be of the same dimension of that matrix. That notation is just deriving every single component independently and then joining them together, so it will be better understood as as

$$\frac{\partial u^{T}Su}{\partial u}=\left[\begin{array}{c}\frac{\partial u^{T}Su}{\partial u_1}\\ \dots \\ \frac{\partial u^{T}Su}{\partial u_M}\end{array}\right]=\left[\begin{array}{c}2(Su{)}_1\\ \dots \\ 2(Su{)}_M\end{array}\right]=2Su$$

So we can prove each derivative independently, it's just a lot of manual work! We see that $u^{T}Su$ is just a quadratic form, studied in Massimi minimi multi-variabile#Forme quadratiche so it is just computing this:

$$u^{T}Su=\sum _{i,j=1,1}^M{u}_i{u}_j{S}_{ij}⟹\frac{\partial u^{T}Su}{\partial u_1}=2u_1{S}_{11}+\sum _{j\ne 1}^M(u_j{S}_{1j}+u_j{S}_{j1})=2\left(u_1{S}_{11}+\sum _{j\ne 1}{u}_j{S}_{1j}\right)=2(Su{)}_1$$

Last equation is true because $S$ is a symmetric matrix, then we easily see that indeed it's true that indeed it's the first row of the $Su$ matrix multiplied by 2.

Known theorems

The Multivariate Chain Rule

Let $\mathbf{x}=(x_1,x_2,\dots ,x_n)$ be an $n$-dimensional vector, and let each $x_i$ depend on a scalar variable $t$, i.e.,

$$x_i=x_i(t),\text{for }i=1,2,\dots ,n.$$

Suppose we have a function $f$ that maps ${\mathbb{R}}^n\to \mathbb{R}$, i.e.,

$$f:{\mathbb{R}}^n\to \mathbb{R},f=f(x_1,x_2,\dots ,x_n).$$

Then, the total derivative of $f$ with respect to $t$ is given by:

$$\frac{df}{dt}=\sum _{i=1}^n\frac{\partial f}{\partial x_i}\frac{d{x}_i}{dt}.$$

or, in vector notation:

$$\frac{df}{dt}=\nabla f\cdot \frac{d\mathbf{x}}{dt},$$

where:

• $\nabla f=\left(\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\dots ,\frac{\partial f}{\partial x_n}\right)$ is the gradient of $f$.
• $\frac{d\mathbf{x}}{dt}=\left(\frac{d{x}_1}{dt},\frac{d{x}_2}{dt},\dots ,\frac{d{x}_n}{dt}\right)$ is the time derivative of $\mathbf{x}$.

Proof

By definition, the total derivative of $f$ with respect to $t$ measures the rate of change of $f$ as $t$ varies:

$$\frac{df}{dt}=\underset{\Delta t\to 0}{\lim }\frac{f(\mathbf{x}(t+\Delta t))-f(\mathbf{x}(t))}{\Delta t}.$$

Since $f$ is a function of $\mathbf{x}$, we perform a first-order Taylor expansion (see Hopital, Taylor, Peano) around $\mathbf{x}(t)$:

$$f(\mathbf{x}(t+\Delta t))\approx f(\mathbf{x}(t))+\sum _{i=1}^n\frac{\partial f}{\partial x_i}{|}_{\mathbf{x}(t)}\cdot \Delta x_i.$$

Dividing by $\Delta t$ and taking the limit:

$$\frac{df}{dt}=\underset{\Delta t\to 0}{\lim }\sum _{i=1}^n\frac{\partial f}{\partial x_i}\frac{\Delta x_i}{\Delta t}$$

Since ${\lim }_{\Delta t\to 0}\frac{\Delta x_i}{\Delta t}=\frac{d{x}_i}{dt}$, we obtain:

$$\frac{df}{dt}=\sum _{i=1}^n\frac{\partial f}{\partial x_i}\frac{d{x}_i}{dt}.$$

Using vector notation we have:

$$\frac{df}{dt}=\nabla f\cdot \frac{d\mathbf{x}}{dt}.$$

This represents the directional derivative of $f$ along the trajectory $\mathbf{x}(t)$, showing how $f$ evolves as $t$ changes.

Total Derivative Rule

This is a simple extension of the multi-variable chain rule described above:

Let $f(\mathbf{w},\theta )$ be a function of:

• A vector $\mathbf{w}\in {\mathbb{R}}^n$ which itself depends on $\theta$, i.e., $\mathbf{w}=t(\theta ,ϵ)$.
• A scalar parameter $\theta$.

The total derivative of $f(\mathbf{w},\theta )$ with respect to $\theta$ is given by:

$$\frac{d}{d\theta }f(\mathbf{w},\theta )=\frac{\partial f}{\partial \mathbf{w}}\cdot \frac{d\mathbf{w}}{d\theta }+\frac{\partial f}{\partial \theta }.$$

This result follows from the multivariate chain rule. For a function $f(x_1,x_2,\dots ,x_n,\theta )$ where each $x_i$ depends on $\theta$, the total derivative is:

$$\frac{df}{d\theta }=\sum _{i=1}^n\frac{\partial f}{\partial x_i}\frac{d{x}_i}{d\theta }+\frac{\partial f}{\partial \theta }\frac{\partial \theta }{\partial \theta }$$

In our case:

• The variables $x_i$ correspond to the components of $\mathbf{w}$.
• $\mathbf{w}$ is a vector, so we sum over its components.

Thus, applying the chain rule:

$$\frac{d}{d\theta }f(\mathbf{w},\theta )=\sum _{i=1}^n\frac{\partial f}{\partial w_i}\frac{d{w}_i}{d\theta }+\frac{\partial f}{\partial \theta }.$$

We can rewrite the above in vector notation. Since $\mathbf{w}$ is an $n$-dimensional vector, we rewrite the sum as a dot product:

$$\frac{d}{d\theta }f(\mathbf{w},\theta )={\nabla }_{\mathbf{w}}f\cdot \frac{d\mathbf{w}}{d\theta }+\frac{\partial f}{\partial \theta }=\frac{\partial f}{\partial \mathbf{w}}\cdot \frac{d\mathbf{w}}{d\theta }+\frac{\partial f}{\partial \theta }$$

where:

• $\frac{\partial f}{\partial \mathbf{w}}$ is the gradient $\left[\frac{\partial f}{\partial w_1},\frac{\partial f}{\partial w_2},\dots ,\frac{\partial f}{\partial w_n}\right]$.
• $\frac{d\mathbf{w}}{d\theta }$ is the Jacobian $\left[\frac{d{w}_1}{d\theta },\frac{d{w}_2}{d\theta },\dots ,\frac{d{w}_n}{d\theta }\right]$.

One application of this formalism can is the reparametrization trick in Variational Inference.

Common derivatives

Determinant

Also see wikipedia.

$$\frac{\partial \det (\mathbf{A}(t))}{\partial \mathbf{t}}=\det (\mathbf{A})\cdot \left(\text{tr}({\mathbf{A}}^{-1})\cdot \frac{\partial A(t)}{\partial x}\right)$$

In the special case we have:

$$\frac{\partial \det (\mathbf{A})}{\partial \mathbf{A}}=\det (\mathbf{A})\cdot ({\mathbf{A}}^{-1}{)}^{\top }$$

Proof:

$$\begin{array}{rl}\frac{\partial \det (\mathbf{A})}{\partial \mathbf{A}}& =\det (\mathbf{A})\cdot \frac{\partial \ln \det (\mathbf{A})}{\partial \mathbf{A}}\\ & =\det (\mathbf{A})\cdot \frac{\partial \text{tr}(\ln A)}{\partial \mathbf{A}}\\ & \\ & =\det (\mathbf{A})\cdot ({\mathbf{A}}^{-1}{)}^{\top }\end{array}$$

I don't think I have understood this thing quite well...

Matrix Inverse

$$\frac{\partial {\mathbf{A}}^{-1}}{\partial \mathbf{A}}=-{\mathbf{A}}^{-1}\otimes {\mathbf{A}}^{-1}.$$ $$\begin{array}{rl}\frac{\partial }{\partial \mathbf{A}}(\mathbf{A}{\mathbf{A}}^{-1})& =\frac{\partial \mathbf{I}}{\partial \mathbf{A}}=0\\ & ⟹\frac{\partial \mathbf{A}}{\partial \mathbf{A}}\cdot {\mathbf{A}}^{-1}+\mathbf{A}\cdot \frac{\partial {\mathbf{A}}^{-1}}{\partial \mathbf{A}}=0\\ & ⟹\mathbf{I}\cdot {\mathbf{A}}^{-1}+\mathbf{A}\cdot \frac{\partial {\mathbf{A}}^{-1}}{\partial \mathbf{A}}=0\\ & ⟹\frac{\partial {\mathbf{A}}^{-1}}{\partial \mathbf{A}}=-{\mathbf{A}}^{-1}\cdot {\mathbf{A}}^{-1}.\end{array}$$

Quadratic Form

$$\frac{\partial }{\partial \mathbf{A}}\left({\mathbf{v}}^{\top }\mathbf{Av}\right)=\mathbf{v}{\mathbf{v}}^{\top }.$$

This should be easy, and quite similar to the above case when we have derived $v$.

Quadratic Inverse

$$\frac{\partial }{\partial \mathbf{A}}\left({\mathbf{v}}^{\top }{\mathbf{A}}^{-1}\mathbf{v}\right)=-{\mathbf{A}}^{-1}\mathbf{v}{\mathbf{v}}^{\top }{\mathbf{A}}^{-1}.$$

You can interpret this as a function composition.

Vector Matrix derivative

Suppose yo uhave a $v\in {\mathbb{R}}^h$ and a $M\in {\mathbb{R}}^{m\times h}$ so $Mv\in {\mathbb{R}}^m$ then the derivative is:

$$\frac{\partial Mv}{\partial M}=I_m\otimes v^T$$

Where $M$ is the matrix and $I_m$ is the identity matrix of size $m\times m$ and $\otimes$ is the Kronecker product. So the dimension of the derivative is:

$$\frac{\partial Mv}{\partial M}\in {\mathbb{R}}^{m\times mh}$$