Gaussians are one of the most important family of probability distributions. They arise naturally in the law of large numbers and have some nice properties that we will briefly present and prove here in this note. They are also quite common for Gaussian Processes and the Clustering algorithm. They have also something to say about Maximum Entropy Principle. The best thing if you want to learn this part actually well is section 2.3 of (Bishop 2006), so go there my friend :)

The Density Function

The single variable Gaussian is as follows:

$$\mathcal{N}(\mu ,\sigma )=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$$

This can be generalized for the multi variable case

$$\mathcal{N}(\mu ,\Sigma )=\frac{1}{(2\pi {)}^{d/2}\sqrt{|\Sigma |}}\exp \left(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right)$$

Where $d$ is the dimensionality for the multidimensional Gaussian.

Integral is 1

We now prove that the integral of the Gaussian PDF is 1, this is a requirements needed to be considered a probability distribution function.

First, let's prove a famous equality:

$$I={\int }_{-\infty }^{\infty }\exp (-x^2)dx=\sqrt{\pi }$$

This is kinda surprising, we need some care to prove it:

$$\begin{array}{rl}& \\ I^2& ={\int }_{-\infty }^{\infty }\exp (-x^2-y^2)dxdy\\ & ={\int }_0^{2\pi }{\int }_0^{\infty }r\exp (-r^2)drd\theta \\ & =2\pi \cdot \left(-\frac{1}{2}\right){\int }_0^{\infty }-2r\exp (-r^2)dr\\ & =-\frac{2\pi }{2}\exp (-r^2){|}_0^{\infty }=\pi \\ & ⟹I=\sqrt{\pi }\end{array}$$

Note that in the second step we changed variables: This is quite interesting. If we do the same derivation with $\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$, first doing a change of variables $y=(x-\mu )$ where we have $dy=dx$, then doing another change of variables $z=\frac{y}{\sqrt{2{\sigma }^2}}$ where we get $\sqrt{2{\sigma }^2}dz=dy$ now we have the same integral, plus an added constant multiplicative term. So we have

$${\int }_{-\infty }^{\infty }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)dx=\sqrt{2{\sigma }^2}{\int }_{-\infty }^{\infty }\exp (-z^2)dz=\sqrt{2\pi {\sigma }^2}$$

Which finishes the derivation of the normalizing term.

Error Function

Sometimes, for example calculating the mean of the folded Gaussian, is useful to consider the error function. This is defined as

$$\text{erf}(x)=\frac{2}{\sqrt{\pi }}{\int }_0^x\exp (-t^2)dt$$

Sometimes this is also written, using the symmetry over the $x-$axis as

$$\text{erf}(x)=\frac{1}{\sqrt{\pi }}{\int }_{-x}^x\exp (-t^2)dt$$

We observe that the limit $x\to +\infty$ is 1, and that $x\to -\infty$ is -1.

Another useful relation is with the Gaussian CDF:

$$\text{erf}\left(\frac{x}{\sqrt{2}}\right)=2\Phi (x)-1$$

We also note that it is anti-symmetric:

$$\text{erf}(-x)=-\text{erf}(x)$$

Some properties of Gaussians

The conditional Gaussian

If we have $X,Y$ which are jointly Gaussian, then the distribution $p(X=x\mid Y=y)$ is a gaussian with the following mean and variance:

$${\mu }_{X\mid Y=y}={\mu }_X+{\Sigma }_{XY}{\Sigma }_{YY}^{-1}(y-{\mu }_Y)$$

And

$${\Sigma }_{X\mid Y}={\Sigma }_{XX}-{\Sigma }_{XY}{\Sigma }_{YY}^{-1}{\Sigma }_{YX}$$

The proof is presented in section 2.3 of (Bishop 2006)

Product of Gaussians are Gaussian

This is a little more difficult to detail, see this chatgpt response. It's just an Unnormalized Gaussian.

Marginals are Gaussians

One can prove that any finite marginals of Gaussians are still multivariate Gaussians.

Let's now write a closed for for this. Let's assume we have these two random variables:

$$p(A,B)=\left[\begin{array}{c}A\\ B\end{array}\right]\sim \mathcal{N}(\mu ,\Sigma )$$

Where:

$$\mu =\left[\begin{array}{c}{\mu }_A\\ {\mu }_B\end{array}\right],\Sigma =\left[\begin{array}{cc}{\Sigma }_{AA}& {\Sigma }_{AB}\\ {\Sigma }_{BA}& {\Sigma }_{BB}\end{array}\right]$$

We want to find the value of $p(A)$ and of $p(B\mid A)$. To prove this it is useful to remember the value of the following matrix:

$${\Sigma }^{-1}=V=\left[\begin{array}{cc}{V}_{11}& V_{12}\\ V_{21}& V_{22}\end{array}\right]=\left[\begin{array}{cc}I& V_{12}{V}_{22}^{-1}\\ 0& I\end{array}\right]\cdot \left[\begin{array}{cc}{V}_{11}-V_{12}{V}_{22}^{-1}{V}_{21}& 0\\ 0& V_{22}\end{array}\right]\cdot \left[\begin{array}{cc}I& 0\\ V_{22}^{-1}{V}_{21}& I\end{array}\right]$$

Then the inverse $(ABC{)}^{-1}=C^{-1}{B}^{-1}{A}^{-1}$ which is equal to:

$$V^{-1}=\left[\begin{array}{cc}I& 0\\ -V_{22}^{-1}{V}_{21}& I\end{array}\right]\cdot \left[\begin{array}{cc}(V_{11}-V_{12}{V}_{22}^{-1}{V}_{21}{)}^{-1}& 0\\ 0& V_{22}^{-1}\end{array}\right]\cdot \left[\begin{array}{cc}I& -V_{12}{V}_{22}^{-1}\\ 0& I\end{array}\right]=\left[\begin{array}{cc}(V_{11}-V_{12}{V}_{22}^{-1}{V}_{21}{)}^{-1}& -{\Sigma }_{11}{V}_{12}{V}_{22}^{-1}\\ -{\Sigma }_{11}{V}_{22}^{-1}{V}_{22}& V_{22}^{-1}(V_{12}{V}_{22}^{-1}{V}_{21}{\Sigma }_{11}+1)\end{array}\right]$$

One can note now with the inverse thing that $V_{22}=({\Sigma }_{22}-{\Sigma }_{21}{\Sigma }_{11}^{-1}{\Sigma }_{12}{)}^{-1}$ This allows to write $V_{12}$ nicely as

$${\Sigma }_{12}=-{\Sigma }_{11}{V}_{12}{V}_{22}^{-1}⟹V_{12}=-{\Sigma }_{11}{\Sigma }_{12}^{-1}{V}_{22}^{-1}=-{\Sigma }_{11}{\Sigma }_{12}^{-1}({\Sigma }_{22}-{\Sigma }_{21}{\Sigma }_{11}^{-1}{\Sigma }_{12}$$

Because then it is easily invertible and one can observe that ${\Sigma }_{11}=(V_{11}-V_{12}{V}_{22}^{-1}{V}_{21}{)}^{-1}$, this is used for the marginalization calculation. One can find in this manner that

$$A\sim \mathcal{N}({\mu }_A,{\Sigma }_{AA})$$

And that

$$B\mid A\sim \mathcal{N}({\mu }_B-V_{BB}^{-1}{V}_{AB}(A-{\mu }_A),V_{BB})$$

If you are a student ad ETH watch this for the derivation, minute 46. Rewriting with the above properties for $V_{BB}$ and $V_{AB}$ we obtain:

$$B\mid A\sim \mathcal{N}({\mu }_B+{\Sigma }_{21}{\Sigma }_{11}^{-1}(A-{\mu }_A),{\Sigma }_{22}-{\Sigma }_{21}{\Sigma }_{11}^{-1}{\Sigma }_{12})$$

Which is a ok form, but very very long to derive.

Gaussian characteristic function

Characteristic functions are sometimes useful to prove that two distributions are the same as each other. One can prove that the characteristic function for Gaussians is

$$\mathbb{E}[\exp (itX)]=\exp \left(\mu it-\frac{1}{2}{t}^T\Sigma t\right)$$

Let's prove the uni-variate case, we will see that it will be exactly this value. We need to compute the value:

$$\mathbb{E}[\exp (itX)]={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}+itx\right)dx$$

The idea is to complete the square, and the by knowing the value of the integral of the completed square, we simplify.

$$\begin{array}{r}\\ {\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}+itx\right)dx\\ ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-(\mu +{\sigma }^{2}it){)}^2-2\mu it{\sigma }^2+{\sigma }^4{t}^2}{2{\sigma }^2}\right)dx\\ =\exp \left(\mu it-\frac{1}{2}{\sigma }^2{t}^2\right)\cdot {\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-(\mu +{\sigma }^{2}it){)}^2}{2{\sigma }^2}\right)dx\\ =\exp \left(\mu it-\frac{1}{2}{\sigma }^2{t}^2\right)\end{array}$$

Sum Gaussians are Gaussian

This is easily provable, if we have $X\sim \mathcal{N}({\mu }_X,{\Sigma }_X)$ and a compatible distribution $Y\sim \mathcal{N}({\mu }_Y,{\Sigma }_Y)$ then we have that the distribution $X+Y=\mathcal{N}({\mu }_X+{\mu }_Y,{\Sigma }_X+{\Sigma }_Y)$ The proof should use characteristic functions in the line of linear Gaussians.

$$\begin{array}{rl}\mathbb{E}[\exp (it(X+Y))]& =\mathbb{E}[\exp (itX)]\mathbb{E}[\exp (itY)]\\ & =\exp \left({\mu }_{X}it-\frac{1}{2}{t}^T{\Sigma }_{X}t\right)\exp \left({\mu }_{Y}it-\frac{1}{2}{t}^T{\Sigma }_{Y}t\right)\\ & =\exp \left(({\mu }_X+{\mu }_Y)it-\frac{1}{2}{t}^T({\Sigma }_X+{\Sigma }_Y)t\right)\end{array}$$

Which finishes the proof. One can also extend this result to every linear combination of Gaussians.

Properties to remember

• Compact representation of high dimensional joint distributions: instead of using $2^n$ variables we just need $n^2$, this is why Gaussian Processes are analytically handy.
• Closed form inference (I think about the Conjugacy of itself, this is because Gaussians are in the The Exponential Family.)

Confidence Intervals

Gaussians are a nice distribution. We have listed many of its properties by now. But one of the most over-utilized feature is the ease in computing $1-\alpha$ confidence intervals where $\alpha$ is called significance level: meaning we want to find the interval where our prediction lies there with $1-\alpha$ probability. This is usually easy to compute with $z$ tables. The Standard Error of a Gaussian is $\frac{\sigma }{\sqrt{n}}$ and is related to the square root of the mean variance.

So after we have computed this values, the confidence interval for a prediction is just

$$\bar{x}\pm z\cdot SE$$

Where $\bar{x}$ is the expected value for our prediction.

Information theoretic properties

Entropy of a Gaussian distribution

We compute here the Entropy of a Univariate Gaussian distribution $\mathcal{N}(x;\mu ,{\sigma }^2)$. So we need to compute the following value:

$$\begin{array}{rl}\int p(x)\log \frac{1}{p(x)}dx& =-\int \frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)\cdot \left(-\frac{1}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right)dx\\ & =\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{1}{2{\sigma }^2}{\mathbb{E}}_x[(x-\mu {)}^2]\\ & =\frac{1}{2}\log (2\pi {\sigma }^2)+\frac{1}{2}\\ & =\frac{1}{2}\log (2\pi {\sigma }^{2}e)\end{array}$$

With just the above proof one can prove that Gaussians are the distributions with maximum entropy for a given mean and variance. See Maximum Entropy Principle.

We can extend this to the multivariate case, observing the following:

$$\begin{array}{rl}{\mathbb{E}}_{x\sim p}[-\log p(x)]& ={\mathbb{E}}_{x\sim p}\left[\frac{d}{2}\log (2\pi )+\log \det \Sigma +\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right]\\ & =\frac{d}{2}\log (2\pi )+\log \det \Sigma +\frac{1}{2}{\mathbb{E}}_{x\sim p}[(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )]\\ & =\frac{d}{2}(1+\log (2\pi ))+\log \det \Sigma \\ & =\frac{d}{2}\log (2\pi e)+\log \det \Sigma \\ & =\frac{1}{2}\log ((2\pi e{)}^d|\Sigma |)\end{array}$$

Where in the last step we used this equality:

$$\begin{array}{rlr}{\mathbb{E}}_{x\sim p}[(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )]& ={\mathbb{E}}_{x\sim p}[\text{tr}((x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))]& \text{ trace of real number}\\ & ={\mathbb{E}}_{x\sim p}[\text{tr}({\Sigma }^{-1}(x-\mu )(x-\mu {)}^T)]& \text{ eq. 16 Matrix Cookbook}\\ & =\text{tr}({\mathbb{E}}_{x\sim p}[{\Sigma }^{-1}(x-\mu {)}^T(x-\mu )])& \text{ linearity of trace}\\ & =\text{tr}({\Sigma }^{-1}{\mathbb{E}}_{x\sim p}[(x-\mu )(x-\mu {)}^T])& \text{ linearity of expectation}\\ & =\text{tr}({\Sigma }^{-1}\Sigma )& \text{ definition of covariance}\\ & =d& \text{ trace of identity matrix}\end{array}$$

The Matrix Cookbook refers to this resource.

Mutual information of Gaussians

Suppose we have a Gaussian $X\sim \mathcal{N}(\mu ,\Sigma )$ and $Y=X+\epsilon ,\epsilon \sim \mathcal{N}(0,{\sigma }_n^{2}I)$ TODO

We will see that this is equal to:

$$I(X,Y)=\frac{1}{2}\log |I+{\sigma }_n^{-2}\Sigma |$$

General KL divergence between Gaussians

The KL divergence between two Gaussians is given by:

$$KL(p\mid \mid q)=\frac{1}{2}\left(\log \frac{|{\Sigma }_q|}{|{\Sigma }_p|}-d+tr({\Sigma }_q^{-1}{\Sigma }_p)+({\mu }_q-{\mu }_p{)}^T{\Sigma }_q^{-1}({\mu }_q-{\mu }_p)\right)$$

This is a good resource for a proof.

Forward KL

One can prove that the forward KL divergence between two Gaussians defined as $p\sim \mathcal{N}({\mu }_1,diag\left\{{\sigma }_1^2,\dots ,{\sigma }_d^2\right\})$ and $q=\mathcal{N}(0,1)$ is given by:

$$KL(p\mid \mid q)=\frac{1}{2}\sum _{i=1}^d\left({\sigma }_i^2+{\mu }_i^2-\log {\sigma }_i^2-1\right)$$

Let’s interpret this. The $\mu$ term works to pull the mean toward zero. The $\sigma$ term introduces a penalty for high variance values, while the $\log {\sigma }^2$ term imposes a cost for low values of $\sigma$. This forward KL is what is used for Autoencoders.

Reverse KL

Given the same assumptions we have that the KL of $q$ over $p$ is given by:

$$KL(q\mid \mid p)=\frac{1}{2}\sum _{i=1}^d\left(\frac{{\mu }_i^2}{{\sigma }_i^2}+{\sigma }_i^{-2}+\log {\sigma }_i^2-1\right)$$

References

[1] Bishop “Pattern Recognition and Machine Learning” Springer 2006