NOTE: this is an old set of note, and it is of quite bad quality, it should be completely rewritten.

How can we transform a uniform into a random variable? It is true that we have

$$F(x)={\int }_{-\infty }^{x}f(t)dt$$

A volte la densità non è definita, mentre la funzione cumulativa lo è , per questo spesso cominciamo a definire partendo dalla definizione.

Suppose we have a $x\sim F_X(x)$ where $F$ is a cumulative distribution function, same thing, we just need to take the set, normal cumulative distribution function that we saw a lot in other courses.

$$F_X(x)=\mathbb{P}(X\le x)={\int }_{-\infty }^x{f}_X(z)dz$$

Generalized inverse

This is known as the universality of the uniform, every invertible CDF can be written with respect to the uniform distribution.

Definition

We want to have an inverse of the cdf, but i don't know why. By some definition the transformation is still a random variable. The definition of the generalized inverse is:

$$F_X^{-1}(u)=inf\left\{x;F_X(x)\ge u\right\}$$

This has sense because we know that the inverse is a continuous function (we do or not?). This is useful because when we have a cumulative probability distribution this allows to recreate the original random variable distribution, and it's quite easy to get it in this way.

Probability Inverse transform

Sample from $X$ when it's difficult to sample from that distribution (for example function difficult to calculate?, Difficult to implement function?)

Requirements:

1. I know the functional form of $F_X(x)$
2. $X$ in continuous.

For the exam you will be given the cumulative distribution and you need to use this theorem to sample

Theorem

The inverse distribution the cumulative of the uniform distribution is the same for that random variable

$$U\sim F_X(x)⟺\mathbb{P}(Y\le x)=X$$

Where $Y=F_X^{-1}(U)$ this is just the definition of the cumulative function, the catch is the ???

1. $F_X(u)$ is continuous then it is invertible
2. If $U\sim Unif(0,1)$ then the c.d.f is easy, and it's equal to

$$\{\begin{array}{l}0,u\le 0\\ u,0\le u\le 1\\ 1,u\le 1\end{array}$$

And this is easy. This is proved to be the only distribution with this property, that the CDF is an identify 3. $F_X(X)=U$ this is also called PIT See here for proof (it's cool)

Proof 1

$$\mathbb{P}(F_X^{-1}(U)\le x)=\mathbb{P}(F_X\left[F_X^{-1}(U)\right]\le F_X(x))=\mathbb{P}(U\le F_X(x))=F_U[F_X(x)]=F_X(x)$$

In the first passage we used that the cumulative distribution function is monotonically increasing, in the second a clear property for the inverse, and at the end we used a property of the cumulative uniform distribution.

Proof 2

Proof of the same fact, so we have: Proof of the point 3.

$$U=F_U(u)=\mathbb{P}(U\le u)=\mathbb{P}(F_X(X)\le F_X(x))=\mathbb{P}(F_X^{-1}(F_X(X))\le F_X^{-1}(F_X(x)))=\mathbb{P}(X\le x)=F_X(x)$$

After you have this you can just use the inverse ->

$$F_X^{-1}(U)=X$$

So now you can sample.

Main Proof

This proof is my favorite. Recall the definition of change of variables for distributions:

$$p(y)=p(x)\left|\frac{\delta x}{\delta y}\right|$$

Suppose now $p(x)$ is the uniform over $[0,1]$, meaning it has density $\forall x\in [0,1]:p(x)=1$ and 0 elsewhere, and $x=h(\hat{y})={\int }_{-\infty }^{\hat{y}}p(y)dy$, then plugging everything we see that:

$$p(y)=p(x)\left|\frac{\delta x}{\delta y}\right|=1\left|\frac{\delta x}{\delta y}\right|=\left|\frac{\partial h(y)}{\partial y}\right|=|p(y)|=p(y)$$

This implies if we use the reverse transformation for $h$, we can effectively sample $y$ starting from the uniform distribution $x$. This needs two main ingredients:

1. The function $h$ must be invertible
2. The function $h$ must exist and be well behaved, i.e. must be continuous

Example of application

$$X\sim Exp(\lambda =1)$$

And given $F_X(x)=1-e^{-x}$, with $x\in {\mathbb{R}}^{+}\cup \left\{0\right\}$ Find the value of $X$ random variable.

Solution example:

1. $F_X(X)=U$, then set up the equation and solve

$$1-e^{-X}=U⟹e^{-X}=1-U⟹X=-\log (1-U)$$

We can notice that $1-U$ and $U$ are both uniform distribution, so the above is the same as $X=-\log (U)$ And in this way you get the correct random variable. Now we can sample from the uniform and get the correct result! -> Direct transformation method. With this process we prooved that $-\log U\sim Exp(\lambda =1)$

Famous function CDF

Logistic function

This is a function very similar to that used in Logistic Regression. We have $X\sim \text{Logistic}(\mu ;\beta )$, then

$$F_X(x)=\frac{1}{1+e^{-(x-\mu )/\beta }}$$

Let's make the derivation:

$$U=F_X(X)=\frac{1}{1+e^{-(X-\mu )/\beta }}⟹1+e^{-(X-\mu )/\beta }=\frac{1}{U}⟹-(X-\mu )/\beta =\log (\frac{1}{U}-1)$$ $$⟹X=-\beta \log \left(\frac{1}{U}-1\right)+\mu$$

Where $\beta$ and $\mu$ are parameters of the distribution.

Cauchy distribution

How to simulate data from a Cauchy distribution?

$$F_X(x)=\frac{1}{2}+\frac{1}{\pi }\arctan ((x-\mu )/\sigma )$$

The derivation

$$U=F_X(X)=\frac{1}{2}+\frac{1}{\pi }\arctan ((X-\mu )/\sigma )⟹\tan (\pi \left(U-\frac{1}{2}\right))=(X-\mu )/\sigma ⟹X=\sigma \cdot \tan (\pi \left(U-\frac{1}{2}\right))+\mu$$

So also in this case we have a way to sample a Cauchy distribution by just manipulating a uniform distribution.

How for the Gaussian?

$$\Phi (x)={\int }_{-\infty }^x{f}_X(z)dz{\int }_{-\infty }^x\frac{1}{\sqrt{2\pi }{\sigma }^2}\exp \left\{-\frac{(z-\mu {)}^2}{2{\sigma }^2}\right\}dz$$

Doesn't have a clear evaluation function because it's difficult, in R it uses the Probability inverse transform.

Empirical C.D.F

The definition is simple:, if we have a i.i.d. sample

$$(X_1,\dots ,X_n):{\hat{F}}_{X,m}(x)=\frac{1}{n}\cdot \sum _{i=1}^{n}1(X_i\le x)$$

It's just the sum of all the sampled data that is lower than a certain value! (unbiased, on average, the emprirical cdf we have the right cdf!).

Other famous distributions

Chi squared distribution math exchange

if we have $X_i\sim Exp(1)$ then it is true that

$$Y=2\sum _{i=1}^n{X}_i\sim {\chi }_{2n}^2$$

Where $n$ is the degrees of freedom, bad thing is that this usually just even, so useful for that. (chi square gamma distribution)

$$Y=\beta \sum _{i=1}^n{X}_i\sim G(a,\beta ):a\in {\mathbb{N}}^{\ast }$$

Which is a Gamma distribution, exponential is special case of gamma?

$$Y=\frac{{\sum }_{i=1}^a{X}_i}{{\sum }_{i=1}^{a+b}{X}_i}\sim Be(a,b)$$

Which is a Beta regression, used for microbiome in the guts, I don't know why.

Transformation of random variables (no exam)

La cosa strana per questi statistici e che non si capisce per quale fine stiamo facendo queste trasformazioni, che non sono molto utili a primo impatto. Nel senso che non so nemmeno quale sia il problema che stiamo provando a risolvere!

Suppose $X\sim f_X(x)$, and $Z=g(X)$ with $g$ an invertible function, then suppose $Z\sim f_Z(z)$ Then the support of the density is in $X$ , we have that

$$f_Z(z)=\frac{\delta F_Z(z)}{\delta z}=f_X\left[g^{-1}(z)\right]\cdot \frac{\delta g^{-1}(z)}{\delta z}$$

Non so esattamente cosa mi possa servire questo e non so nemmeno perché funziona. Dopo: alla fine la dimostrazione di questo è molto semplice (io sono un po' incapace nelle dimo a quanto pare e tendo ad impararle a memoria) Comunque lo trovi in https://en.wikipedia.org/wiki/Random_variable

Uniform random variable

Permette la creazione di uniform tale per cui non siano sempre nel classico 0, 1

$$U\sim Unif(0,1)⟹f_U(u)=\{\begin{array}{l}0:u\notin \left[0,1\right]\\ 1:u\in \left[0,1\right]\end{array}$$

Allora la trasformazione sarebbe:

$$X=(b-a)U+a,b>a$$

Then

$$U=\frac{x-a}{b-a}=g^{-1}(X)⟹\frac{\delta g^{-1}(X)}{\delta x}=\frac{1}{b-a}⟹f_X(x)=f_U\left[\frac{x-a}{b-a}\right]\cdot \frac{1}{b-a}$$

and we have that $f_U$ is in $\left[0,1\right]$ when we need to have $x\ge a\wedge x\le b$ in order to have a density different than 0.

Standard Gaussian

$X\sim N(\mu ,{\sigma }^2)$ then $Z=(X-\mu )/\sigma$ . Which is closed under linear transformations! and $X=g^{-1}(Z)=\sigma \cdot Z+\mu$ Then:

$$f_Z(z)=f_X(\sigma z+\mu )\sigma =\frac{1}{\sqrt{2\pi }}\exp \left\{-\frac{Z^2}{2}\right\}$$

Where the first part is the density of a Gaussian

Box-Muller algorithm

This is one resource. This is another. Also (Bishop & Bishop 2024) section 14.1.2 presents this.

This algorithm is used to produce Gaussian random variables: we have $U_1$ and $U_2$ that are normal uniform, then we can take

$$X_1=\sqrt{-2\log (U_1)}\cos (2\pi U_2),X_2=\sqrt{-2\log (U_2)}\sin (2\pi U_1),$$

Without any approximation, it is exact for some reason

Multivariate cases

We can use the re-parametrization trick to gen $N(\mu ,{\sigma }^2)$ just multiplying by the variance and add the mean. We are asking if we can do the same even for the multivariate case, how can we get $N_p(\mu ,\Sigma )$ ? Let's take a bi-variate Gaussian for example. It seems like a result that we have that the square root of a matrix is almost the same as a Cholesky decomposition. That will be our sigma. it's always possible because we have a positive definite symmetric matrix for the Sigma. See Algebra lineare numerica for cholesky

The discrete case

Vogliamo andare a prendere molte probabilità in questo modo

$$p_0=P_{\theta }(X\le 0),p_1=P_{\theta }(X\le 1)evia$$

È sempre come conoscere una CDF discreta. Poi prendiamo

$$X=k,p_{k-1}\le U\le p_k$$

e così facciamo sampling della variabile per le distribuzioni discrete $X$

References

[1] Bishop & Bishop “Deep Learning: Foundations and Concepts” Springer International Publishing 2024