✏Information-Theory

Sembra essere molto simile a Central Limit Theorem and Law of Large Numbers però per Entropy. This is also called Shannon’s source coding theorem see here Enunciato AEP $$ -\frac{1}{n} \log p(X_{1}, X_{2}, \dots, X_{n}) \to H(X) $$ in probability (la definizione data in Central Limit Theorem and Law of Large Numbers#Convergence in probability). Un modo alternativo per enunciarla è così, segue il metodo in (MacKay 2003). $$ \left\lvert \frac{1}{N} H_{\delta}(X^{N}) - H(x) \right\rvert \leq \varepsilon $$Ossia a grandi linee: dato una variabile aleatoria $X$ e $N$ estrazioni della stessa, possiamo comprimere questa sequenza in $NH(X)$. ...

Questo è stato creato da 1948 Shannon in (Shannon 1948). Questa nozione è basata sulla nozione di probabilità, perché le cose rare sono più informative rispetto a qualcosa che accade spesso. Introduction to Entropy The Shannon Information Content $$ h(x = a_{i}) = \log_{2}\frac{1}{P(x = a_{i})} $$ We will see that the entropy is a weighted average of the information, so the expected information content in a distribution. Kolmogorov complexity è un modo diverso per definire la complessità. Legato è Neural Networks#Kullback-Leibler Divergence. ...

The course will be more about the the quantization, talking about lossless and lossy compression (how many bits will be needed to describe something? This is not a CS course so it will not be so much algorithmically focused course), then we will talk about channel and capacity and DMC things. Most of the things explained in the Lapidoth course will be theoretical there will be some heavy maths. The professor starts with some mathy definitions (not very important, just that the $\mathbb{E}[ \cdot]$ needs a domain to be defined, so notations like $\mathbb{E}[x]$ do not make sense, while $\mathbb{E}[g(x)]$ do make sense because $g(x) : \mathcal{X} \to \mathbb{R}$). ...