Notes

We first start with some design goals for the language Language Design Principles Simplicity Syntax and semantics can easily be understood by users and implementers of the language Expressiveness: Language can (easily) express complex processes and structures, conflicting with simplicity. Safety: Language discourages errors and allows errors to be discovered and reported, ideally at compile time Modularity: Language allows modules to be type-checked and compiled separately Performance: Programs written in the language can be executed efficiently Productivity: Language leads to low costs of writing programs like Python. Backwards Compatibility: Newer language versions work and interface with programs in older versions (dependency injection for examples helps for this.) ...

Il Tipo di Dato Astratto Introduzione Per questi tipi di dato non ci interessa di sapere cosa ci sia sotto (storato come bit? storato come sabbia boh), ci interessa solamente che abbia quei metodi, che possiamo in un certo senso identificare come la sua capsula, opaca in questo caso. Quando si può andare a modificare solamente attraverso questo metodo potrei dire che sia safe collegato alla Algebra dei tipi, nel senso che vengono soddisfatte sempre le proprietà del tipo. ...

Introduzione Tipi Monomorfici Quando non posso utilizzare un tipo come parametro. Ossia non possiamo definire una funzione generica. Polimorfismo Polimorfismo, come dice il nome, significa avere tante forme, in questo caso tanti tipi. Ma avere tanti tipi non è una cosa ambigua? Questa cosa si risolve solitamente a compile time (facendo checks di sottotipo, oppure dispatch della funzione corretta). A program part is polymorphic if it can be used for objects of several classes ...

$$ p(\theta \mid x_{1:n}, y_{1:n}) = \frac{1}{z} p(y_{1:n} \mid \theta, x_{1:n}) p(\theta \mid x_{1:n}) \approx q(\theta \mid \lambda) $$For Bayesian Linear Regression we had high dimensional Gaussians which made the inference closed form, in general this is not true, so we need some kinds of approximation. Laplace approximation Introduction to the Idea $$ \psi(\theta) \approx \hat{\psi}(\theta) = \psi(\hat{\theta}) + (\theta-\hat{\theta} ) ^{T} \nabla \psi(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta} ) ^{T} H_{\psi}(\hat{\theta})(\theta-\hat{\theta} ) = \psi(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta} ) ^{T} H_{\psi}(\hat{\theta})(\theta-\hat{\theta} ) $$ We simplified the term on the first order because we are considering the mode, so the gradient should be zero for the stationary point. ...

XOR operation È una operazione binaria abbastanza semplice però ci sarà importante per andare ad analizzare dei cifrari di un certo genere. Come il ONE TIME PAD che faremo fra poco in OTP and Stream Ciphers. Teorema cifratura con XOR Prendiamo $X$ una variabile aleatoria in $\left\{ 0,1 \right\}^{n}$ uniforme, sia $Y$ una variabile aleatoria su uno stesso dominio come vogliamo. Tali per cui $X, Y$ siano indipendenti Allora avremo che $C = X \oplus Y$ è una variabile aleatoria uniforme. ...

Introduzione alla Randomicità Questo è principalmente basato su (Li & Vit{'a}nyi 2019) Capito 1.9 Sembra che la nozione di random sia alla fine una cosa molto profonda. Per esempio, un caso lampante che le definizioni non funzionano nel caso di numeri trascendenti è che catalogano i numeri di $\pi$ come se fossero casuali, mentre in realtà possono essere trovati mediante procedimenti precisi. È una distinzione filosoficamente molto interessante. Alla fine sembra ci sia un link molto diretto con la crittografia, si può vedere (Stinson 2005). ...

Introduction to Inheritance Difference between Inheritance and Subtyping We say that a subclass is inheritance and subtyping. We have studied subtyping in Typing and Subtyping and it entails mainly two things: Liskov Substitution Polymorphism of the types While inheritance is just the code reuse and specialization. Another example of code reuse can be the “has-a” relation composition, which couples two objects together. With aggregations, we can also see delegation calls, when you have a class, and call a method of that class. ...

Mixture of Experts There is a gate that opens a subset of the experts, and the output is the weighted sum of the outputs of the experts. The weights are computed by a gating network. One problem is load balancing, non uniform assignment. And there is a lot of communication overhead when you place them in different devices. LoRA: Low-Rank Adaptation We only finetune a part of the network, called lora adapters, not the whole thing. There are two matrices here, a matrix A and B, they are some sort of an Autoencoders, done for every Q nd V matrices in the LLM attention layer. The nice thing is that there are not many inference costs if adapters are merged post training: ...

First, the brain is organized into functionally specific areas, and second, neurons in different parts of the vertebrate nervous system, indeed in all nervous systems, are quite similar. Small comparison with Computers A gross observation between computer’s transistors and human neurons is that there a big difference of numbers: trillions of transistors vs billions of neurons. 6 orders of magnitude frequency difference (Ghz versus 1kHz for neurons). Many many neural types and different types of connections. And the digital vs analog and chemical modes of communication. Parallel processor abilities. Fixed vs plastic architectures But this is comparing with transistors with one higher level object, so this comparison might not be completely fair. They are very different from this point of view. And only some brain areas are similar to real neural networks. ...

At the time of writing, the compute requirements for machine learning models and artificial intelligence are growing at a staggering rate of 200% every 3.5 months. Interest in the area is being quantified as 10k papers per month on the topic, while dollar investments on compute (energy, cooling, sustainability of compute in general) have had a hard time keeping up with the continuous new requests. Image from here ...