Mixture of Experts # There is a gate that opens a subset of the experts, and the output is the weighted sum of the outputs of the experts. The weights are computed by a gating network. One problem is load balancing, non uniform assignment. And there is a lot of communication…
We have resources, but need to know how to assign these to the jobs that need them. This note presents some of the most common resource management policies for cloud clusters. Introduction to cluster management # How can we allocate the resources in a cluster in an efficient…
This allows us to extend the memory hierarchy (see Memoria ) that we have today. The problem is that we have heterogeneous access patterns specifications and hardware. One of the main trends is disaggregation : we want to be able to scale different resources independently.…
Queueing theory is the theory behind what happens when you have lots of jobs, scarce resources, and subsequently long queues and delays. It is literally the “theory of queues”: what makes queues appear and how to make them go away. This is basically what happens in clusters,…
Normalizing flows have both latent space and can produce tractable explicit probability distributions (closer to Autoregressive Modelling , they have tractable distributions, but not a latent space). This means we are able to get the likelihoods of a certain sample. This…
Definition of problems # Object detection # Bisogna trovare all'interno dell'immagine quali siano gli oggetti presenti, e in più vogliamo sapere dove siano quindi utilizzare una bounding box per caratterizzarli sarebbe buono. Object segmentation # È riuscire a caratterizzare…
On Autoregressivity # The main idea of autoregressivity is to use previous prediction to predict the next state. The Autoregressive property # Autoregressive models model a joint distribution of aleatoric variables by assuming a chain rule like decomposition: p ( x ) = i = 1 ∏ n…
Introduction: a neuron # I am lazy, so I'm skipping the introduction for this set of notes. Look at Andrew Ng's Coursera course for this part ( here are the notes). Historical paper is (Rosenblatt 1958) . One can view a perceptron to be a Log Linear Models with the temperature…
The perceptron is a fundamental binary linear classifier introduced by (Rosenblatt 1958) . It maps an input vector x ∈ R n to an output y ∈ { 0 , 1 } using a weighted sum followed by a threshold function. Introduction to the Perceptron # A mathematical model # Given an input…
3D representations # In this section, we present some of the most common 3D representations used in computer graphics and computer vision. Each representation has its own advantages and disadvantages, and the choice of representation often depends on the specific application.…