Stein Variational Gradient Descent (SVGD) is a particle-based approach... to variational inference. SVGD iteratively transports a set of particles to match a target distribution by applying a form of functional gradient descent that minimizes the KL divergence.show more

Langevin Monte Carlo allows you to draw samples from... a probability distribution using its log gradient ∇ log p(x). By performing a sort of gradient ascent with noise you can navigate around the distribution. Langevin MC is heavily related to modern diffusion models.show more

Oldies but goldies: Y Nesterov, A method for solving... a convex programming problem with convergence rate O(1/k^2). Improves the 1/k rate of vanilla gradient descent to 1/k^2.show more

Poljak heavy ball and Nesterov methods speed up gradient... descent by introducing momentum. Corresponding to second order ODEs.show more

Oldies but goldies: B Poljak, Some methods of speeding... up the convergence of iteration methods, 1964. Introduces momentum methods to speed up gradient descent schemes.show more

Neat li'l #CSS gradient trick: A gradient circular chart... effect 👀 1. a gradient mask 2. a conic gradient all the details on my blogshow more

Flow Matching aims to learn a "flow" that transforms... a simple source distribution (e.g. Gaussian) to an arbitrarily complex target distribution. This video shows the evolution of the marginal probability path as a source distribution is transformed to a target distribution.show more

You can right-click on a gradient fill in the... gradient panel to redistribute gradient stops evenly in @Figma Design. (This used to be triggered previously with a double click.) #FigmaTipshow more

Introducing "Plant gradient map"✨ This gradient set is perfect... for adding a unified coloring to your forest illustrations. Use it on grayscale drawings for the best result and choose from over 30 gradients🌳 🔽Details here:show more

When optimization problems have multiple minima, algorithms favor specific... solutions due to their implicit bias. For ordinary least squares (OLS), gradient descent inherently converges to the minimal norm solution among all possible solutions.show more

Our book "Generative AI and Stochastic Thermodynamics: A Tale... of Free Energies" is out this month. With Sirui Lu and Max Welling I'm posting about topics it covers. Yesterday: heat and work in variational EM. Today: the variational free energy, and why the ELBO is one.show more

VARIATIONAL MIGHT DELAY TGE. AND THAT COULD BE BULLISH... Polymarket gives Variational a 77% chance to launch a token by December 31. By November 30? Only 18%, and dropping. I think the market is too confident. Launching a token into current sentiment is a risk. Weak demand, thin liquidity, farmers ready to dump on day one. Nobody wants their TGE to become a chart that only goes down. There's a real chance Variational pushes TGE back another 1-2 quarters. And that might be the best outcome for farmers: > more time to farm = more points > room to give a bigger % of supply to the airdrop > time to raise more capital at a better valuation > more bullish catalysts before launch: new markets, partnerships, product updates > launching into a strong market, not a weak one A rushed TGE in a bad market helps nobody. A delayed TGE with a bigger drop, more funding and better timing helps everyone holding points. Pure speculation, nothing confirmed. If it happens, don't panic. Keep farming.show more

“The descent of Ganga is the descent of grace,... guided by Shiva.” Maghi Purnima: Today Divine darshan of Ma Ganga appearing on your timeline💦show more

That gradient shift is wilddd. Insane attention to detail... by Oğuz Yağız Karashow more

Drones followed by a wave of missiles. Tehran is... applying the drone approach to disperse air defenses and facilitate hypersonic missile wave missions.show more

This is Gandikota, Andhra Pradesh straight out of Avatar’s... Pandora. To all Tamilians: this place has a Tamil connection. It was ruled by the Renati Cholas, a dynasty that claimed descent from our great Tamil Chola king, Karikala.show more

[VAE] by Hand ✍️ A Variational Auto Encoder (VAE)... learns the structure (mean and variance) of hidden features and generates new data from the learned structure. In contrast, GANs only learn to generate new data to fool a discriminator; they may not necessarily know the underlying structure of the data. The International Conference on Learning Representations (ICLR) this year announced its first ever "Test of Time Award" to recognizes the VAE paper, published 10 years ago. This exercise demonstrates how to calculate a VAE by hand. [1] Given: ↳ Three training examples X1, X2, X3 ↳ Copy training examples to the bottom ↳ The purpose is to train the network to reconstruct the training examples. ↳ Since each target is a training example itself, we use the Greek word "auto" which means "self." This crucial step is what makes an autoencoder "auto." [2] Encoder: Layer 1 + ReLU ↳ Multiply inputs with weights and biases ↳ Apply ReLU, crossing out negative values (-1 -> 0) [3] Encoder: Mean and Variance ↳ Multiply features with two sets of weights and biases ↳ 🟩 The first set predicts the means (𝜇) of latent distributions ↳ 🟪 The second set predicts the standard deviation (𝜎) of latent distributions [4] Reparameterization Trick: Random Offset ↳ Sample epsilon ε from the normal distribution with mean = 0 and variance = 1. ↳ The purpose is to randomly pick a offset away from the mean. ↳ Multiply the standard deviation values with epsilon values. ↳ The purpose is to scale the offset by the standard deviation. [5] Reparameterization Trick: Mean + Offset ↳ Add the sampled offset to predicted mean ↳ The result are new parameters or features 🟨 as inputs to the Decoder. [6] Decoder: Layer 1 + ReLU ↳ Multiply input features with weights and biases ↳ Apply ReLU, crossing out negative values. Here, -4 is crossed out. [7] Decoder: Layer 2 ↳ Multiply features with weights and biases ↳ The output is Decoder's attempt to reconstruct the input data X from reparameterized distributions described by 𝜇 and 𝜎. [8]-[10] KL Divergence Loss [8] Loss Gradient: Mean 𝜇 ↳ We want 𝜇 to approach 0. ↳ A lot of math called SGVB simplifies the calculation of loss gradients to simply 𝜇 [9,10] Loss Gradient: Stdev 𝜎 ↳ We want 𝜎 to approach 1. ↳ A lot of math simplifies the calculation to 𝜎 - (1/ 𝜎) [11] Reconstruction Loss ↳ We want the reconstructed data Y (dark 🟧) to be the same as the input data X. ↳ Some math involving Mean Square Error simplifies the calculation to Y - X.show more

In 2023, neuroscientists uncovered that our brain distinguishes reality... from imagination on a gradient, with a threshold marking the jump from one state to another. Learn about the biological breakthroughs of 2023:show more

I took album cover of GNX by Kendrick Lamar... and turned it into a 3D particle model in Threejs with TSL Used Grok 4 by xAI to improve the noise that moves particles.show more