Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Do diffusion models want Perlin-like noise? SDXL-turbo spends the first blocks simply creating Perlin-like noise of different frequencies, amplitudes and orientations. The model ignores the prompt conditioning during these layers.

Rudy Gilman

3,100 subscribers

59,764 views • 1 year ago •via X (Twitter)

Arts Science & Technology Education

Anya Rossi• Live Now

Private livecam show

12 Comments

Rudy Gilman1 year ago

You can ablate the conditioning prompt info during those layers. The model doesn't care. It's busy building a noise latticework. This infrastructure it's creating here is completely independent of the prompt.

Rudy Gilman1 year ago

Ken Perlin made his eponymous noise explicitly so that it would look natural when used for procedural generation of textures and terrains—it's cool that models have learned something similar! Here's link to SDXL-turbo. Will open to a layer showing the Perlin-like noise:

Rainmaker2 years ago

Can Machine Learning beat the market? Check out this post on my free Substack where I share code and commentary for an XGBoost model and a Random Forest model that both deliver powerful performances.

Ethan is in SF1 year ago

Have a section here back in the days of guided diffusion where you could init with perlin noise instead of Gaussian and skip the first whole 10-20% of steps!

Rudy Gilman1 year ago

oh nice! that was exactly my thinking, you're way ahead!

Diego Porres1 year ago

StableDiffusion yearns for Perlin

Rudy Gilman1 year ago

excellent! this was exactly the hope of reply I was hoping to get.

Peter Baylies1 year ago

👍 I wasn't the first, and I'm sure I won't be the last; but yeah, I think a little perlin noise can often help.

FeepingCreature1 year ago

If you inject randomized perlin noise at these layers, does the output break? That is, does it want that specific noise or just any noise? Or is it doing octave shifting on the input noisemap?

Rudy Gilman1 year ago

Great question. The noise maintains scale, orientation and stretch across seeds. Watch what happens when cycle through seeds below. Also note that when change the prompt nothing happens.

Sasha1 year ago

it seems to be similar to DiT study shows that conditioning mostly affects on mid blocks

Rudy Gilman1 year ago

Nice. Yes I would imagine that's what the research should show. I think often the models just aren't ready yet to bring in the semantic payload.

Related Videos

More internal optimizations on my #Dreamcast and #Gamecube game engine: • 2D Perlin noise generation is now 67% faster (59 cycles per sample) • 3D Perlin noise generation is now 40% faster (97 cycles per sample) • Further optimized worst-case heap allocations, bringing the total improvement to 40x since before my last post (yeah, allocations were that inefficient for a while) Perlin noise generation is one of the game's most resource-intensive functions. Since the Dreamcast doesn't support conventional SIMD, I optimized Perlin noise using purely scalar code. The Dreamcast's SIMD capabilities are mainly tailored for graphics, with matrix-vector multiplication and dot product being the only SIMD instructions available. These optimizations are invisible to the player, so in the meantime, here's an interesting spot I discovered while testing the game on the Dreamcast. #retrogaming #indiegamedev

More internal optimizations on my #Dreamcast and #Gamecube game engine: • 2D Perlin noise generation is now 67% faster (59 cycles per sample) • 3D Perlin noise generation is now 40% faster (97 cycles per sample) • Further optimized worst-case heap allocations, bringing the total improvement to 40x since before my last post (yeah, allocations were that inefficient for a while) Perlin noise generation is one of the game's most resource-intensive functions. Since the Dreamcast doesn't support conventional SIMD, I optimized Perlin noise using purely scalar code. The Dreamcast's SIMD capabilities are mainly tailored for graphics, with matrix-vector multiplication and dot product being the only SIMD instructions available. These optimizations are invisible to the player, so in the meantime, here's an interesting spot I discovered while testing the game on the Dreamcast. #retrogaming #indiegamedev

A Flock of Meese

23,084 views • 1 year ago

SDXL-turbo isn't given positional information—so it makes its own. You can see the positional grid forming in the first few blocks, starting from the borders and rippling inwards, carried by successive layers of 3x3 convs.

SDXL-turbo isn't given positional information—so it makes its own. You can see the positional grid forming in the first few blocks, starting from the borders and rippling inwards, carried by successive layers of 3x3 convs.

Rudy Gilman

68,684 views • 1 year ago

Quick test with SDXL Turbo, another amazing, super-fast diffusion model. It works right out of the box with diffusers unofficial demo txt2img and img2img:

Quick test with SDXL Turbo, another amazing, super-fast diffusion model. It works right out of the box with diffusers unofficial demo txt2img and img2img:

Radamés Ajna

219,854 views • 2 years ago

Clean these noise makers off the streets of Cities just like you did to the Vendors.

Clean these noise makers off the streets of Cities just like you did to the Vendors.

Shimon Aine

43,838 views • 3 months ago

Through the length of SDXL-Turbo you can see the image forming in the activations like an object taking shape in the afternoon clouds. Like David emerging from a block of granite. At two-thirds through the model you can see the image clearly. It's spatially crisp, refined. Then in the last third of the model something strange but predictable happens: the model translates this well-formed image back into noise. By the output of the U-net, the activations are again completely uninterpretable. Why does this happen? Because we're asking the model to predict the noise rather than the clean image. This is awkward. Like Michelangelo visualizing the chips of granite surrounding the David rather than the David itself. Like asking a friend to look at a cloud and imagine not a shape, but the extra bits of cloud that when removed will leave a shape remaining.

Through the length of SDXL-Turbo you can see the image forming in the activations like an object taking shape in the afternoon clouds. Like David emerging from a block of granite. At two-thirds through the model you can see the image clearly. It's spatially crisp, refined. Then in the last third of the model something strange but predictable happens: the model translates this well-formed image back into noise. By the output of the U-net, the activations are again completely uninterpretable. Why does this happen? Because we're asking the model to predict the noise rather than the clean image. This is awkward. Like Michelangelo visualizing the chips of granite surrounding the David rather than the David itself. Like asking a friend to look at a cloud and imagine not a shape, but the extra bits of cloud that when removed will leave a shape remaining.

Rudy Gilman

11,641 views • 1 year ago

In our new short course you’ll learn exactly how algorithms like DALL·E and Stable Diffusion work. Dive into the fascinating world of diffusion models and build your own model from scratch with Sharon Zhou ✈️ ICML in "How Diffusion Models Work." Sign up:

In our new short course you’ll learn exactly how algorithms like DALL·E and Stable Diffusion work. Dive into the fascinating world of diffusion models and build your own model from scratch with Sharon Zhou ✈️ ICML in "How Diffusion Models Work." Sign up:

DeepLearning.AI

56,075 views • 3 years ago

Diffusion models clicked for me when I started seeing them through the lens of particle motion. I built this interactive playground where you too can clickety-clack to understand how drift, noise, and other hyperparams control diffusion. I hereby submit this as penance for the sin of YouTube edu-tainment 😇 Link in the first comment.

Diffusion models clicked for me when I started seeing them through the lens of particle motion. I built this interactive playground where you too can clickety-clack to understand how drift, noise, and other hyperparams control diffusion. I hereby submit this as penance for the sin of YouTube edu-tainment 😇 Link in the first comment.

Julia Turc

32,207 views • 4 months ago

“There’s nothing like it.” Behind the lights, the noise, and the pressure of the Six Invitational stage. 🎙️Iain Chambers

“There’s nothing like it.” Behind the lights, the noise, and the pressure of the Six Invitational stage. 🎙️Iain Chambers

Rainbow Six Esports

35,259 views • 6 months ago

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 views • 3 years ago

“Do you regret not having Kirk Minihane in the Internet Invitational seeing the noise around twitter during that?” -Ryan Whitney “Don’t ask leading questions like that” -Dave Portnoy

“Do you regret not having Kirk Minihane in the Internet Invitational seeing the noise around twitter during that?” -Ryan Whitney “Don’t ask leading questions like that” -Dave Portnoy

Viva La Stool

240,069 views • 9 months ago

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

Tom Yeh

238,166 views • 2 years ago

If we *must* demolish a building, let's do it like the Japanese. From the bottom up! Meaning less waste, more recycling of materials, less pollution and less noise

If we must demolish a building, let's do it like the Japanese. From the bottom up! Meaning less waste, more recycling of materials, less pollution and less noise

Philip Oldfield

16,173,812 views • 1 year ago

Google Earth's data catalog is expanding. 🌍 We've rolled out new data layers to Australia, Western Europe, and the Nordics. Access critical layers like our digital elevation model, EV infrastructure, population density, heat resilience layers, and more. Make high-stakes decisions with the confidence of the richest, freshest geospatial models and imagery in the world at your fingertips.

Google Earth's data catalog is expanding. 🌍 We've rolled out new data layers to Australia, Western Europe, and the Nordics. Access critical layers like our digital elevation model, EV infrastructure, population density, heat resilience layers, and more. Make high-stakes decisions with the confidence of the richest, freshest geospatial models and imagery in the world at your fingertips.

Google Earth

12,152 views • 6 months ago

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

AK

16,062 views • 1 year ago

"Broad diffusion is really important. I want Fable – and better models – in the hands of all sorts of people. That's how you keep the balance. Odds of really bad outcomes like nationalization go up if diffusion is not as broad." Dean W. Ball, on the eve of joining @openai

"Broad diffusion is really important. I want Fable – and better models – in the hands of all sorts of people. That's how you keep the balance. Odds of really bad outcomes like nationalization go up if diffusion is not as broad." Dean W. Ball, on the eve of joining @openai

Nathan Labenz

30,661 views • 25 days ago

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,608 views • 2 years ago

Simply by modifying one line of code, I made the game feel like an entirely different game. From tactical ForHonor + Sekiro like game to be more like fast paced hack and slash game. But do I want it to be like that?

Simply by modifying one line of code, I made the game feel like an entirely different game. From tactical ForHonor + Sekiro like game to be more like fast paced hack and slash game. But do I want it to be like that?

Thousands Layered Blade: Reforged

15,386 views • 5 months ago