Do diffusion models want Perlin-like noise? SDXL-turbo spends the first blocks simply creating Perlin-like noise of different frequencies, amplitudes and orientations. The model ignores the prompt conditioning during these layers.

You can ablate the conditioning prompt info during those layers. The model doesn't care. It's busy building a noise latticework. This infrastructure it's creating here is completely independent of the prompt.

Ken Perlin made his eponymous noise explicitly so that it would look natural when used for procedural generation of textures and terrains—it's cool that models have learned something similar! Here's link to SDXL-turbo. Will open to a layer showing the Perlin-like noise:

Can Machine Learning beat the market? Check out this post on my free Substack where I share code and commentary for an XGBoost model and a Random Forest model that both deliver powerful performances.

Have a section here back in the days of guided diffusion where you could init with perlin noise instead of Gaussian and skip the first whole 10-20% of steps!

oh nice! that was exactly my thinking, you're way ahead!

StableDiffusion yearns for Perlin

excellent! this was exactly the hope of reply I was hoping to get.

👍 I wasn't the first, and I'm sure I won't be the last; but yeah, I think a little perlin noise can often help.

If you inject randomized perlin noise at these layers, does the output break? That is, does it want that specific noise or just any noise? Or is it doing octave shifting on the input noisemap?

Great question. The noise maintains scale, orientation and stretch across seeds. Watch what happens when cycle through seeds below. Also note that when change the prompt nothing happens.

it seems to be similar to DiT study shows that conditioning mostly affects on mid blocks

Nice. Yes I would imagine that's what the research should show. I think often the models just aren't ready yet to bring in the semantic payload.