SDXL-turbo isn't given positional information—so it makes its own. You can see the positional grid forming in the first few blocks, starting from the borders and rippling inwards, carried by successive layers of 3x3 convs.

The model spends the first few layers creating this positional information and refining its noise lattice (more on that soon). It ignores the conditioning text during these stages because it's occupied fashioning for itself the infrastructure we didn't see fit to provide.

You can see when the positional grid is complete because it's clear in the activations. You can also see it in the attention patterns. SDXL-Turbo is now able to implement attention patterns that rely on positional information.

Here I share an XGBoost model that delivers a 25% CAGR with minimal drawdown on Visa stock. In this free Substack post I share code and commentary for a powerful Machine Learning strategy that delivers powerful returns.

Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

this looks excellent, haven't seen before thank you!

ALL the conv based arch with zero padding and enough layers WILL introduce implicit positional information Bcuz the kernel is not symetric in each axis, and the zero padding provide the "edge/bound" info.

Yes absolutely! I'm just noting that it's a lot of work to have to create your own positional information using only that bootstrap. Each 3x3 conv can only move information so far in a single layer, so we need many layers before this can cover the grid.

so cool

Nice! For SD-1.5/2.1/turbo, which have a similar architecture, its also quite well extractable using a linear probe (~90% accuracy in the lower unet layers)

that looks convoluted

Which visualizer is used for this kind of interactive plot?

This is a tool I'm building to make neural networks visual and interactive