Video wird geladen...
Video konnte nicht geladen werden
Deep dreams on modern LLMs are so cool (optimizing an image to maximize P(target caption)) Gemma 12B (left) has no vision encoder — it reads pixels like token embeddings — and stamps recognisable objects around the canvas. E4B (right) has one, and drifts to texture instead.
95,378 Aufrufe • vor 4 Monaten •via X (Twitter)
18 Kommentare

of course 12B is a bigger model, so it is not a fully fair comparison. But it's an interesting glimpse into how these models see.

these runs are doing gradient descent to optimize the probability of the caption given an input image, represented as an FFT. Higher frequencies are suppressed in the FFT, and the RGB colours are decorrelated and normalized to a target standard deviation, then tanh

final frames

Gemma 4 12B’s Jungle Waterfall dream

VQGAN+CLIP. Amazing times:)

WOMBO was awesome

Used to do this back in the days. So many tricks to improve quality of img. Not a fan of the FFT step. Just signing the grads gets you far. Adding a small gaussian blur or offsetting by 1 pixel back and forth. Also adding a 3x3 transform on RGB channels. Never found golden rule

Remember when AI was this 😁 was not that far away

Gemma 12b reads pixels as token embeddings, the one with the vision encoder reads them as texture first

Could be interesting maximizing individual features identified by SAE as well

How different they are compared to smaller models like ResNets, etc? Are there any significant insights?

The vision encoder difference really shows in how each one drifts

Here is an image i did with DeepDaze on a 2060super. the time stamp on the file is April 2021 the prompt was like "tripping on lsd in the mountains"

can you optimize the same image using both of them?

I forgot all about this.

Que incrível

Is it possible to get audio by doing this? IIRC there is also no audio encoder?

hierarchically please :( I need to know if things look even more cool that way... I.e. start at 128p, dream, upscale, dream, upscale...

