Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Deep dreams on modern LLMs are so cool (optimizing an image to maximize P(target caption)) Gemma 12B (left) has no vision encoder — it reads pixels like token embeddings — and stamps recognisable objects around the canvas. E4B (right) has one, and drifts to texture instead.

95,378 Aufrufe • vor 4 Monaten •via X (Twitter)

18 Kommentare

Profilbild von Matt Henderson
Matt Hendersonvor 4 Monaten

of course 12B is a bigger model, so it is not a fully fair comparison. But it's an interesting glimpse into how these models see.

Profilbild von Matt Henderson
Matt Hendersonvor 4 Monaten

these runs are doing gradient descent to optimize the probability of the caption given an input image, represented as an FFT. Higher frequencies are suppressed in the FFT, and the RGB colours are decorrelated and normalized to a target standard deviation, then tanh

Profilbild von Matt Henderson
Matt Hendersonvor 4 Monaten

final frames

Profilbild von Matt Henderson
Matt Hendersonvor 4 Monaten

Gemma 4 12B’s Jungle Waterfall dream

Profilbild von rurounigit
rurounigitvor 4 Monaten

VQGAN+CLIP. Amazing times:)

Profilbild von Noah Vandal
Noah Vandalvor 4 Monaten

WOMBO was awesome

Profilbild von Ariel Ekgren
Ariel Ekgrenvor 4 Monaten

Used to do this back in the days. So many tricks to improve quality of img. Not a fan of the FFT step. Just signing the grads gets you far. Adding a small gaussian blur or offsetting by 1 pixel back and forth. Also adding a 3x3 transform on RGB channels. Never found golden rule

Profilbild von hεt 🧬
hεt 🧬vor 4 Monaten

Remember when AI was this 😁 was not that far away

Profilbild von netrunner
netrunnervor 4 Monaten

Gemma 12b reads pixels as token embeddings, the one with the vision encoder reads them as texture first

Profilbild von Ethan
Ethanvor 4 Monaten

Could be interesting maximizing individual features identified by SAE as well

Profilbild von Sambyal🇮🇳
Sambyal🇮🇳vor 4 Monaten

How different they are compared to smaller models like ResNets, etc? Are there any significant insights?

Profilbild von AI Mastery Guide
AI Mastery Guidevor 4 Monaten

The vision encoder difference really shows in how each one drifts

Profilbild von cheez | Local LLM Benchmarks
cheez | Local LLM Benchmarksvor 4 Monaten

Here is an image i did with DeepDaze on a 2060super. the time stamp on the file is April 2021 the prompt was like "tripping on lsd in the mountains"

Profilbild von Max
Maxvor 4 Monaten

can you optimize the same image using both of them?

Profilbild von Lexeme
Lexemevor 4 Monaten

I forgot all about this.

Profilbild von Bruno Kodjaoglanian
Bruno Kodjaoglanianvor 4 Monaten

Que incrível

Profilbild von Awe of Machines
Awe of Machinesvor 4 Monaten

Is it possible to get audio by doing this? IIRC there is also no audio encoder?

Profilbild von Tim
Timvor 4 Monaten

hierarchically please :( I need to know if things look even more cool that way... I.e. start at 128p, dream, upscale, dream, upscale...

Ähnliche Videos