Загрузка видео...

Не удалось загрузить видео

На главную

Deep dreams on modern LLMs are so cool (optimizing an image to maximize P(target caption)) Gemma 12B (left) has no vision encoder — it reads pixels like token embeddings — and stamps recognisable objects around the canvas. E4B (right) has one, and drifts to texture instead.

95,378 просмотров • 4 месяцев назад •via X (Twitter)

Комментарии: 18

Фото профиля Matt Henderson
Matt Henderson4 месяцев назад

of course 12B is a bigger model, so it is not a fully fair comparison. But it's an interesting glimpse into how these models see.

Фото профиля Matt Henderson
Matt Henderson4 месяцев назад

these runs are doing gradient descent to optimize the probability of the caption given an input image, represented as an FFT. Higher frequencies are suppressed in the FFT, and the RGB colours are decorrelated and normalized to a target standard deviation, then tanh

Фото профиля Matt Henderson
Matt Henderson4 месяцев назад

final frames

Фото профиля Matt Henderson
Matt Henderson4 месяцев назад

Gemma 4 12B’s Jungle Waterfall dream

Фото профиля rurounigit
rurounigit4 месяцев назад

VQGAN+CLIP. Amazing times:)

Фото профиля Noah Vandal
Noah Vandal4 месяцев назад

WOMBO was awesome

Фото профиля Ariel Ekgren
Ariel Ekgren4 месяцев назад

Used to do this back in the days. So many tricks to improve quality of img. Not a fan of the FFT step. Just signing the grads gets you far. Adding a small gaussian blur or offsetting by 1 pixel back and forth. Also adding a 3x3 transform on RGB channels. Never found golden rule

Фото профиля hεt 🧬
hεt 🧬4 месяцев назад

Remember when AI was this 😁 was not that far away

Фото профиля netrunner
netrunner4 месяцев назад

Gemma 12b reads pixels as token embeddings, the one with the vision encoder reads them as texture first

Фото профиля Ethan
Ethan4 месяцев назад

Could be interesting maximizing individual features identified by SAE as well

Фото профиля Sambyal🇮🇳
Sambyal🇮🇳4 месяцев назад

How different they are compared to smaller models like ResNets, etc? Are there any significant insights?

Фото профиля AI Mastery Guide
AI Mastery Guide4 месяцев назад

The vision encoder difference really shows in how each one drifts

Фото профиля cheez | Local LLM Benchmarks
cheez | Local LLM Benchmarks4 месяцев назад

Here is an image i did with DeepDaze on a 2060super. the time stamp on the file is April 2021 the prompt was like "tripping on lsd in the mountains"

Фото профиля Max
Max4 месяцев назад

can you optimize the same image using both of them?

Фото профиля Lexeme
Lexeme4 месяцев назад

I forgot all about this.

Фото профиля Bruno Kodjaoglanian
Bruno Kodjaoglanian4 месяцев назад

Que incrível

Фото профиля Awe of Machines
Awe of Machines4 месяцев назад

Is it possible to get audio by doing this? IIRC there is also no audio encoder?

Фото профиля Tim
Tim4 месяцев назад

hierarchically please :( I need to know if things look even more cool that way... I.e. start at 128p, dream, upscale, dream, upscale...

Похожие видео