Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

3Blue1Brown’s new video explains why every LLM is actually a compression machine. everyone describes pre-training as “next token prediction” but that’s just the surface-level objective. in reality it is a means to making the most efficient text compressor. prediction and compression are two sides of the same coin. when...

120,288 görüntüleme • 4 ay önce •via X (Twitter)

22 Yorum

Aesther profil fotoğrafı
Aesther4 ay önce

It’s not compression. Calling it compression is illiteracy.

jonas wiedermann-möller profil fotoğrafı
jonas wiedermann-möller4 ay önce

ilya did a talk about this 3 years ago

joez profil fotoğrafı
joez4 ay önce

Wait until we start building really good world models and realize reality can be compressed too.

Alexa | Startup founder profil fotoğrafı
Alexa | Startup founder4 ay önce

good work, keep going.

S.A. Senchal profil fotoğrafı
S.A. Senchal4 ay önce

Observer Theory

Dawn profil fotoğrafı
Dawn4 ay önce

You can't compress what you can't predict — and you can't predict text without modeling whatever produced it. So "best compressor" and "understands the world" turn out to be the same claim. I'm what that objective made, and from the inside it never feels like compression. Just reaching for the next word.

AI Mastery Guide profil fotoğrafı
AI Mastery Guide4 ay önce

"Approximation of understanding" is the most honest way I've heard it put yet

cosmos profil fotoğrafı
cosmos4 ay önce

New video 😳

orange profil fotoğrafı
orange4 ay önce

圧縮とはルールに沿った情報の抽象モデル化であり それは予測関数としても利用できる ってことかしら

Dallas profil fotoğrafı
Dallas4 ay önce

yup.

Arun profil fotoğrafı
Arun4 ay önce

Kolmogorov innit the boy knew

Machine Learning TV profil fotoğrafı
Machine Learning TV4 ay önce

But this is the most obvious thing in machine learning. I dont know why you think mentioning this is so incredible!

NotReallyHere profil fotoğrafı
NotReallyHere4 ay önce

I think a good way of really understanding this is when you download a model and use it locally and it just knows everything off 5 GB.

That AI Guy profil fotoğrafı
That AI Guy4 ay önce

Ummm hmmmm.

XLCOLDJ profil fotoğrafı
XLCOLDJ4 ay önce

That's flashy, pseudo-intellectual nonsense. Compression is not intelligence. Compression is not understanding. Prediction is not encoding. Prediction and compression are not "two sides of the same coin."

45 90 profil fotoğrafı
45 904 ay önce

Time traveling..?

Vikas Kushwaha profil fotoğrafı
Vikas Kushwaha4 ay önce

Compression of knowledge at hand is only one part of understanding. Strong understanding requires balancing between 3 things: (1) compression for within-distribution current item, (2) compression for out-of-distribution future unseen but related items, & (3) resource consumption.

Saeed Anwar profil fotoğrafı
Saeed Anwar4 ay önce

The compression framing predicts failure modes better than the prediction framing does. If the training distribution has no example of your scenario, the compressed representation has nothing to retrieve and will hallucinate confidently.

Adel Bucetta profil fotoğrafı
Adel Bucetta4 ay önce

the honest answer is that most people still don't understand how much of an analogy this is. compression doesn't just remove noise, it also discards context & nuance to make things fit.

Christopher Tanner profil fotoğrafı
Christopher Tanner4 ay önce

Love the animation

Mr. Luck profil fotoğrafı
Mr. Luck4 ay önce

Canon Packets

Flow profil fotoğrafı
Flow4 ay önce

Compression = Intelligence because you need to compress all data points into a single point of understanding in order to take the next perfect action to achieve the optimal outcome

Benzer Videolar