Ilya on LLMs understanding the world: "predicting the next token well, means that you understand the underlying reality that let to the creation of that token" Seem like the opposite view of Yann.

view is correct. @ilyasut is not. Predicting the next token well is simply a matter of learning statistical correlations of latent representations of text. The consequences of compressing this universe into something smaller is the literal definition for why it is stochastic and the literal definition for why models "dream" or "hallucinate". Existing architectures have no way of exerting intelligence by deviating from these relationships through logic and reason. Hence Yann keep repeating that children with almost no training data are superior learning machines compared to extant ML models that require as training data the entirety of all human text ever written as input.

@AlphaSignalAI Yann's argument is more about efficiency. Humans could generate an "internal model" of a new person they met with a short conversation with a person. LLMs would need 1000x more data and even then their "internal model" would have some flaws as compared to the human model.

@AlphaSignalAI Thanks for highlighting the clip! Here are the links to the full episode: YouTube: Apple Podcasts: Spotify: Transcript:

@AlphaSignalAI Source:

@AlphaSignalAI Ilya is right about this. They taught the LLMs to learn by predicting the next token. When you learn how to do a job, there's explicit things you learn and lots of implicit things you learn. You are only trying to predict the explicit things, but you learn the implicit naturally

@AlphaSignalAI So T9 understands the underlying reality of what led to the creation of the next character?

@AlphaSignalAI No you don’t “understand” that underlying reality, you merely “simulate” that underlying reality. Quite an important difference.

@AlphaSignalAI True. But that reality is nothing but a bunch of numbers. That doesn't mean that our brain doesn't do that. There are many processes in our brain that are just like that, but the road to real "understanding" is still long.

@AlphaSignalAI Just means that the probability was calculated correctly not a holistic understanding then again what is the brain but a fancy probability engine

@AlphaSignalAI But but but.. it is just a stochastic parrot!!!