Ilya Sutskever's has a bold take. LLMs are doing much more than predicting the next word. They are learning our world model. Text is a projection of the world.

Source:

How is this a bold take? It's literally what it is. Text isn't a projection of the world, information is. Everything is information, and language is a means to distill information into a text form.

I don't think it's a bold take I think anyone who spends a bit of time thinking about how LLMs actually work has to come to that conclusion In order for LLMs to predict the next world really, really well they have to develop some sort of understanding of the world, including some reasoning capabilities

@AlphaSignalAI Looks more like a bald take if you ask me

@AlphaSignalAI That's not bold, this is 𝗯𝗼𝗹𝗱.

Bold how? This is literally how all of ML/AI works. It's simply hyperbole for marketing impact to claim it's "the world". Given all the text ever written by humans, and a large enough model, it's obvious that next-token prediction would be proficient in generating text. It's also obvious that such a model would have no real logic and reasoning capabilities required to be intelligent and hence pose no threat whatsoever to humanity.

@AlphaSignalAI Really distracting hairdo...

@AlphaSignalAI 1. Not an original take. 2.He uses learning as thought it implies understanding. A wax imprint of a song has ‘learned’ the song, but it doesn’t entail any understanding of it. No more than a memory foam pillow ‘understands’ your head.

@AlphaSignalAI Would love to hear @ylecun and @geoffreyhinton thoughts on this.

Ilya Sutskever's perspective on Large Language Models (LLMs) like GPT-3 and its successors is indeed thought-provoking. His view suggests that these models are not just simple tools for predicting the next word in a sentence but are, in fact, developing a deeper understanding or "model" of the world. The idea that "text is a projection of the world" implies that the vast amount of text data fed into these models encapsulates a representation of human knowledge and experience. Through processing and learning from this text, LLMs are thought to acquire a form of understanding or representation of the world, albeit in a manner that's fundamentally different from human cognition. However, it's important to note that while LLMs can mimic certain aspects of understanding and can generate coherent and contextually appropriate responses, their "knowledge" is limited to patterns found in the data they were trained on. They lack true understanding or consciousness and do not have experiences or awareness. Their "world model" is a statistical representation based on language patterns, rather than a conscious or intentional understanding of reality. This perspective opens up fascinating discussions about the capabilities and limitations of AI, and the nature of understanding and intelligence.