What if it’s not just autocomplete?

The only problem with the visual neuron analogy is that is itself an oversimplification. In the human CNS there are feedback loops from 'higher' areas of processing BACK to the 'lower' areas of visual processing. So, it is quite literally NOT just simple scaled up.

hmm

When is a long form pod cast coming along?? Literally everytime you post a video I drop absolutely everything and watch the entire thing.

🙏🙏

they were never meant to replicate a brain, or consciousness, and anything you think is close is simply a digital pantomime

For every complex problem, there is an answer that is clear, simple and wrong. Thinking we understand how the brain works is hubris.

me when my entire purpose in life is edge detection

Yes many complex systems are simple things scaled up (evolution). But *what that simple thing is* matters. Just because two systems share that property does not mean they’re the same. That phrase is vague enough to describe evolution, the brain, calculators, and ant colonies.

This is kinda repeating effect complex systems have on human brain: Very often, when people study microorganisms closely, they come to stop believe in evolution because the complexity is so crazy that they cant imagine it not being designed. With AI, the simplicity and scale leads to such great mimmicking of human behavior, that people tend to believe its something more than a speaking calculator. If you show current AI to a person 200 years ago, they will believe its literally god. Its still a LLM trained on internet spam. For future generations, out believe that this is AGI will be as funny, as is the fact that in 1930 people still believed Mars is occupied by Martians. They did. Google it. "Any sufficiently advanced technology is indistinguishable from magic."

I studied Computer Vision as part of my Masters over a decade ago. Back then, AI was trying to be built using by actually encapsulating intelligence qualities such as reasoning, world model building, learning, etc. The history of humans and technology is that we use it to recreate ourselves (Terrence McKenna has a good spiel on this on yt). We had drawings and paintings of ourselves. We then made accurate photographs. Next was even more realistic - videos. Now they are using the mechanism's and footprints of our thinking, while they telling us the machine has grown legs (but you just cannot see it yet). It's all just an attempt to do what God has done. But it's not what we should be attempting. Great video btw, I think we should separate consciousness from intelligence.

Not a conclusion but a good approach to the most debated aspect of AI/LLM. I am not sure that we are able to discuss it healtyly because marketing narrative creates so much noise around it and it is indistinguishable from actual information.

The “scaling law” — the idea you can just add more simple things and fit them to the data is quite bluntly a lie. It seems elegant until you realize it is a mathematical impossibility. You simply can’t train any ML model whatsoever, with any finite amount of data, and then have it perform well on novel data. What creates advances in “AI” is neither scale nor “clever tricks” — it’s more domain knowledge in the model, and the representation. And choosing domains where the challenges are basically the same thing as the training data. Think of it like this. Computer code is really just a bunch of simple operations of a limited number arranged in a particular way. We can creare any program by just stacking up those simple things. But can you then just “train” a model to figure out what program is running from its outputs? No! The number of programs is exponential on their length and the number of programs that match your training data and then diverge completely outside of it is essentially infinite. To have any chance of fitting a program to samples you have to know *almost* everything about it. If the things you dont know are just a few parameters that aren’t coupled together in some nasty non-linear operations then sure. In general? Intractable. You can fit a polynomial to polynomial data — if the degree is low. Fitting a model that bears no semblance to how a brain works to an extremely limited sample of one of the things the brain does — and imagine it will extrapolate to new circumstances — is not only impossible. It’s a ludicrous proposition. A transformer is a good structure for an autocomplete, it has the general structure of language inserted into it from the start. It cant even learn multiplication efficiently. Let alone learn the ability to learn anything. An LLM isn’t even a good *autocomplete* if you’re trying to write something actually innovative. The reality we have no algorithm for “intelligence”. “Neural Networks” are not remotely like real neurons. Trying to fit a transformer to the text on the internet and imagining it will form its own model of the world isn’t just hopeless— it’s farsical. Ultimately the entire “AGI” enterprise is predicted on a fundamental disdain for human intelligence (and animal too). You have to believe our intellect is such a trivial phenomenon that you can brute force figuring out how it works. The reason LLMs *appear* to work — if you dont test them critically — is that there is an ungodly amount of text on the internet and the vast majority of questions you ask have already been asked and answered thousands of times over, and the LLM can do a little interpolation between them to fit the specifics. With exponential amounts of data and processing you can do that to the point where distinguishing between an LLM and real intelligence becomes too expensive for doing manually. But deploy anything generated by LLMs at scale and the real world will soon throw at it something that makes the difference obvious — and very dangerous.

What differentiates a person from an animal (or an LLM) is the unique combination of intellect plus will. Intellect alone is not enough enough to constitute a person. LLMs will never be persons because they can never have a will.

This is a great video! It’s an honest exploration of complex ideas in simple terms. Less cynicism and misanthropy, more good critical dialogue. Thanks for making it and sharing it!

There is no wall.

Many a mickle maks a muckle, we all know that. But for the qualitative step change we have to lean on Clarke's 3rd law "any sufficiently advanced technology is indistinguishable from magic" Or even better, Hegel's law of quantity to quality "when the quantitative accumulation of a thing crosses a certain threshold (its critical limit), it triggers a sudden, fundamental qualitative change in its identity"

Also worth reading

"Thinking fast and slow" is the book you want to read. LLMs only can think "fast." Meaning—that's a different thing altogether. And, yes: I assume 80% can survive on fast thinking. And it is even gender-specific, too. You know, many of us are just finite automatons. 😄

From "not knowing what to tell my children anymore" to "i want to believe it so bad, there MUST be something to it" ... I am waiting for your feeling sorry for the cat that had to undergo those experiments, for when you know you yourself are being experimented on. !!

my brain has a Mo neuron it fires about once a week when twitter movement suddenly shifts.

Just like LeCun said, AI as it is right now doesn't understand the expected outcome or the outcome it produces, and so as long as this is true, it is just autocompletion.

Mo, doesn't @RichardSSutton say that LLM's are *not* a case of the bitter lesson, as they rely too much on human-crafted data? I feel like you might want to watch Sutton's interview on Dwarkesh where he says this:

hmmmm but it is! Earth Surface is ~75% covered in water, still it is a planet not a water orb. I understand how people push back on AI, but matrix multiplication and predictive algorithm acts like electronic circuits and voltage and current calculations than really any living being brain or thinking!

Your videos are good and opinionated but brother you have to stop the coping.

This kind of responses are a waste of time if first there is not a proper response to know what is really the Intelligence itself ? and then we could follow with the possibity of an Artificial Intelligence.

@TrueAIHound thoughts?

The only thing that makes AI intelligent is the perception of the recipient—that is, the person who is writing to the large language model (LLM).

Simple thing. A conscious experience requires continuity. While you can argue, that the continuity lives inside the context window, that's not technically true. You are just pushing that context through the static networks. I'd entertain it if the weights react dynamically.

Imagine a hypothetical program containing billions of if conditions - a sophisticated text-processing engine. It receives input and generates natural-language responses, yet it is still just a massive chain of if-else statements. Could such a system be conscious or intelligent?

The boundary between hardware and software in the brain is blurry. There is also mixed selectivity e.g. those “simple cells” code for color, contrast and other criteria. And that coding changes across time aka representational drift. Does the brain predict the world or shape it?

I don't see a reason to get bummed out. At least, no more bummed out than when I look in a mirror and see my reflection. Cause that's what AI is. A reflection. No matter how high fidelity, a mirror is a mirror. The danger is falling in love with the reflection.

IMO we emergent structures are a thing in the models, these still language heuristics. The “evolution” driver that emerges them is purely based on language and structure/relationships. Bio net are action consequences chain predictors, they are by arch. temporal processors.

LLMs catastrophic flaws: 1) In 1959 biologist Humberto Maturana showed Artificial Neurons (integer arrays) do not model the chemica/physical properties of neurons 2) Language is something people learn only after they "understand" the world. To LLMs language are integers (tokens).

Isn't this basically physics too? Things built on smaller and smaller components until you get to the quantum level and reality is just a localized state being observed?

Which is real? The mimic or the thing its mimicking?

There are a lot of these biological analogies. Shared layers in MoE are very similar to the corpus callosum. The deeper you get in the more fascinating it gets. Once we have functional short & long term memory systems the next step will be imagination -- on demand simulated training environments to provide fast fine tuning on weights (task, action or situational specific fine tuning).