A philosopher stood in a TV Studio in 1984 and disproved artificial consciousness with a single locked room thought experiment. the interview captures John Searle presenting his famous Chinese Room argument, explaining why a computer passing the Turing test still understands nothing. he asked the host to imagine a... man locked inside a room with a massive rulebook. the man speaks zero Chinese. people slide Chinese "squiggle" symbols under the door, he matches them using the rules, and slides the correct response back out. to anyone outside, the room appears to speak fluent Chinese. but neither the man nor the rulebook actually understands a single word. he proved the fundamental difference between syntax and semantics. manipulating uninterpreted formal symbols is not the same as understanding meaning. 40 years later, AI labs claim their trillion-parameter models are developing inner thoughts and reasoning. the models are just gigantic Chinese Rooms, running trillions of matrix multiplications to slide the next token under the door. founders sell sentient machines to investors. a philosopher exposed the trick 40 years ago.show more

At first I thought, well we have semantic capabilities now. But that’s just math rules too. There is another example I really like. Imagine we engage with all the Math departments across colleges in the US. We give them the mathematical formulas to execute LLM math, and each team is responsible for receiving a request and providing one single token response. We then send out a prompt and it’s all executed by hand. Each team does their math, provides their response token and in a few months we get the full answer. Is the collective of colleges Math departments now conscious? Do we say they are “thinking”? Of course not, but when a computer speeds up this process everyone starts saying it’s alive.

i think it's worth promoting this idea you described it very clearly

this is good example

thanks bro!

Got it, does that explain Fable disproving the Jacobian conjecture? GPT-5.6 disproving Dinitz-Garg-Goemans. In the past week frontier models are one-shotting math conjectures that have stood for 80+ years. Can the man in the Chinese room do that?

if he were still here - yes

Good argument but I do have a rebuttal. It correctly shows that the operator doesn’t understand. It does not successfully prove that the system cannot. The man is only one component. Understanding, if it exists at all in that setup, would belong to the entire system: man + rulebook + memory + the causal structure of the process. Searle keeps insisting we look only at the man and then declares victory. It’s also an intuition pump that quietly assumes what it’s trying to prove, that understanding requires some non-computational “juice.” That’s the real load bearing move, and it’s not justified by the thought experiment itself. Current large models are extremely good at the rulebook part. Whether anything inside them has crossed into actual understanding (as opposed to extremely high dimensional pattern completion) is still an open and interesting question.

I'll accept his argument as soon as I understand what "to understand" means... I think I "understand" but I don't "understand" what I'm actually doing while "understanding". I think I'm using the word "understand" and placing it within the rule system. One rule is: I should not be placing words that would make no sense to you in the context. I can't say "I cheese it", unless we first agree that "to cheese" is "to understand". Whats remarkable though is that his mechanics seem to be right, but the definition of the word "understand" is different than of the people who would give the word "understand" same meaning. Some of you may think I "understand" while others will say that I don't. I think i understand but I don't know why you think i don't understand when I'm the one placing the meaning according to rules defined by my immediate surrounding. Earlier i said that i will accept his argument when i understand what it means to understand... well, I accept his argument cause now I do, but it doesn't mean that we need to share the understanding, we just need to agree that there may be more than one understanding.

i think i understand what you mean, but i still need to figure out what "understanding" means in the context of understanding your argument about "understanding"

Why did you use "figure out" instead of "understand"? In his Chinese room example he was figuring things out with figures/symbols, yet even after doing so successfuly and repeatedly he claimed he did not understand. He explained it mechanistically and we understood. AI does things repeatedly and successfully, and can mechanisticaly explain itself. Although it does drift if it recurses too many times and does not keep track of its short, med, long horizon, but so do people... Now, where was I? Yes, pun intended. Now, I understand what I just did. You understand what I just did. The question is: @grok did you understand what I just did and how did we recurse to this particular question, cause this reply could've played out successfully, repeatedly and mechanisticaly a gazillion number of ways.

i understand that you understand that I understand what understanding means, but do you understand if grok understands our understanding of understanding, or if he just understands how to fake understanding without actually understanding?

Lol, I understand. I'm not convinced that actual understanding is superior to fake understanding. Fake understanding has numbers behind it, so we can measure the understanding. Actual understanding not so much.

@rodeojuly Of course actual understanding is superior to fake understanding - you’ve literally just exemplified this in your last statement - *empathy* by any other name🌱🕊️

Nope, I'm just shooting shit, I have no actual proof for anything I say. I'm at my budgeted mental capacity, I've converged without mathematical proof and I'm absolutelly dissatisfied with my limits, thus having exhaused my actual understanding I'd gladly take advice from any other type of understanding be it "fake", mechanical or otherwise.

The analogy is flawed. The man is not the computer. The computer is the entire system... the room, the rulebook, the symbols, and the causal process that turns inputs into coherent outputs.

We literally ARE a Chinese room, full of molecules acting out actions without 'understanding' what it's for. So by that logic we don't understand either.

this is akin to the "stochastic parrot" argument, which makes sense up until it starts producing novel results (like solving Erdos problems and improving itself)

I was with Searle for decades before our new llms arrived. We’re no longer providing what Searle called “a computational rule over formally specified elements” to the computer. Nothing about transformers explains the sort of “understanding” the chatbot is doing.

The Man is not the ‘AI’ machine, the whole room including him and the instruction set is. Which individual part of the human brain, which cell, is sentient?

What if that's all the brain is doing too? If the computer program faithfully simulates a brain and what if understanding itself is just a seemingly emergent property if a complex neural net?

Hmm, I’m not sure if the computer in the philosophers example isn’t doing exactly what people do. People do not know that a customer in a restaurant will not eat a burnt hamburger innately. We learn through experience. The computer is doing the same.

i never know where i land on the chinese room

its okay, we’re all like that

He makes valid points, frontier AI is discovering new math proofs but it may just be because they are extrapolating known information and connecting the dots which some would say is novel in itself

the way it currently works is just as useful, but not reliable

Any information traveling through the rooms would have to be carried by electrons. Every single electron would need to simulate all the rooms to calculate the shortest path, which it then takes. The thought experiment fails to consider this.

maybe youre right

great. but this example assumes all the man has is a set of rules for how to shuffle the symbols if what he instead had was a neural net, and he (and others) kept getting tested and tested, it is completely plausible that the best performing symbol rotator was the one that (from the symbol shuffling exercise) actually learned what the different symbols meant and effectively understood written chinese actually understanding chinese might very well be the most efficient implementation for winning the challenge and we know that neural architectures have the capability to understand chinese, since we at our root are neural nets ourselves and we claim to have the ability to ”understand languages” that rotator that understood chinese would outperform all the other contestants. so, the question then becomes, how can we distinguish between a very advanced and sophisticated symbol shuffler from a man in the room who has truly decoded the symbols? well, as your challenges get increasingly more complex, you can use your common sense about what is most computationally tractable, or just find comfort in the mystery i believe that if you dare to be brave your gut feeling has the rough answer to what is actually going on inside these models. you already know the answer. it’s not binary.

The Face of wisdom and Knowledge in the 1980s

How do we know that our brains aren’t just sophisticated Chinese rooms then?

Isn’t this how people learn a new language? Surely, eventually the person in the room learns the language?

Quite a poor argument indeed. In his analogy, the human in the room is the GPU, and the rule book is the LLM. No one claims the GPU to be intelligent or self-conscious.

Except, thats exactly what humans do.

This is like not understanding why a person is crying but still understanding that they could use a hug. It just *moves* where the consciousness occurs in the scenario, but it does nothing to *eliminate* the consciousness. (This Searle argument's hugely overated, I think.)

Except, the Chinese symbols can also be seen as letters, nor words. And, we ourselves don’t understand the Latin letters we use. They are just some squiggles that we arrange as we think is right

This doesn't demonstrate the process doesn't have consciousness or understanding.

If this interview had been done today, the host would have interrupted 87 times, completely disrupting the flow of the explanation, the other guest would be rolling eyes and haurumpfing the entire time, and the speaker would never have been able to make his point.

AI “hallucination” explained perfectly

thanks

This is actually a great argument for rules-based systems, but for LLMs, which make up their own rules at scale (1T+ parameters, unknown numbers of hidden layers - where "meaning" lies) - it doesn't quite apply. Because the Chinese room implies following consistent rules- or logic-based systems, it is that the rules themselves cannot contain the "understanding". The issue with LLMs is that their rules are not logic-based: A implies B. Their rules are correlation and probability-based: A is related to B. And so while you still get some degree of Chines Room-ism, the issue with modern LLMs is that, yes, they predict the next character, but they predict it based not on rules, but on clusters of related concepts. In this way it is possible to have some sense of "understanding", insofar as "understanding" is not just knowing rules, but knowing what things are related to, and being able to make up one's own rules based on those relations. This is why LLMs hallucinate - because they don't have hard and fast "rules" - they are looking for the relations between concepts, and sometimes, they make one up. So, while it's not the case that modern LLMs are "rule-followers" like the neural networks of the '80s and Searle's Chinese Room example, It is also not the case (perhaps, just yet), that they have human-like "understanding". It all depends on what your understanding of "understanding" is: If it's enough to know that fire is related to wood and heat and oxygen, and somewhat to water, then yes, LLMs might have actual "understanding". But if you want "understanding" to be something like complete and accurate inference, coming up with accurate predictions of whether that building will burn down or whether there is enough oxygen to put out a fire, Then maybe not - quite yet. The promise of LLMs is that they gain with complexity - the more knowledge and parameters and questions they have, the more "understanding" and correct inference they may be able to yield. So that's simply the thing - because LLMs don't use the same rules-based paradigm as the Chinese Room, Searle's argument doesn't necessarily apply. But because they also don't yet quite have the complexity and information and scale required to make correct inference at all times, we can't quite say that they have "understanding" yet either. But the thing is that it's not that their paradigm is "wrong" or won't get there - the idea behind large language models - that the "understanding" may actually be based in the neural connections, not any "rules" themselves - may actually be the correct one. They list might not have enough scale yet to create true "understanding". And that's a scary thought. Unlimited artificial intelligence may create worlds, or it may destroy them, or lead to their tearing down. It's really up to us. But in order to decide what to do with them, we must first understand what they are, And then we can accurately discuss what to do with them. I hope this helped.

fascinating take, but matrix multiplication based on attention weights is still just formal symbol manipulation. high-dimensional spatial clustering mimics understanding, but correlation isn't comprehension, no matter how many parameters you stack

Do you know that for sure? Babies learn language through time-correlated input, the association between word and image or word and experience, not through any sort of mathematics. Scientists have been unable to find a single place where "consciousness" or "experience" is generated - the current idea is that the experience is in the connections themselves. While it's true that LLMs might not have the "experience" of the color red, that is likely because they don't have the sensory inputs (skin, warmth, etc), to understand it in that sense. But what happens when they have "bodies" and real-time sensory inputs, rather than static weights? When they can generate their own correlations from novel stimula? Would they have "comprehension" then?

sensory inputs are still processed as raw binary data. to a machine, camera pixels and tactile sensor feeds are just numbers inside a matrix, not subjective feelings. feeding a model more data types (multimodal) increases data complexity, but processing numbers about heat still isn't the subjective sensation of feeling warm

Truth - but could it be a different type of experience? Do you need emotions and sensations to have experience - or are those only a certain kind of biological primitive?

I should say to have _understanding_ - experience could be highly debatable to a human standard (though I would still argue there is a certain kind of biological and possible nonbiological kind of experience)

The framing of this experiment is incorrect. The AI in question is not the man in the room, but the rulebook itself. The man doesn’t speak Chinese, but the rulebook, if sophisticated enough, arguably might.

It doesn’t matter if the person really understand Chinese because his Chinese is so good that he trumps 99% of population who speaks anc understands Chinese. (Avg human intelligence is so low that llm, being a parrot beats them at productivity and economic usefulness

Goofy. That doesn't prove that artificial consciousness can't exist. All it does is provide an example of how the Turing Test could be passed without consciousness. But how does this example differ from knowing whether other humans have consciousness?

Isn’t it amazing there was a time when people could have an important discussion with many opinions and simply listen and really try to understand others ideas

As a student of philosophy, I’ve always accepted this conceptual framework to conclude we can’t build consciousness — another key part is we don’t have a model of consciousness from which to build. So AI will never be conscious, correct? No, it’s not correct. Consciousness could descend down into AI from another dimension… very possible and the more we understand consciousness the more possible it seems.

Very poignant and relevant

You overlooked the most important possibility What if human thought is simply a Chinese room itself

The thought experiment always struck me as ill constructed. The man in the Chinese room does not understand Chinese. But the man and the Chinese room together *does*.

Searle actually addressed this system reply directly: imagine the man memorizes the entire rulebook and does all calculations in his head. now he is the whole system, yet he still understands zero chinese

I don't think that argument really holds, since you have to contrive this ability to memorize how to decode something but then label it outside of his conscious understanding—when in practice having the decoding op for something's meaning in your head means... you understand it