🚨 Shocking: Frontier LLMs score 85-95% on standard coding benchmarks. We gave them equivalent problems in languages they couldn't have memorized. They collapsed to 0-11%. Presenting EsoLang-Bench. Accepted to the Logical Reasoning and ICBINB workshops at ICLR 2026 🧵

1/ Here's the intuition. When you learn Fibonacci in Python, you can write it in Java tomorrow without years of Java training. You transfer the logic. The loop, the state, the termination condition. Syntax is just a costume. LLMs claim to do this. We wanted to see if they actually can.

2/ Our method: test them on esoteric programming languages. Brainfuck. Befunge-98. Whitespace. Unlambda. Shakespeare. All Turing-complete. All requiring identical reasoning to Python. All with 1,000-100,000x fewer GitHub repos than mainstream languages. Same problems. Radically less training data.

3/ 80 problems across 4 difficulty tiers. Easy: sum two integers, reverse a string. Medium: Fibonacci, factorial. Hard: count primes, balanced parentheses. Extra-Hard: longest increasing subsequence, Josephus problem. Trivial in Python. A different story in esoteric languages.

4/ We tested GPT-5.2, O4-mini, Gemini 3 Pro, Qwen3-235B, and Kimi K2 across 5 prompting strategies. Models scoring 85-95% on HumanEval scored 0-11% on equivalent esoteric tasks. And every model, every language, every strategy scored 0% beyond the Easy tier. Not 2%. Not 5%. Zero.

5/ We threw everything at it to try to close the gap. Few-shot examples. Self-reflection. ReAct pipelines. Coder-critic pairs. Average improvement from few-shot: +0.8 percentage points. Statistically insignificant. ICL works by activating knowledge that already exists from pretraining. When that knowledge isn't there to begin with, a few examples in the context window can't substitute for it.

6/ The error profiles make the data coverage story concrete. Brainfuck and Befunge-98 have more online presence → models get syntax right but fail on logic. They understand the grammar, not the meaning. Unlambda and Shakespeare have almost none → 88-95% compile failures across every model. Models can't even produce valid syntax from scratch. Performance tracks data coverage remarkably cleanly.

7/ After the paper was finalized, we ran agentic systems that mimic how humans would learn to solve problems in esoteric languages. We supplied our agents with a custom harness + tools on the same benchmark. They absolutely crushed the benchmark. Stay tuned 👀

8/ This work was done by @inceptmyth under the supervision of @paraschopra at @lossfunk. If you work on evals or OOD generalization, we'd love to hear what you think. Pinging: @karpathy @fchollet @GaryMarcus @ylecun @AndrewYNg @demishassabis @drfeifei @goodfellow_ian @MelMitchell1 @percyliang @pmddomingos

@inceptmyth @paraschopra @karpathy @fchollet @GaryMarcus @ylecun @AndrewYNg @demishassabis @drfeifei @goodfellow_ian 9/ We're releasing everything: 🌐 Website: 📄 Paper: 🤗 Dataset: 💻 Code:

Do you have a human baseline? Solving problems in brainfuck is just empirically harder. I’m skeptical you’re actually testing memorization vs “esolangs made to be hard are hard”.

How do you expect anyone or anything to know something that wasn’t taught? This is nonsense.

Is o4-mini seriously the only reasoning model you tried? The word reasoning literally appears 44 times in your paper; seems like this should matter a *lot*.

Curious how it would perform if instructed to write a transpiler to these esoteric languages in a language that it's familiar with

After the paper was done, we tried modern agentic tools like claude code, gave them tools and instructed them to explore/learn We found it actually wrote something like this by itself (without instructing) Stay tuned for this update.

For this to be meaningful you need a human baseline with representatice median programmers unfamiliar with the languages. And I guarantee you they would not do well either as these languages suck by design. Theoretical Turing equivalence is irrelevant, LLMs aren't alien beings of pure logic. They work much more like humans, heavily leaning on pattern recognition and scaffolding. What would be much more interesting here is creating a de novo practical language that the median human programmers do well on and seeing how far models get with ICL (giving same reference docs for each).

@phl43 I bet they suck at writing haikus in Klingon too

I am afraid this amounts to sensationalist clickbait without proper background. Check for exotic code syntax comprehension in first place. Eg, after training on physics problems in english, not solving them in russian does not mean generalization deficits for physics.

should have included Excel

What reasoning level(s) did you try for GPT 5.2?

this is unreasonable. humans don't generalize to brainfuck either

Do you test human baseline

Disappointed ArnoldC didn’t make the cut in EsoLang-Bench: GET TO THE CHOPPER

“Unseen language = collapse” is a neat headline, not a law of nature. I’ve tested models on Aver and NanoLang too. When the language is small, regular, and backed by a strong corrective loop, they do far better than this framing suggests. The real variable is not just memorization. It’s feedback.

Recently trained a small transformer (~472M tokens) with BPE, RoPE, GQA, etc., and now I’m exploring applying similar ideas to Indian classical music. Specifically looking at representing ragas as sequence data (starting with MIDI, possibly moving to audio later). Curious about whether transformers can actually capture deeper raga structure , not just note sequences, but progression, mood, and inherent constraints. do you think this is something transformers can learn with scale, or would it require a different modeling approach / inductive bias? Would love your perspective.

Bruh. Someone could publish a paper where the test dataset consists of tribal languages which LLMs would not have been trained on, claim credit, and put a 🚨 emoji in a post.

Really liked this from @paraschopra. Same logic, new surface form, and LLMs collapse. Tells you a ton about where fluency ends and reasoning begins. It's similar to why we're differently approaching reasoning at @pathway_com. Recent result:

😂

oh no, the llm has to take a few seconds and write up a transpiler anyway

It's too bad this will only work once, they'll just generate a ton of training data for esolangs if this result becomes too prominent. You'd need to generate esolangs.

Now test college students (1) based on problems they have seen before vs (2) problems that are basically the same but worded in a slightly different manner from what they have ever encounter before.

let me propose a test. Rename all the keywords of a mainstream, py or anything. Let's say, if into what if, else into otherwise, for into for them all, or something similar. What's the accuracy drop?

Whats the human baseline? Lots of tests could be created that are nonsensical

Yeah, because LLMs can't generalize Let the scam go on tho, maybe some of us peasants will get an opportunity to buy into it at some point, too

We called it 'intelligence' when it memorized training data. The real test was always: can it learn what it hasn't seen? Most models just failed that test.

Were you expecting that models wd nail languages they have basically never seen? What human being wd be able to master a programming language they barely saw before?

Beyond retarded. Comparing Python to Java and then use Brainfuck. Which looks like this

benchmarks were never reasoning tests, they were memorization tests with nicer charts

@theo probably up your alley

New Hill Identified Time to climb @gdb @roon @apples_jimmy

no shit, same with humans. nobody cares about languages that practically don't exist 😂

0% accuracy speaking a language that it does not know sounds like you have achieved AGI. 0% is what a human would achieve for that test.

ok well that's 100% better than me. eso is not real lmao

Isn't this like asking an English speaking person questions in Chinese? 👀 Of course they wouldn't answer correctly even if they are super intelligent. Shocking sure, but LLM are pattern matching engines, not intelligence engines

this is the non bait tweet that shows that harness are the next frontier