Prime Intellect engineer: "everyone's bragging about a million-token context. here's what they don't tell you. at 256k tokens GPT-5.5 scores 80% on retrieval. push it to a million and it drops to 36%. the model accepts the context, it just can't reason across it. people call it context rot."... in a 20-minute talk he explains why bigger context windows won't save your agents. continual learning + training on your own traces + real environments - that's the fix. Watch the talk, then save!show more

what are you talking about? Chroma published Context Rot in the summer of 2015 and everyone knows this

))

🫶🏻

Jackmin the goat

yeah, no. It's hard to use a model that has less than 400k because they go and look at my mono-repo and explode into oblivion

I thought we’ve known this for a while now?

yooo it's @jackminong

context drift is a challenge

don't tell YC or Garry Tan about this

the real bottleneck isn't context size it's context understanding 1M tokens sounds impressive but memory and retrieval are what actually make agents reliable bigger prompts won't fix bad reasoning

Context rot is the bane of my existence and the biggest problem AI labs need to solve. Having more context is SO important, but it’s useless if the model loses 60 IQ whenever it’s near the cap.

@unxed

Bigger context isn't everything how well a model uses it matters far more than the headline number.

I have yet to use a model past 300k tokens that ddnt seem like a drooling zombie

@willcb This was determined a year ago. It’s the better half of the Ralph loop premise all of the tokenmaxxers left out

So I suppose it's a coincidence that's exactly what his company does? I'm all for more customized models, but let's not pretend every engineer needs one for decent performance at long context.

@_yaksan @Alex_Car12 👂

If you understand basic statistics and what an LLM is.. it makes perfect sense

Purpose built niche small language models

Niceeee

i have an "Andrej Karpathy" skill and it always tell me to keep the context small :-)

This is a fact that people don't like to hear. 384k is absolute max attention span.

This is true. The thinking block of a reasoning model consumes a lot of context space itself. Sure the thinking block is generally not included in conversation history, but at least while it is reasoning it does take up context space. Of course that also says nothing about high context utilization and accuracy...

Limits of continual learning:

context rot is real, but the fix isn't just continual learning. it's retrieval architecture. most agents don't need a million tokens. they need a memory layer that pulls the right 8k at the right time. bigger window is a workaround for not building the system correctly.

@jackminong

@willcb 🎯

That is why the context file needs to be kept as small as possible, so that it does not lose any information

was more or less common knowledge no?

A million tokens sounds impressive until the model has to connect two facts 800k apart.

256k에서 80%, 1M에서 36%라면 컨텍스트 크기보다 ‘필요한 근거를 다시 꺼내는 검색 단계’를 따로 설계해야겠군요.

has been a known fact for 2 years now. 1M context is a scam.

The only people that benefits from large context are the inference providers

for now, human handles experience 1 billion times better than llm handles context: we are safe

That is why I have stopped using sonnet and opus 1m context models in claude.. Better to stick with the lower context models and compact automatically or when needed.. Needs a bit of reminding every now and then but much better experience.

This is exactly why raw context window size is overhyped. Agents need memory systems + continual learning, not just bigger haystacks. Great share 🔥

@willcb @mixedbreadai is the only way