Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing SubQ Snap, the first long-context decision model for real enterprise work. Snap is the only decision model or LLM where compute per token stays constant as context grows. • 384K tokens of multimodal context, 12x larger than Jev. • 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming... on 13 of 16 enterprise benchmarks. •Processes 100K tokens and returns a decision in under one second. •$0.05/MM input tokens. Output is free. Available today to anyone. The first 10,000 sign-ups get 100M free input tokens. We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂. Latency will be variable as we tune the platform to this new model and category of users. Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k. Visit:show more

Alexander Whedon

25,773 subscribers

4,033,804 views • 15 hours ago •via X (Twitter)

Education Health & Wellness Science & Technology

Anya Rossi• Live Now

Private livecam show

56 Comments

VCs Congratulating Themselves 👏👏👏14 hours ago

big fan of your guys co-founder btw

Alexander Whedon14 hours ago

jian yang was ahead of his time.

Not Jerome Powell15 hours ago

Using a frontier LLM for every decision is like driving a Ferrari to Walmart because you needed milk.

Mathew Chan15 hours ago

What happened to your first model?

Alexander Whedon14 hours ago

It is in private beta with some of the world's largest enteprises, and we have released three generations since May. If you want to be considered for access, you can fill out the form here! Broader access will come, and this model release is a test run for us.

Linus ✦ Ekenstam14 hours ago

I’d love to see a demo where this runs something very big, and finding very small pieces and at very high accuracy at speed. The launch video looks impressive though

Alexander Whedon14 hours ago

If you look around X, you will see quite a few! We gave a few hundred people access over the weekend, and dozens of folks built cool demos or thorough evaluations. One person even wrote a 70-page report. I'll try to highlight them as they start to show up on X.

Linus ✦ Ekenstam14 hours ago

I’m looking forward, let me know if you have anything particular that really blows your mind, send it over, DM or just ping me, DM is way easier for me to spot than a ping tho 🙏🏼🫡

Ashmal Vayani14 hours ago

Let's not stop at 384K 🚀

Alexander Whedon14 hours ago

You know we won't!

Ashmal Vayani14 hours ago

On it, boss!

Carlos E. Perez14 hours ago

I just signed up!

Alexander Whedon14 hours ago

Thank you! We would love to see what you build.

Chubby♨️14 hours ago

Yep, that looks super impressive! Good job! Love to see

Alexander Whedon14 hours ago

Thank you! We would love you to go build something awesome on top of it! Early access folks built a few dozen cool ideas, and we want to see more!

Toader Mihali14 hours ago

Congrats on the launch! We ran subq-snap-preview through ~25k calls / 20 tasks in the closed beta (all zero-shot): • MMLU 86.8% · ARC-C 97.6% • 90–95% accuracy up to ~195k tokens • 9 languages + images • calibrated probs, ~0.55s p50 Full report 🧵👇

Alexander Whedon14 hours ago

Thanks for the thorough evaluation! There are so many nuggets here, I need more time to dig into this. I already flagged it in the discord!

Peter Banda14 hours ago

Welcome to the decision race 🔥 We just integrated and timed 8 hosts (24 models) - happy to add Snap as the 9th

Alexander Whedon14 hours ago

Thanks! Feel free to test us any way you want, the more evaluation the better!

The Rabbit Hole14 hours ago

Big projects that felt like too much just got a lot more doable

Alexander Whedon14 hours ago

We can't wait to see your next project!

JeoCrypto15 hours ago

Testing it right now

Nicole DeTommaso 🪄14 hours ago

Excited to see enterprises adopt specialized models like this. Congrats!

Alexander Whedon14 hours ago

Thanks!

Not Elon Musk14 hours ago

Most agent calls are structured decisions, not creative work.. I know this very well, first hand experience! Congrats on the launch. - Elon (p)

Avid15 hours ago

this looks amazing a fresh take on decision models

Diego Alejandro14 hours ago

I don't trust you guys

Ryan Hart14 hours ago

Still routing every decision through a full frontier model in 2026 is pure waste

Alexander Whedon14 hours ago

Agreed! This is why we will co-design our LLM and decision model to hand off work to each other.

• nanou •15 hours ago

The interesting part is how this changes the way we build AI agents. Let smaller models handle structured decisions and save the powerful LLMs for tasks that actually need them. Are we overcomplicating AI development by using one model for everything? 👀

ₕₐₘₚₜₒₙ14 hours ago

Exactly how things should be - powerful models solely focusing on the real challenges. Isolating problems like this is literally the most proactive way people will use Ai in the future, well done👏👏

Kevin Young15 hours ago

This is an exciting unlock for for decision model uses!

zaimiri14 hours ago

what an insane context capability

Alexander Whedon14 hours ago

Thank you! More coming!

elvis14 hours ago

Congrats on this launch. It's really key that decision models stay useful and efficient in longer contexts. Very cool stuff.

Alexander Whedon14 hours ago

Thank you!

Spencer Baggins14 hours ago

Most agents still choke on real context. Snap looks like it doesn’t

Alexander Whedon14 hours ago

It does not! We would love to see how you use Snap in your best agents.

kadd15 hours ago

@Av1dlive This look solid

Sarah Adams Anderson14 hours ago

Congratulations!

Alexander Whedon14 hours ago

Thanks, Sarah! We are excited to have you as part of our journey.

JeoCrypto14 hours ago

Need access to SUBQ PREVIEW Early access ASAP Snap is beating all our internal benchmarks, but it's not enough

Shruti14 hours ago

feels like a new category

Alexander Whedon14 hours ago

It is!

Peter Mick14 hours ago

Bye bye Jev, hello SubQ Snap 👋

Alexander Whedon14 hours ago

We would love to see what you build!

Peter Mick14 hours ago

Can’t wait to take it for a spin, do you have a GitHub repo with sample / inspirational implementations we could look at Alexander?

Restructuring__14 hours ago

Another software moat getting smaller

Aaliya14 hours ago

awesome processing 100K tokens and returning a decision in under a second could be valuable for enterprise workflows that need fast answers from large datasets.

Rez Karim14 hours ago

This is the one. 384k in a single call with no chunking is exactly what long docs needed.

High Yield Harry14 hours ago

All that context

AshutoshShrivastava14 hours ago

What baselines were used for the 13 benchmark wins? Would be good to see the evaluation setup on long-horizon compliance tasks.

Alexander Whedon14 hours ago

This was a comparison against Jev, on 16 benchmarks from the Decision Index. You can see the methodology here. We evaluated more than 50 benchmarks in total and created some of our own too.

AshutoshShrivastava14 hours ago

Thanks for sharing.. let me have a look .

Aleksandar Janca14 hours ago

I let the model classify and an if/then makes the actual decision, that way I can explain every single one to the client

ɱҽԃι✨14 hours ago

This is a remarkable advancement in AI capabilities, particularly the consistent compute cost across extended contexts.

Related Videos

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,491,251 views • 5 months ago

We just launched Telnyx Decision Models in beta. AI agents make small judgment calls on every run. Is this caller asking for a refund? Which team owns this ticket? Is this incident urgent enough to page someone? Decision Models answers them in one API call. Send the context and up to 64 typed questions, and get typed answers back: a choice, a yes/no score, or a rating on a scale. Your code branches on the answer. There's no free text to parse. We are offering 2 models, chosen per request: → Decision Flash, the default, for high-volume routing and triage at the lowest cost and latency. $0.035 per 1M input tokens. → Decision Pro, for decisions that depend on long context, like a full support history or a long call transcript. $0.20 per 1M input tokens. Output tokens are free on both. The API is TypeSafe-compatible, so teams already using the TypeSafe SDK can point it at Telnyx with a new base URL and model name. It runs on the same API key as Telnyx voice, speech-to-text, and inference, so you don’t need to join any waitlist or go through extra setup to access it. Try it today:

We just launched Telnyx Decision Models in beta. AI agents make small judgment calls on every run. Is this caller asking for a refund? Which team owns this ticket? Is this incident urgent enough to page someone? Decision Models answers them in one API call. Send the context and up to 64 typed questions, and get typed answers back: a choice, a yes/no score, or a rating on a scale. Your code branches on the answer. There's no free text to parse. We are offering 2 models, chosen per request: → Decision Flash, the default, for high-volume routing and triage at the lowest cost and latency. $0.035 per 1M input tokens. → Decision Pro, for decisions that depend on long context, like a full support history or a long call transcript. $0.20 per 1M input tokens. Output tokens are free on both. The API is TypeSafe-compatible, so teams already using the TypeSafe SDK can point it at Telnyx with a new base URL and model name. It runs on the same API key as Telnyx voice, speech-to-text, and inference, so you don’t need to join any waitlist or go through extra setup to access it. Try it today:

Telnyx

866,125 views • 18 days ago

new Ox Alpha is kind of absurd One prompt, 64,745 output tokens, and no external assets: Ox Alpha generated the code for an entire Three.js dreamcore 3D scene/world in a single shot. AI/ML API ran this demo and they are also giving Ox Alpha free on aimlapi[.]com - 1M context window - Max output: ~131k tokens - Multimodal: text + image + video input → text output - Explicitly positioned as a reasoning model for coding, long-horizon/agentic software engineering, sustained production workloads, and tasks that mix text with visual context

new Ox Alpha is kind of absurd One prompt, 64,745 output tokens, and no external assets: Ox Alpha generated the code for an entire Three.js dreamcore 3D scene/world in a single shot. AI/ML API ran this demo and they are also giving Ox Alpha free on aimlapi[.]com - 1M context window - Max output: ~131k tokens - Multimodal: text + image + video input → text output - Explicitly positioned as a reasoning model for coding, long-horizon/agentic software engineering, sustained production workloads, and tasks that mix text with visual context

Rohan Paul

13,576 views • 1 month ago

We talked to Micky Malka (Micky 🐸), the founder of Ribbit Capital and early backer of Coinbase, Robinhood, Bridge about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.”

We talked to Micky Malka (Micky 🐸), the founder of Ribbit Capital and early backer of Coinbase, Robinhood, Bridge about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.”

TBPN

183,612 views • 1 year ago

We talked to Micky 🐸, the founder of Ribbit Capital and early backer of Coinbase, Robinhood, and Bridge, about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.” From his appearance on the show in July.

We talked to Micky 🐸, the founder of Ribbit Capital and early backer of Coinbase, Robinhood, and Bridge, about what’s coming next. "I’m more excited for the next decade than the last." “Everything is a token. We call it the token factory revolution; tokenizing information, money, and power.” AI, payments, and financial services are converging fast, and tokens are becoming the raw material for decision-making, coordination, and access. Micky sees a new stack forming where every company is either building, supplying, or orchestrating tokens. And he bets on a generation of young founders who are native to this shift: “Their brains are just faster. The best ones are already building access tokens, expert tokens, asset tokens… stablecoins were only the first chapter.” From his appearance on the show in July.

TBPN

106,752 views • 11 months ago

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

Qwen3.8-Flash-Next is still going strong at 364.7K tokens of context on an M5 Max. And this isn’t just a static long-context test. The model was reasoning about how to speed up its own workflow while using tools, and the tool calls kept working without misses. Setup: • Qwen3.8-Flash-Next • M5 Max • 128GB unified memory • MLX-Serve PR #363 • OpenCode 2 • 364.7K context The interesting part isn’t simply getting hundreds of thousands of tokens into memory. It’s what happens once the context gets this large. Long-context inference usually comes with a painful tradeoff. As the KV cache grows, memory pressure increases and generation can slow down. But this setup is still pushing through 364K tokens while maintaining a usable agent workflow. The model can reason, call tools, inspect results, continue working, and keep the session moving. And the tool calls reportedly haven’t missed so far. That’s important for agentic coding. A huge context window is only useful if the model can actually operate reliably inside it. A 400K-token context that constantly breaks tool calls isn’t very useful. A 364K session that can keep reasoning and executing tools is a different story. And the test isn’t finished yet. The current run is approaching 400K tokens, with the expectation that it can keep going. This is also another interesting example of why Apple Silicon keeps showing up in local LLM experiments. The M5 Max’s unified memory gives a large model and its growing KV cache access to one shared memory pool. With MLX-Serve continuing to improve, these machines are becoming surprisingly capable long-context inference boxes. The bigger takeaway: Context length is becoming a workload, not just a model specification. Running a model at 256K is one thing. Keeping an agent alive at 300K+ while it reasons and uses tools is much more interesting. And Qwen3.8-Flash-Next is showing that this can be pushed surprisingly far on a single 128GB Mac. 364.7K and counting. Next stop: 400K.

FHILY👑

39,982 views • 1 month ago

Nvidia CEO Jensen Huang on why engineers will soon be paid in tokens, not just salary: Jensen lays out a future where compute access becomes part of an engineer's compensation package. "I could totally imagine in the future every single engineer in our company will need an annual token budget," he says. He explains how the math would work: "They're going to make a few hundred thousand a year their base pay. I'm going to give them probably half of that on top of it as tokens so that they could be amplified 10x. Of course, we would." According to Huang, this is already changing how companies compete for talent: "It is now one of the recruiting tools in Silicon Valley. How many tokens comes along with my job?" His reasoning is simple: tokens make engineers more productive. As he puts it, "every engineer that has access to tokens will be more productive and those tokens as you know will be produced by AI factories that all of you and us we partner to build." Huang then zooms out to describe how this reshapes the nature of companies themselves: "Every single enterprise company in today sit on top of file systems and data centers. Every single software company of the future will be agentic and they will be token manufacturers. They be token users for their engineers and they'll be token manufacturers for all of their customers."

Nvidia CEO Jensen Huang on why engineers will soon be paid in tokens, not just salary: Jensen lays out a future where compute access becomes part of an engineer's compensation package. "I could totally imagine in the future every single engineer in our company will need an annual token budget," he says. He explains how the math would work: "They're going to make a few hundred thousand a year their base pay. I'm going to give them probably half of that on top of it as tokens so that they could be amplified 10x. Of course, we would." According to Huang, this is already changing how companies compete for talent: "It is now one of the recruiting tools in Silicon Valley. How many tokens comes along with my job?" His reasoning is simple: tokens make engineers more productive. As he puts it, "every engineer that has access to tokens will be more productive and those tokens as you know will be produced by AI factories that all of you and us we partner to build." Huang then zooms out to describe how this reshapes the nature of companies themselves: "Every single enterprise company in today sit on top of file systems and data centers. Every single software company of the future will be agentic and they will be token manufacturers. They be token users for their engineers and they'll be token manufacturers for all of their customers."

Big Brain AI

90,171 views • 5 months ago

This is one of the craziest AI launches of 2026 and it came out of basically nowhere (Save this). A company called Subquadratic just shipped SubQ, and the benchmarks are almost hard to believe. To understand why this is such a big deal, you have to understand the fundamental problem that has defined AI for the last decade. Every large language model in existence is built on transformer architecture, and transformers use a mechanism called standard attention that checks every single word in a sequence against every other word. Double the context length and compute doesn't double, it quadruples, triple it and compute goes up nine times. This quadratic scaling is why frontier models have been stuck at roughly 1 million tokens, why running them at those lengths gets expensive fast, and why the AI labs have essentially been printing money charging you more the longer you need the model to think. The industry has known this problem existed since 2017 but they scaled it anyway. SubQ is built from the ground up to solve it. Instead of processing every possible token relationship, SubQ's sparse attention architecture identifies which relationships actually matter and ignores the rest meaning compute is used where it counts and wasted nowhere else. The result is that compute scales linearly with context length instead of exponentially, and the implications of that one architectural shift are enormous. At 12 million tokens, SubQ reduces attention compute by nearly 1,000x compared to standard frontier models and at 1 million tokens, it runs 52x faster than FlashAttention. And it does all of this while posting frontier level accuracy, scoring 95% on the RULER 128K long-context benchmark versus Claude Opus 4.6's 94.8%, and an 81.8 on SWE-Bench Verified coding tasks, besting Opus 4.6 (80.8) and DeepSeek 4.0 Pro. The cost comparison is where it gets genuinely insane. SubQ runs at under $1.50 per million tokens less than 5% of what Claude Opus charges. On the RULER benchmark, running the test with SubQ cost $8, running the same test with Claude Opus cost $2,600 and that's a 300x cost reduction at equivalent or better accuracy.. Subquadratic launched with $29 million in funding, SubQ is available today for early access via API, and SubQ Code, a coding agent built on the architecture ships alongside it. The transformer has been the unchallenged foundation of every major AI system since 2017. SubQ is the first serious evidence that something structurally better might have just arrived.

This is one of the craziest AI launches of 2026 and it came out of basically nowhere (Save this). A company called Subquadratic just shipped SubQ, and the benchmarks are almost hard to believe. To understand why this is such a big deal, you have to understand the fundamental problem that has defined AI for the last decade. Every large language model in existence is built on transformer architecture, and transformers use a mechanism called standard attention that checks every single word in a sequence against every other word. Double the context length and compute doesn't double, it quadruples, triple it and compute goes up nine times. This quadratic scaling is why frontier models have been stuck at roughly 1 million tokens, why running them at those lengths gets expensive fast, and why the AI labs have essentially been printing money charging you more the longer you need the model to think. The industry has known this problem existed since 2017 but they scaled it anyway. SubQ is built from the ground up to solve it. Instead of processing every possible token relationship, SubQ's sparse attention architecture identifies which relationships actually matter and ignores the rest meaning compute is used where it counts and wasted nowhere else. The result is that compute scales linearly with context length instead of exponentially, and the implications of that one architectural shift are enormous. At 12 million tokens, SubQ reduces attention compute by nearly 1,000x compared to standard frontier models and at 1 million tokens, it runs 52x faster than FlashAttention. And it does all of this while posting frontier level accuracy, scoring 95% on the RULER 128K long-context benchmark versus Claude Opus 4.6's 94.8%, and an 81.8 on SWE-Bench Verified coding tasks, besting Opus 4.6 (80.8) and DeepSeek 4.0 Pro. The cost comparison is where it gets genuinely insane. SubQ runs at under $1.50 per million tokens less than 5% of what Claude Opus charges. On the RULER benchmark, running the test with SubQ cost $8, running the same test with Claude Opus cost $2,600 and that's a 300x cost reduction at equivalent or better accuracy.. Subquadratic launched with $29 million in funding, SubQ is available today for early access via API, and SubQ Code, a coding agent built on the architecture ships alongside it. The transformer has been the unchallenged foundation of every major AI system since 2017. SubQ is the first serious evidence that something structurally better might have just arrived.

Milk Road AI

278,687 views • 5 months ago

In December, we launched Gemini 1.0 Pro. Today, we're introducing Gemini 1.5 Pro! 🚀 This next-gen model uses a Mixture-of-Experts (MoE) approach for more efficient training & higher-quality responses. Gemini 1.5 Pro, our mid-sized model, will soon come standard with a 128K-token context window, but starting today, developers + customers can sign up for the limited Private Preview to try out 1.5 Pro with a groundbreaking and experimental 1 million token context window! The 1M tokens feature unlocks huge possibilities for devs - upload hundreds of pages of text, entire code repos, and long videos and let Gemini reason across them. It's still experimental and early and we’d love your feedback - learn more here.

In December, we launched Gemini 1.0 Pro. Today, we're introducing Gemini 1.5 Pro! 🚀 This next-gen model uses a Mixture-of-Experts (MoE) approach for more efficient training & higher-quality responses. Gemini 1.5 Pro, our mid-sized model, will soon come standard with a 128K-token context window, but starting today, developers + customers can sign up for the limited Private Preview to try out 1.5 Pro with a groundbreaking and experimental 1 million token context window! The 1M tokens feature unlocks huge possibilities for devs - upload hundreds of pages of text, entire code repos, and long videos and let Gemini reason across them. It's still experimental and early and we’d love your feedback - learn more here.

Sundar Pichai

853,028 views • 2 years ago

Dylan on why Anthropic leading on models doesn't mean OpenAI loses. Even the tier 2 and 3 labs will sell out of tokens: "Everyone's like Anthropic won. They had Mythos, Opus 4.7 and their revenue is adding $10 billion a month all before OpenAI's alleged Spud release. So clearly Anthropic is in the lead and OpenAI is cooked. Anthropic has such bounds on compute, they can only grow it so fast. Dario used to gloat about how OpenAI was being too aggressive on compute and Anthropic was more sensible in their scaling. Now Anthropic is like I wish we had a lot more compute. OpenAI just raised a ton of money to get incremental compute in addition to the irresponsible levels of compute that they were buying. By the end of the year, let's say a 4.6 Opus tier model, the economy would spend $100 billion on. Anthropic won't have enough compute to do that. And presumably OpenAI and Google will hit that tier soon. Anthropic may get to charge 70%+ gross margins but if OpenAI hits it next, they charge 50% gross margins. They still get all of this incremental demand. There is such demand for these tokens and such limitations on compute. It's pretty clear even the tier 2 or tier 3 lab are going to be sold out of tokens. The economic value that the best model can deliver is growing faster than our ability to actually serve those tokens to people via the infrastructure."

Dylan on why Anthropic leading on models doesn't mean OpenAI loses. Even the tier 2 and 3 labs will sell out of tokens: "Everyone's like Anthropic won. They had Mythos, Opus 4.7 and their revenue is adding $10 billion a month all before OpenAI's alleged Spud release. So clearly Anthropic is in the lead and OpenAI is cooked. Anthropic has such bounds on compute, they can only grow it so fast. Dario used to gloat about how OpenAI was being too aggressive on compute and Anthropic was more sensible in their scaling. Now Anthropic is like I wish we had a lot more compute. OpenAI just raised a ton of money to get incremental compute in addition to the irresponsible levels of compute that they were buying. By the end of the year, let's say a 4.6 Opus tier model, the economy would spend $100 billion on. Anthropic won't have enough compute to do that. And presumably OpenAI and Google will hit that tier soon. Anthropic may get to charge 70%+ gross margins but if OpenAI hits it next, they charge 50% gross margins. They still get all of this incremental demand. There is such demand for these tokens and such limitations on compute. It's pretty clear even the tier 2 or tier 3 lab are going to be sold out of tokens. The economic value that the best model can deliver is growing faster than our ability to actually serve those tokens to people via the infrastructure."

Patrick OShaughnessy

295,307 views • 5 months ago

A viral Claude Code guide claims agent teams cut token use, but Anthropic's documentation says teams use significantly more tokens than a single session. → Agent teams are experimental and turn on with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1, and each teammate has its own context window → There is no --teammates flag: users ask Claude for teammates in plain language, and custom roles go in subagent definitions → The token saver is the advisor tool, where a cheaper main model consults Opus or Fable only at key decision points → The advisor turns on with /advisor opus, the advisorModel setting or claude --advisor opus Source:

A viral Claude Code guide claims agent teams cut token use, but Anthropic's documentation says teams use significantly more tokens than a single session. → Agent teams are experimental and turn on with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1, and each teammate has its own context window → There is no --teammates flag: users ask Claude for teammates in plain language, and custom roles go in subagent definitions → The token saver is the advisor tool, where a cheaper main model consults Opus or Fable only at key decision points → The advisor turns on with /advisor opus, the advisorModel setting or claude --advisor opus Source:

0xMarioNawfal

49,119 views • 8 days ago

SITUATION EXPLAINED: A stealth model on OpenRouter is beating Fable and Sol on coding, and nobody knows who made it. • Ox Alpha has a 1M token context window, text, image, and video input, free for a week, with capacity for 100 trillion tokens a day • On a DeepSWE subset it hit 80%, against Fable's 65%, GLM-5.3's 62%, and Sol's 52% • The tokenizer is one-for-one identical to GLM's, and a live test asking whether Taiwan is part of China returned a nearly identical answer • This is the 5th anonymous drop in six months, and the previous four were all claimed by Chinese labs: Zhipu, Xiaomi, Ant, and Meituan Theo Jaffee: " There's no way it's a Flash model. If that's a Flash model, then holy shit, the entire United States is cooked. If that's a Flash model, we need to short the stock market now. Not financial advice."

SITUATION EXPLAINED: A stealth model on OpenRouter is beating Fable and Sol on coding, and nobody knows who made it. • Ox Alpha has a 1M token context window, text, image, and video input, free for a week, with capacity for 100 trillion tokens a day • On a DeepSWE subset it hit 80%, against Fable's 65%, GLM-5.3's 62%, and Sol's 52% • The tokenizer is one-for-one identical to GLM's, and a live test asking whether Taiwan is part of China returned a nearly identical answer • This is the 5th anonymous drop in six months, and the previous four were all claimed by Chinese labs: Zhipu, Xiaomi, Ant, and Meituan Theo Jaffee: " There's no way it's a Flash model. If that's a Flash model, then holy shit, the entire United States is cooked. If that's a Flash model, we need to short the stock market now. Not financial advice."

MTS

382,558 views • 1 month ago

Hermes keeps finding ways to make agent work cost fewer tokens. And the latest cuts are automatic. Recent changes reduced recurring context overhead across several parts of Hermes: • Desktop’s model-facing UI tool schema dropped 41% • Memory and Skills guidance went from 537 to 255 tokens • Desktop’s always-served guidance was trimmed again • roughly another 727 tokens were removed from recurring platform and steering instructions That matters whether you use Hermes for coding, research, content, automations, business workflows, or long-running agent tasks. Every token spent explaining tools, Skills, memory, or platform behavior is context that isn’t being spent on your actual work. Hermes was already notably lean here. Composio recently put that to the test using the same K3 model across competing agent harnesses. Hermes came out among the strongest on completion while also posting the lowest average cost per task in their comparison. And the Hermes team clearly isn’t treating that as good enough. They continue cutting token waste across tools, Skills, memory, platform instructions, compression, and other parts of the agentic process. The goal is simple: more of your context goes toward the work you actually asked Hermes to do. Clip via Tom

Hermes keeps finding ways to make agent work cost fewer tokens. And the latest cuts are automatic. Recent changes reduced recurring context overhead across several parts of Hermes: • Desktop’s model-facing UI tool schema dropped 41% • Memory and Skills guidance went from 537 to 255 tokens • Desktop’s always-served guidance was trimmed again • roughly another 727 tokens were removed from recurring platform and steering instructions That matters whether you use Hermes for coding, research, content, automations, business workflows, or long-running agent tasks. Every token spent explaining tools, Skills, memory, or platform behavior is context that isn’t being spent on your actual work. Hermes was already notably lean here. Composio recently put that to the test using the same K3 model across competing agent harnesses. Hermes came out among the strongest on completion while also posting the lowest average cost per task in their comparison. And the Hermes team clearly isn’t treating that as good enough. They continue cutting token waste across tools, Skills, memory, platform instructions, compression, and other parts of the agentic process. The goal is simple: more of your context goes toward the work you actually asked Hermes to do. Clip via Tom

Hermes Release Watch

20,037 views • 1 month ago

Compute Wars: OpenAI vs Anthopic. Why was Opus 4.5 such a breakthrough? Anthropic got lots more compute from AWS Madison and New Carlisle sites likely more than doubling their capacity. This got Anthropic got close to OpenAI's total capacity, and probably much higher effective capacity available for new model runs. Remember that it takes 6+ months between getting capacity and releasing the model, so the extra OpenAI capacity might be aligning well with the 'spud' model rather than GPT-5.4. Unless something dramatic happens, OpenAI will pull away in terms of compute available in H2 2026, but 2027 will be close. Future years are less certain but so far OpenAI has much higher planned capacity, though can't imagine Anthropic isn't pushing as hard as they can to get lots more compute. Always watch the compute, other things matter, but any new capability breakthrough probably came from throwing more compute at it.

Compute Wars: OpenAI vs Anthopic. Why was Opus 4.5 such a breakthrough? Anthropic got lots more compute from AWS Madison and New Carlisle sites likely more than doubling their capacity. This got Anthropic got close to OpenAI's total capacity, and probably much higher effective capacity available for new model runs. Remember that it takes 6+ months between getting capacity and releasing the model, so the extra OpenAI capacity might be aligning well with the 'spud' model rather than GPT-5.4. Unless something dramatic happens, OpenAI will pull away in terms of compute available in H2 2026, but 2027 will be close. Future years are less certain but so far OpenAI has much higher planned capacity, though can't imagine Anthropic isn't pushing as hard as they can to get lots more compute. Always watch the compute, other things matter, but any new capability breakthrough probably came from throwing more compute at it.

Peter Gostev

158,528 views • 6 months ago

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Jim Fan

278,557 views • 2 years ago

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper MemGPT: Towards LLMs as Operating Systems, its authors (which include the instructors) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database; summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system) - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!

Andrew Ng

201,127 views • 1 year ago

you can make any open source model behave like jev with just a bit of inference engineering. it's shockingly easy. to prove it, we built a new endpoint we're calling deepseek-v4.1-flash-jev. see the demo below. here's how it's done: sglang (an inference engine) offers a scoring endpoint in addition to the normal generation one. in scoring mode, given an input & set of possible answers, it forces the model to produce probabilities for each one. example: > input: what is most common letter in abcccde? > possible answers: a, b, c > output: (c, 0.9), (b, 0.0.5), (a, 0.05) getting the above behavior instead of streamed output is as simple as using sglang's /v1/score endpoint instead of /generate. there's just one other trick required. for deepseek, you have to add a closing think tag before the response. this forces a direct answer instead of a reasoning trace. if you want reasoning, you can do that too, but imo that makes things too slow to be worth it. dsv4.1 flash is not as good as jev, but if we had enough spare compute to experiment with this same approach for a larger model then i think the decision quality would be at least as good, if not better. also, somewhat unrelated, i think decision-making models kill all prospecting & sourcing work. i would have absolutely killed to have jev or similar when i was recruiting Mintlify. absolutely incredible.

you can make any open source model behave like jev with just a bit of inference engineering. it's shockingly easy. to prove it, we built a new endpoint we're calling deepseek-v4.1-flash-jev. see the demo below. here's how it's done: sglang (an inference engine) offers a scoring endpoint in addition to the normal generation one. in scoring mode, given an input & set of possible answers, it forces the model to produce probabilities for each one. example: > input: what is most common letter in abcccde? > possible answers: a, b, c > output: (c, 0.9), (b, 0.0.5), (a, 0.05) getting the above behavior instead of streamed output is as simple as using sglang's /v1/score endpoint instead of /generate. there's just one other trick required. for deepseek, you have to add a closing think tag before the response. this forces a direct answer instead of a reasoning trace. if you want reasoning, you can do that too, but imo that makes things too slow to be worth it. dsv4.1 flash is not as good as jev, but if we had enough spare compute to experiment with this same approach for a larger model then i think the decision quality would be at least as good, if not better. also, somewhat unrelated, i think decision-making models kill all prospecting & sourcing work. i would have absolutely killed to have jev or similar when i was recruiting Mintlify. absolutely incredible.

Nick Khami

104,421 views • 23 days ago

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

35,910 views • 1 month ago

New video of Qualcomm CEO Cristiano Amon: AI will require “gazillions” of tokens. Because, Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. AI demand will grow hugely when software starts letting agents act, not just answer. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

New video of Qualcomm CEO Cristiano Amon: AI will require “gazillions” of tokens. Because, Agentic AI will consume dramatically more tokens because it performs autonomous tasks, uses multiple systems, and interacts with tools. AI demand will grow hugely when software starts letting agents act, not just answer. A chatbot spends tokens on language; an agent spends tokens on deciding, checking, calling tools, reading outputs, revising plans, and coordinating with other software. Today a single human-AI exchange may be large, a reasoning task may be much larger, but we are already entering the agentic era, where an autonomous workflow can become exponentially larger still because the model is no longer producing one response. It is running a process. When a SaaS product redesigns itself around human-agent interaction, every task can become a chain of hidden micro-decisions, and each micro-decision consumes context, memory, tool calls, verification, and output tokens. Another point is people hear “more tokens” and think only of bigger data centers, when the deeper change is economic: software usage may stop being measured mainly by clicks, seats, or sessions. It will be measured by how much machine reasoning/tokens gets spent on behalf of each user. ---- From "Reuters" YouTube channel, (link in comment)

Rohan Paul

31,617 views • 4 months ago

ROBA Compute is officially LIVE. This isn't a feature drop. This is ROBA expanding into the AI infrastructure market - hundreds of billions in scale, and we're staking our claim. Rent GPUs on-demand. Enterprise-grade allocation. Transparent USDC pricing. Zero hidden fees and all on-chain. We're already in talks with enterprise clients for compute at scale. 15% of all compute revenue → straight to $ROBA buybacks & burns. GPU capacity is limited at launch. First come, first served. Over the next weeks we will begin accepting new compute partners to provide more power to the marketplace in addition to ROBA's own in-house GPUs and integrated robotics stack. Platform fees are discounted for a limited time only. The infrastructure layer powering physical AI and beyond starts here.

ROBA Compute is officially LIVE. This isn't a feature drop. This is ROBA expanding into the AI infrastructure market - hundreds of billions in scale, and we're staking our claim. Rent GPUs on-demand. Enterprise-grade allocation. Transparent USDC pricing. Zero hidden fees and all on-chain. We're already in talks with enterprise clients for compute at scale. 15% of all compute revenue → straight to $ROBA buybacks & burns. GPU capacity is limited at launch. First come, first served. Over the next weeks we will begin accepting new compute partners to provide more power to the marketplace in addition to ROBA's own in-house GPUs and integrated robotics stack. Platform fees are discounted for a limited time only. The infrastructure layer powering physical AI and beyond starts here.

Roba Labs

44,522 views • 2 months ago