Loading video...

Video Failed to Load

Go Home

Cursor posted an open problem - Character Prefix Conditioning on their blog post. This is my attempt to solving the problem using DFA + trie + SmolLM-135M. LLMs generate tokens, not characters. But we type characters. If the user has typed a few characters in the current word(e.g. pri)...

archiexzzz's profile picture

Archie Sengupta

67,734 subscribers

24,973 views • 1 year ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Recently met Sasha Rush and he started giving me an impromptu lecture on how targeted on-policy self-distillation works. I asked him if I could record it on my iPhone. The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory. So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made. Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required. The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.
13:14

Recently met Sasha Rush and he started giving me an impromptu lecture on how targeted on-policy self-distillation works. I asked him if I could record it on my iPhone. The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory. So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made. Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required. The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

Dwarkesh Patel

430,505 views • 3 months ago

🚨 BREAKING: THERE ARE RUMORS YOU CAN NOW CREATE "SAFE TOKENS" DIRECTLY ON ETHERVISTADEX What are "Safe Tokens"? "Safe Tokens" are tokens generated through our SafeTokenFactory smart contract. These tokens are designed to eliminate vulnerabilities such as mintable functions or scammy taxes and come with a standardized implementation. Before swapping, users can easily verify whether a token is "safe" or if additional caution is needed. This marks a significant step forward in enhancing the quality of projects launched on Ethervista. But does this compromise the customizability of ERC tokens? Not at all. The Ethervista Protocol smart contract allows for a limitless range of applications. Take the $VISTA contract, for example. It's a standard ERC20 token, but with the Ethervista Protocol smart contract, it automatically buys and burns tokens. Similar logic can be applied to any ERC20 token using EthervistaDEX’s unique Protocol feature. What other features would you like to see? Wen dashboards? Wen streaming? We're on it—we just hired a full-time full-stack engineer! Special shoutout to Bonzi - FIRST MEME and MASCOT @ Ethervista and Clippy - Microsoft Anti AI Helper @ Ethervista, the first whitelisted tokens. We will continue to strongly support tokens that burn part of their liquidity before the 5-day lock period and those with strong communities and utility. A final note to creators: We would like to emphasize that burning lp-tokens does not alter your share of rewards UNTIL you remove, add, or claim rewards, which automatically updates your pool share ratio based on your current balance and the total lp-supply, as outlined in our whitepaper. This DOES NOT affect protocol fees, which are used to support both the protocol and creators.
0:50

🚨 BREAKING: THERE ARE RUMORS YOU CAN NOW CREATE "SAFE TOKENS" DIRECTLY ON ETHERVISTADEX What are "Safe Tokens"? "Safe Tokens" are tokens generated through our SafeTokenFactory smart contract. These tokens are designed to eliminate vulnerabilities such as mintable functions or scammy taxes and come with a standardized implementation. Before swapping, users can easily verify whether a token is "safe" or if additional caution is needed. This marks a significant step forward in enhancing the quality of projects launched on Ethervista. But does this compromise the customizability of ERC tokens? Not at all. The Ethervista Protocol smart contract allows for a limitless range of applications. Take the $VISTA contract, for example. It's a standard ERC20 token, but with the Ethervista Protocol smart contract, it automatically buys and burns tokens. Similar logic can be applied to any ERC20 token using EthervistaDEX’s unique Protocol feature. What other features would you like to see? Wen dashboards? Wen streaming? We're on it—we just hired a full-time full-stack engineer! Special shoutout to Bonzi - FIRST MEME and MASCOT @ Ethervista and Clippy - Microsoft Anti AI Helper @ Ethervista, the first whitelisted tokens. We will continue to strongly support tokens that burn part of their liquidity before the 5-day lock period and those with strong communities and utility. A final note to creators: We would like to emphasize that burning lp-tokens does not alter your share of rewards UNTIL you remove, add, or claim rewards, which automatically updates your pool share ratio based on your current balance and the total lp-supply, as outlined in our whitepaper. This DOES NOT affect protocol fees, which are used to support both the protocol and creators.

Ethervista

130,541 views • 2 years ago

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.
1:07

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.

Leandro von Werra

405,709 views • 4 months ago

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.
0:29

Redis built a cache that cuts LLM costs by 90%! Production LLM apps do not receive completely new questions every time. A customer-support assistant might receive all three of these: - "Can I get a refund after buying the monthly plan?" - "Is the monthly subscription refundable?" - "Can I cancel the plan and get my money back?" The wording is different, but the underlying question and its answer remain the same. Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated. Prefix caching reduces part of these repeated calls. When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix. But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded. So even with a prefix-cache hit, there's another generation call involved. To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it. When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again. A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call. In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs. Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud. I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together. For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload. You can try Redis LangCache here: If you want to dive deeper, I have already written a detailed breakdown of KV, prefix, prompt, and semantic caching in the article quoted below. This demo builds on the final technique and shows it running in practice. Read it below.

Avi Chawla

165,441 views • 11 days ago

NEW Cologne Major Shop Update: Everything you need to know ‼️ > No capsules, you have a shop where you can spend tokens > You can purchase whatever stickers you want with these tokens > Tokens can only be purchased when buying stickers and 100 tokens = 0.99$ > The prices of all stickers depend on relative demand > If a sticker price drops by more than 25 tokens within 24 hours of your purchase, you'll get a refund of tokens for the total difference > Each coin upgrade gives you 300 tokens, which you can use on Stickers or Souvenirs > You can create souvenir skins using your existing skins: Select a skin, then select a finished match and the gold stickers from that match will be applied to your skin > Souvenir Prices are determined by the demand for the specific gold team and autograph stickers involved, and the rarity of the weapon you choose for your souvenir > Teams, Players & the TO get 50% of the revenue from the Major Shop & Major Pass
0:58

NEW Cologne Major Shop Update: Everything you need to know ‼️ > No capsules, you have a shop where you can spend tokens > You can purchase whatever stickers you want with these tokens > Tokens can only be purchased when buying stickers and 100 tokens = 0.99$ > The prices of all stickers depend on relative demand > If a sticker price drops by more than 25 tokens within 24 hours of your purchase, you'll get a refund of tokens for the total difference > Each coin upgrade gives you 300 tokens, which you can use on Stickers or Souvenirs > You can create souvenir skins using your existing skins: Select a skin, then select a finished match and the gold stickers from that match will be applied to your skin > Souvenir Prices are determined by the demand for the specific gold team and autograph stickers involved, and the rarity of the weapon you choose for your souvenir > Teams, Players & the TO get 50% of the revenue from the Major Shop & Major Pass

Ozzny

358,016 views • 4 months ago

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.
0:41

A developer in Hangzhou runs an AI that remembers everything about him for $0.40 a year. No vector database. One file that never grows past 4,000 tokens. He published the whole schema. His version starts from the opposite idea. Memory is not storage. It's a write policy. Six fields. Rewritten every time, never appended: > IDENTITY - who you are, what you build. 300 tokens. Changes monthly at most > STATE - what you're on right now. 400 tokens. Rewritten daily > DECISIONS - what's already settled, so nothing gets re-argued. 800 tokens > CORRECTIONS - every time you said "no, not like that." 600 tokens > PEOPLE - names, roles, who's waiting on what. 500 tokens > DEAD - tried and abandoned, so it never comes back as a suggestion. 400 tokens Three thousand tokens. Ceiling of four. When a section fills, the model rewrites it shorter. Nothing is ever added. Only replaced. Kimi K2.5 bills $0.10 per million cached input tokens. Four thousand tokens a turn is $0.0004. That's 2,500 turns for a dollar. The free tier hands you 1.5 million tokens a day. 375 turns before you pay anything at all. CORRECTIONS is the field nobody builds, and it's the one that does the work. A model that remembers being wrong stops repeating it. Everyone else is paying to search their own history. He pays to keep it short. The bill stopped growing when the file did. Your memory system isn't defined by what it stores. It's defined by what it agrees to delete. The article below is the full build - schema, rewrite prompts, the compaction rule that keeps it under the cap. Save it. You'll want it open in the other tab.

wast3

15,862 views • 29 days ago

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."
2:22

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."

Patrick OShaughnessy

36,100 views • 4 months ago

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang
2:41

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

100,519 views • 2 months ago

Kimi K3 is worth every one of its 2.8T parameters. It's the best open-weight model I've tried. It's frontier-level quality, and it can give any model a run for its money. I'm running it on Nebius Token Factory. It comes with an OpenAI-compatible API endpoint and 120 tokens/s (although I've seen much faster responses). If you want to start using it, here is the link:
5:42

Kimi K3 is worth every one of its 2.8T parameters. It's the best open-weight model I've tried. It's frontier-level quality, and it can give any model a run for its money. I'm running it on Nebius Token Factory. It comes with an OpenAI-compatible API endpoint and 120 tokens/s (although I've seen much faster responses). If you want to start using it, here is the link:

Santiago

22,594 views • 1 month ago

Confidential tokens preserve public verifiability while keeping amounts and balances encrypted. But to make them the new default, we need to rally the entire stack of tools we already use today, wallets, explorers, indexers, and more. It's a big lift, and it's happening now. The work the team at Blockscout 🔭 did is a step in that direction: they're the first block explorer to natively support ERC-7984 confidential tokens, verifiable transfers with encrypted amounts. We recorded a demo with DOSH | 🦇🔊 from the Blockscout team, walking through how it works, and what explorers become when encrypted tokens are the default. This demo shows the work Zama is doing across the board: onboarding the tooling the ecosystem already relies on, so confidentiality onchain becomes the new normal, not the exception.
1:02

Confidential tokens preserve public verifiability while keeping amounts and balances encrypted. But to make them the new default, we need to rally the entire stack of tools we already use today, wallets, explorers, indexers, and more. It's a big lift, and it's happening now. The work the team at Blockscout 🔭 did is a step in that direction: they're the first block explorer to natively support ERC-7984 confidential tokens, verifiable transfers with encrypted amounts. We recorded a demo with DOSH | 🦇🔊 from the Blockscout team, walking through how it works, and what explorers become when encrypted tokens are the default. This demo shows the work Zama is doing across the board: onboarding the tooling the ecosystem already relies on, so confidentiality onchain becomes the new normal, not the exception.

Zama

18,609 views • 2 months ago

Palantir CEO Alex Karp says the reason they've become the global leader in AI is because they built full alignment with their customers instead of "masturbating and selling tokens" like other AI companies: "We actually want the customers to succeed." "We are the only company I know of, and definitely the only company at scale, that actually is paid as a derivative of value creation for the enterprises we work with and serve." "We do not have any other way of actually getting paid." "Every other company wants to outsource the risk of failure to the client because they know it's gonna fail." "Because to succeed, you would need a full stack from compute all the way up to model training, and you'd have to absorb the risk of that not working, which means having some kind of business intuition about what the value is to the customer, a willingness to say no, FDEs that aren't just masturbating and selling tokens." "They sell tokens. The FDEs, some of them are ex-Palantirians—they're just selling tokens. Just like some of you guys had a social life in high school that was virtual, except for you're ripping off the client."
1:39

Palantir CEO Alex Karp says the reason they've become the global leader in AI is because they built full alignment with their customers instead of "masturbating and selling tokens" like other AI companies: "We actually want the customers to succeed." "We are the only company I know of, and definitely the only company at scale, that actually is paid as a derivative of value creation for the enterprises we work with and serve." "We do not have any other way of actually getting paid." "Every other company wants to outsource the risk of failure to the client because they know it's gonna fail." "Because to succeed, you would need a full stack from compute all the way up to model training, and you'd have to absorb the risk of that not working, which means having some kind of business intuition about what the value is to the customer, a willingness to say no, FDEs that aren't just masturbating and selling tokens." "They sell tokens. The FDEs, some of them are ex-Palantirians—they're just selling tokens. Just like some of you guys had a social life in high school that was virtual, except for you're ripping off the client."

Jawwwn

18,670 views • 1 month ago

Liquid AI (Liquid AI) CEO Ramin Hasani (Ramin) says: "We're bringing the cost of tokens to zero." "The axis was maximizing intelligence at all costs." Foundation models need to optimize across 3 axes: 1.) Intelligence and capability 2.) Efficiency and cost 3.) Substrate: where the intelligence actually runs "Efficiency is not an afterthought. Energy is not abundant." "If you maximize intelligence at all costs, that axis alone is not going to get you to the place that you want to go." "Efficiency and cost of intelligence as a first-class citizen, and not an afterthought." "Where does this intelligence system go? AI is majorly getting hosted in data centers, but you could also bring intelligence on phones, on laptops, on airplanes, on cars." "Imagine if it's not tokens that are actually important. It's just the outcome that actually matters." "Not just thinking about foundation models as the token machines that are generating money and revenue for the foundation model companies that are useless tokens, and getting them into the place where they can actually unlock true value for enterprises."
6:17

Liquid AI (Liquid AI) CEO Ramin Hasani (Ramin) says: "We're bringing the cost of tokens to zero." "The axis was maximizing intelligence at all costs." Foundation models need to optimize across 3 axes: 1.) Intelligence and capability 2.) Efficiency and cost 3.) Substrate: where the intelligence actually runs "Efficiency is not an afterthought. Energy is not abundant." "If you maximize intelligence at all costs, that axis alone is not going to get you to the place that you want to go." "Efficiency and cost of intelligence as a first-class citizen, and not an afterthought." "Where does this intelligence system go? AI is majorly getting hosted in data centers, but you could also bring intelligence on phones, on laptops, on airplanes, on cars." "Imagine if it's not tokens that are actually important. It's just the outcome that actually matters." "Not just thinking about foundation models as the token machines that are generating money and revenue for the foundation model companies that are useless tokens, and getting them into the place where they can actually unlock true value for enterprises."

Molly O’Shea

12,816 views • 2 months ago

Perplexity CEO Aravind Srinivas on the brutal truth about who actually makes money in AI (and why it's not who you think): Aravind argues that the real value in AI comes from orchestration. He points to products like Codex, Claude Code, and Perplexity Computer: "What is that? It's an orchestration system. It takes a model, pairs it with an agent harness." And what is an agent harness? "The simplest way of describing it is like rules for how the agent loop should run. What are all the skills and sub-agents and connectors and tools it accesses? Without the harness, you don't necessarily capture and convert the intrinsic intelligence in the model into valuable output tokens." This leads to a blunt conclusion about who has a real business in AI, and who doesn't: "If you're literally just a reseller of model tokens, you have no business, because the model will get commoditized. So even if you're a model builder, you don't have a business. As an infra layer, you have some business on serving those output tokens. But as an application layer or model builder, you don't really have a business if you're just a reseller of tokens that come directly out of the model." So where does the value accrue? "You have a business if you know how to take the model, ground it in valuable context, orchestrate it with a really good agent harness, connected to the right set of tools and connectors (whether it's personal connectors or business connectors) and provide the experience to people in one single unified system." Aravind Srinivas then explains Perplexity's specific edge: Beyond orchestrating across tools, files, and connectors, they also orchestrate across models. "That is the differentiation that Anthropic and OpenAI cannot claim, because you wouldn't find GPT-5 inside the Claude Code harness. You wouldn't find Claude Opus inside the Codex harness. These are competing with each other. Whereas you would find both these models inside Perplexity Computer." Why does this matter? Because it all comes down to power. In Aravind's framing, the fundamental cost driver in AI is watts (the one input nobody can subsidize except the government). "Whoever provides the most valuable output tokens with the least amount of power expended to produce them generates the greatest value to the end user, has the most pricing power, has the most value. That is the orchestration problem to solve." His conclusion: "The one single most important metric in AI is token value per watt per user."
2:50

Perplexity CEO Aravind Srinivas on the brutal truth about who actually makes money in AI (and why it's not who you think): Aravind argues that the real value in AI comes from orchestration. He points to products like Codex, Claude Code, and Perplexity Computer: "What is that? It's an orchestration system. It takes a model, pairs it with an agent harness." And what is an agent harness? "The simplest way of describing it is like rules for how the agent loop should run. What are all the skills and sub-agents and connectors and tools it accesses? Without the harness, you don't necessarily capture and convert the intrinsic intelligence in the model into valuable output tokens." This leads to a blunt conclusion about who has a real business in AI, and who doesn't: "If you're literally just a reseller of model tokens, you have no business, because the model will get commoditized. So even if you're a model builder, you don't have a business. As an infra layer, you have some business on serving those output tokens. But as an application layer or model builder, you don't really have a business if you're just a reseller of tokens that come directly out of the model." So where does the value accrue? "You have a business if you know how to take the model, ground it in valuable context, orchestrate it with a really good agent harness, connected to the right set of tools and connectors (whether it's personal connectors or business connectors) and provide the experience to people in one single unified system." Aravind Srinivas then explains Perplexity's specific edge: Beyond orchestrating across tools, files, and connectors, they also orchestrate across models. "That is the differentiation that Anthropic and OpenAI cannot claim, because you wouldn't find GPT-5 inside the Claude Code harness. You wouldn't find Claude Opus inside the Codex harness. These are competing with each other. Whereas you would find both these models inside Perplexity Computer." Why does this matter? Because it all comes down to power. In Aravind's framing, the fundamental cost driver in AI is watts (the one input nobody can subsidize except the government). "Whoever provides the most valuable output tokens with the least amount of power expended to produce them generates the greatest value to the end user, has the most pricing power, has the most value. That is the orchestration problem to solve." His conclusion: "The one single most important metric in AI is token value per watt per user."

Big Brain AI

42,484 views • 1 month ago

Jensen Huang: “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed.” The Nvidia CEO expects his highly paid engineers to be spending at least HALF their salaries on tokens to supercharge their abilities. @jason: “ The conversation we've had on the pod a number of times is, ‘Oh my God, look at the token usage in our companies.’ It is growing massively.” “And some people are asking, ‘Hey, when I join a company, how many tokens do I get? Because I want to be an effective employee.’” “You've postulated, I believe, $75,000 in tokens for each engineer, something like that.” “So are you spending, at Nvidia, $1 billion, $2 billion on tokens for your engineering team right now?” Jensen: “We're trying to.” “Let me give you the thought experiment: Let's say you have a software engineer or AI researcher and you pay them $500,000 a year. We do that all the time.” “That $500,000 engineer, at the end of the year, I'm going to ask them, how much did you spend in tokens?” “If that person said, ‘$5,000,’ I will go ape… something else.” “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed. “And this is no different than one of our chip designers who says, ‘Guess what? I'm just going to use paper and pencil, I don't think I'm going to need any CAD tools.’” Jason: “This is a real paradigm shift, to start thinking about these all-star employees, it almost reminds me of what we learned in the NBA when LeBron James started spending a million dollars a year just on his health and his body, like in maintaining it. Here he is at age 41, still playing.” “These are incredible knowledge workers. Why wouldn't we give them superhuman abilities?”
1:18

Jensen Huang: “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed.” The Nvidia CEO expects his highly paid engineers to be spending at least HALF their salaries on tokens to supercharge their abilities. @jason: “ The conversation we've had on the pod a number of times is, ‘Oh my God, look at the token usage in our companies.’ It is growing massively.” “And some people are asking, ‘Hey, when I join a company, how many tokens do I get? Because I want to be an effective employee.’” “You've postulated, I believe, $75,000 in tokens for each engineer, something like that.” “So are you spending, at Nvidia, $1 billion, $2 billion on tokens for your engineering team right now?” Jensen: “We're trying to.” “Let me give you the thought experiment: Let's say you have a software engineer or AI researcher and you pay them $500,000 a year. We do that all the time.” “That $500,000 engineer, at the end of the year, I'm going to ask them, how much did you spend in tokens?” “If that person said, ‘$5,000,’ I will go ape… something else.” “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed. “And this is no different than one of our chip designers who says, ‘Guess what? I'm just going to use paper and pencil, I don't think I'm going to need any CAD tools.’” Jason: “This is a real paradigm shift, to start thinking about these all-star employees, it almost reminds me of what we learned in the NBA when LeBron James started spending a million dollars a year just on his health and his body, like in maintaining it. Here he is at age 41, still playing.” “These are incredible knowledge workers. Why wouldn't we give them superhuman abilities?”

The All-In Podcast

96,863 views • 6 months ago

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."
0:51

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

Fireside Alpha

140,571 views • 1 month ago

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

157,390 views • 4 months ago

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."
2:08

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,392 views • 4 months ago