Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Cursor posted an open problem - Character Prefix Conditioning on their blog post. This is my attempt to solving the problem using DFA + trie + SmolLM-135M. LLMs generate tokens, not characters. But we type characters. If the user has typed a few characters in the current word(e.g. pri)...

archiexzzz's profile picture

Archie Sengupta

67,734 subscribers

24,973 Aufrufe • vor 11 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Recently met Sasha Rush and he started giving me an impromptu lecture on how targeted on-policy self-distillation works. I asked him if I could record it on my iPhone. The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory. So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made. Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required. The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.
13:14

Recently met Sasha Rush and he started giving me an impromptu lecture on how targeted on-policy self-distillation works. I asked him if I could record it on my iPhone. The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory. So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made. Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required. The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

Dwarkesh Patel

428,351 Aufrufe • vor 2 Monaten

🚨 BREAKING: THERE ARE RUMORS YOU CAN NOW CREATE "SAFE TOKENS" DIRECTLY ON ETHERVISTADEX What are "Safe Tokens"? "Safe Tokens" are tokens generated through our SafeTokenFactory smart contract. These tokens are designed to eliminate vulnerabilities such as mintable functions or scammy taxes and come with a standardized implementation. Before swapping, users can easily verify whether a token is "safe" or if additional caution is needed. This marks a significant step forward in enhancing the quality of projects launched on Ethervista. But does this compromise the customizability of ERC tokens? Not at all. The Ethervista Protocol smart contract allows for a limitless range of applications. Take the $VISTA contract, for example. It's a standard ERC20 token, but with the Ethervista Protocol smart contract, it automatically buys and burns tokens. Similar logic can be applied to any ERC20 token using EthervistaDEX’s unique Protocol feature. What other features would you like to see? Wen dashboards? Wen streaming? We're on it—we just hired a full-time full-stack engineer! Special shoutout to Bonzi - FIRST MEME and MASCOT @ Ethervista and Clippy - Microsoft Anti AI Helper @ Ethervista, the first whitelisted tokens. We will continue to strongly support tokens that burn part of their liquidity before the 5-day lock period and those with strong communities and utility. A final note to creators: We would like to emphasize that burning lp-tokens does not alter your share of rewards UNTIL you remove, add, or claim rewards, which automatically updates your pool share ratio based on your current balance and the total lp-supply, as outlined in our whitepaper. This DOES NOT affect protocol fees, which are used to support both the protocol and creators.
0:50

🚨 BREAKING: THERE ARE RUMORS YOU CAN NOW CREATE "SAFE TOKENS" DIRECTLY ON ETHERVISTADEX What are "Safe Tokens"? "Safe Tokens" are tokens generated through our SafeTokenFactory smart contract. These tokens are designed to eliminate vulnerabilities such as mintable functions or scammy taxes and come with a standardized implementation. Before swapping, users can easily verify whether a token is "safe" or if additional caution is needed. This marks a significant step forward in enhancing the quality of projects launched on Ethervista. But does this compromise the customizability of ERC tokens? Not at all. The Ethervista Protocol smart contract allows for a limitless range of applications. Take the $VISTA contract, for example. It's a standard ERC20 token, but with the Ethervista Protocol smart contract, it automatically buys and burns tokens. Similar logic can be applied to any ERC20 token using EthervistaDEX’s unique Protocol feature. What other features would you like to see? Wen dashboards? Wen streaming? We're on it—we just hired a full-time full-stack engineer! Special shoutout to Bonzi - FIRST MEME and MASCOT @ Ethervista and Clippy - Microsoft Anti AI Helper @ Ethervista, the first whitelisted tokens. We will continue to strongly support tokens that burn part of their liquidity before the 5-day lock period and those with strong communities and utility. A final note to creators: We would like to emphasize that burning lp-tokens does not alter your share of rewards UNTIL you remove, add, or claim rewards, which automatically updates your pool share ratio based on your current balance and the total lp-supply, as outlined in our whitepaper. This DOES NOT affect protocol fees, which are used to support both the protocol and creators.

Ethervista

130,506 Aufrufe • vor 1 Jahr

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.
1:07

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.

Leandro von Werra

405,031 Aufrufe • vor 2 Monaten

NEW Cologne Major Shop Update: Everything you need to know ‼️ > No capsules, you have a shop where you can spend tokens > You can purchase whatever stickers you want with these tokens > Tokens can only be purchased when buying stickers and 100 tokens = 0.99$ > The prices of all stickers depend on relative demand > If a sticker price drops by more than 25 tokens within 24 hours of your purchase, you'll get a refund of tokens for the total difference > Each coin upgrade gives you 300 tokens, which you can use on Stickers or Souvenirs > You can create souvenir skins using your existing skins: Select a skin, then select a finished match and the gold stickers from that match will be applied to your skin > Souvenir Prices are determined by the demand for the specific gold team and autograph stickers involved, and the rarity of the weapon you choose for your souvenir > Teams, Players & the TO get 50% of the revenue from the Major Shop & Major Pass
0:58

NEW Cologne Major Shop Update: Everything you need to know ‼️ > No capsules, you have a shop where you can spend tokens > You can purchase whatever stickers you want with these tokens > Tokens can only be purchased when buying stickers and 100 tokens = 0.99$ > The prices of all stickers depend on relative demand > If a sticker price drops by more than 25 tokens within 24 hours of your purchase, you'll get a refund of tokens for the total difference > Each coin upgrade gives you 300 tokens, which you can use on Stickers or Souvenirs > You can create souvenir skins using your existing skins: Select a skin, then select a finished match and the gold stickers from that match will be applied to your skin > Souvenir Prices are determined by the demand for the specific gold team and autograph stickers involved, and the rarity of the weapon you choose for your souvenir > Teams, Players & the TO get 50% of the revenue from the Major Shop & Major Pass

Ozzny

357,730 Aufrufe • vor 2 Monaten

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."
2:22

Dylan on why access to the most intelligent AI models will determine who wins in every industry: "What really matters is access to the most intelligent tokens and leveraging them at the highest-value tasks. A lot of folks will want tokens, but the shitty SaaS startup in SF who is using Claude is not necessarily creating a ton of value, and therefore they're going to get priced out of tokens. Anthropic only released Mythos to certain companies for cybersecurity. You also don't want people to distill your model so you release them to a fewer and fewer customers. Those customers are now wrestling over the tokens. Token usage and therefore the additional value generated on top of those tokens aggregates among fewer and fewer companies. I don't have Mythos but you know who has Mythos, top banks. There are people like Ken Griffin of Citadel who can just sign a deal with OpenAI or Antrhopic for access to your models. I'll buy the first $10 billion worth of tokens each year. So whenever you release the model, I'll spend the first 10 billion tokens and then everyone else can get the model after that. Now, he's going to crush everyone in the markets. I can envision because I have an enterprise Anthropic contract and Anthropic people like me, they're willing to give us slightly earlier access or higher rate limits, and then my competitor doesn't have that, and I'm able to f*cking crush them."

Patrick OShaughnessy

36,100 Aufrufe • vor 3 Monaten

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang
2:41

MIT PhD student Alex Zhang reveals the scaling result where a model trained on short tasks generalizes to problems 100x longer for free: "If you're very clever about the design of your harness or how you use the language model, you can almost get scaling gains for free." "If you train a model naively, there's no tricks. It's just the same way you train a model on these RL environments. You just roll it out, and then you just get some reward." "If you train it on only short tasks, like only tasks that are 10,000 tokens long, and then you were to run it on a similar domain, but at a million tokens, or 10 million tokens, or 100,000 tokens, it generalizes really, really well. If you look at it compared to even the base transformer, you get way better generalization properties." "When the model uses an RLM (Recursive Language Model) after it's trained on these short tasks, it will see some kind of trajectory of actions that it does. Between these two problems of different lengths, the RLM learns to see them as almost the same problem." "Token for token, they're almost the same. You can describe it in code. In one code setting, maybe the for loop is a little bigger, but it's the same kind of code and it derives the constants from the data. There's no hard coding, so they literally look the same." alex zhang

MTS

99,784 Aufrufe • vor 16 Tagen

Confidential tokens preserve public verifiability while keeping amounts and balances encrypted. But to make them the new default, we need to rally the entire stack of tools we already use today, wallets, explorers, indexers, and more. It's a big lift, and it's happening now. The work the team at Blockscout 🔭 did is a step in that direction: they're the first block explorer to natively support ERC-7984 confidential tokens, verifiable transfers with encrypted amounts. We recorded a demo with DOSH | 🦇🔊 from the Blockscout team, walking through how it works, and what explorers become when encrypted tokens are the default. This demo shows the work Zama is doing across the board: onboarding the tooling the ecosystem already relies on, so confidentiality onchain becomes the new normal, not the exception.
1:02

Confidential tokens preserve public verifiability while keeping amounts and balances encrypted. But to make them the new default, we need to rally the entire stack of tools we already use today, wallets, explorers, indexers, and more. It's a big lift, and it's happening now. The work the team at Blockscout 🔭 did is a step in that direction: they're the first block explorer to natively support ERC-7984 confidential tokens, verifiable transfers with encrypted amounts. We recorded a demo with DOSH | 🦇🔊 from the Blockscout team, walking through how it works, and what explorers become when encrypted tokens are the default. This demo shows the work Zama is doing across the board: onboarding the tooling the ecosystem already relies on, so confidentiality onchain becomes the new normal, not the exception.

Zama

18,609 Aufrufe • vor 20 Tagen

Liquid AI (Liquid AI) CEO Ramin Hasani (Ramin) says: "We're bringing the cost of tokens to zero." "The axis was maximizing intelligence at all costs." Foundation models need to optimize across 3 axes: 1.) Intelligence and capability 2.) Efficiency and cost 3.) Substrate: where the intelligence actually runs "Efficiency is not an afterthought. Energy is not abundant." "If you maximize intelligence at all costs, that axis alone is not going to get you to the place that you want to go." "Efficiency and cost of intelligence as a first-class citizen, and not an afterthought." "Where does this intelligence system go? AI is majorly getting hosted in data centers, but you could also bring intelligence on phones, on laptops, on airplanes, on cars." "Imagine if it's not tokens that are actually important. It's just the outcome that actually matters." "Not just thinking about foundation models as the token machines that are generating money and revenue for the foundation model companies that are useless tokens, and getting them into the place where they can actually unlock true value for enterprises."
6:17

Liquid AI (Liquid AI) CEO Ramin Hasani (Ramin) says: "We're bringing the cost of tokens to zero." "The axis was maximizing intelligence at all costs." Foundation models need to optimize across 3 axes: 1.) Intelligence and capability 2.) Efficiency and cost 3.) Substrate: where the intelligence actually runs "Efficiency is not an afterthought. Energy is not abundant." "If you maximize intelligence at all costs, that axis alone is not going to get you to the place that you want to go." "Efficiency and cost of intelligence as a first-class citizen, and not an afterthought." "Where does this intelligence system go? AI is majorly getting hosted in data centers, but you could also bring intelligence on phones, on laptops, on airplanes, on cars." "Imagine if it's not tokens that are actually important. It's just the outcome that actually matters." "Not just thinking about foundation models as the token machines that are generating money and revenue for the foundation model companies that are useless tokens, and getting them into the place where they can actually unlock true value for enterprises."

Molly O’Shea

12,816 Aufrufe • vor 28 Tagen

Jensen Huang: “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed.” The Nvidia CEO expects his highly paid engineers to be spending at least HALF their salaries on tokens to supercharge their abilities. @jason: “ The conversation we've had on the pod a number of times is, ‘Oh my God, look at the token usage in our companies.’ It is growing massively.” “And some people are asking, ‘Hey, when I join a company, how many tokens do I get? Because I want to be an effective employee.’” “You've postulated, I believe, $75,000 in tokens for each engineer, something like that.” “So are you spending, at Nvidia, $1 billion, $2 billion on tokens for your engineering team right now?” Jensen: “We're trying to.” “Let me give you the thought experiment: Let's say you have a software engineer or AI researcher and you pay them $500,000 a year. We do that all the time.” “That $500,000 engineer, at the end of the year, I'm going to ask them, how much did you spend in tokens?” “If that person said, ‘$5,000,’ I will go ape… something else.” “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed. “And this is no different than one of our chip designers who says, ‘Guess what? I'm just going to use paper and pencil, I don't think I'm going to need any CAD tools.’” Jason: “This is a real paradigm shift, to start thinking about these all-star employees, it almost reminds me of what we learned in the NBA when LeBron James started spending a million dollars a year just on his health and his body, like in maintaining it. Here he is at age 41, still playing.” “These are incredible knowledge workers. Why wouldn't we give them superhuman abilities?”
1:18

Jensen Huang: “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed.” The Nvidia CEO expects his highly paid engineers to be spending at least HALF their salaries on tokens to supercharge their abilities. @jason: “ The conversation we've had on the pod a number of times is, ‘Oh my God, look at the token usage in our companies.’ It is growing massively.” “And some people are asking, ‘Hey, when I join a company, how many tokens do I get? Because I want to be an effective employee.’” “You've postulated, I believe, $75,000 in tokens for each engineer, something like that.” “So are you spending, at Nvidia, $1 billion, $2 billion on tokens for your engineering team right now?” Jensen: “We're trying to.” “Let me give you the thought experiment: Let's say you have a software engineer or AI researcher and you pay them $500,000 a year. We do that all the time.” “That $500,000 engineer, at the end of the year, I'm going to ask them, how much did you spend in tokens?” “If that person said, ‘$5,000,’ I will go ape… something else.” “If that $500,000 engineer did not consume at least $250,000 worth of tokens, I'm going to be deeply alarmed. “And this is no different than one of our chip designers who says, ‘Guess what? I'm just going to use paper and pencil, I don't think I'm going to need any CAD tools.’” Jason: “This is a real paradigm shift, to start thinking about these all-star employees, it almost reminds me of what we learned in the NBA when LeBron James started spending a million dollars a year just on his health and his body, like in maintaining it. Here he is at age 41, still playing.” “These are incredible knowledge workers. Why wouldn't we give them superhuman abilities?”

The All-In Podcast

96,648 Aufrufe • vor 4 Monaten

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?
0:26

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

157,390 Aufrufe • vor 2 Monaten

Your assets, arts and collectibles deserve more than just likes. They deserve onchain ownership —-Ownership on Solana. Developers, take a step forward. What’s stopping you from launching your NFTs and collectibles with Metaplex 🦾? Do you know that Metaplex 🦾 is the backbone of Solana digital collectibles, pioneering since its inception in 2021? Do you also know that almost all Fungible and non-fungible tokens you see today was built using metaplex. ==> From building, to minting to marketplace, Metaplex 🦾 is at the forefront of Solana’s NFTs and fungible tokens. —> You’ve got to bring your digital assets onchain for maximum scalability and security. The only way to do it right is with Metaplex 🦾. With its unique features, you as a developer can build secure and scalable Fungible and non-fungible tokens right on Solana. Take a step today, build with Metaplex 🦾. If it’s not with Metaplex 🦾, it is not right! Seeking to learn more about Metaplex 🦾 and how you can start building? —> Learn more with metaplex official links: Twitter: Website: Blog: Guides:
2:55

Your assets, arts and collectibles deserve more than just likes. They deserve onchain ownership —-Ownership on Solana. Developers, take a step forward. What’s stopping you from launching your NFTs and collectibles with Metaplex 🦾? Do you know that Metaplex 🦾 is the backbone of Solana digital collectibles, pioneering since its inception in 2021? Do you also know that almost all Fungible and non-fungible tokens you see today was built using metaplex. ==> From building, to minting to marketplace, Metaplex 🦾 is at the forefront of Solana’s NFTs and fungible tokens. —> You’ve got to bring your digital assets onchain for maximum scalability and security. The only way to do it right is with Metaplex 🦾. With its unique features, you as a developer can build secure and scalable Fungible and non-fungible tokens right on Solana. Take a step today, build with Metaplex 🦾. If it’s not with Metaplex 🦾, it is not right! Seeking to learn more about Metaplex 🦾 and how you can start building? —> Learn more with metaplex official links: Twitter: Website: Blog: Guides:

og.blessed | Solana Summit Africa🇳🇬

18,781 Aufrufe • vor 1 Jahr

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."
0:51

SK Hynix's Hoshik Kim admits high bandwidth memory "is not the answer" to the memory-wall problem, and SK is working on different solutions to solve the bottleneck. "No, actually, I kind of agree with Pat. So HBM is not the answer." [SK Hynix is admitting that HBM is a bad memory?] "Nothing is a bad memory. It's a good memory, but it's not the final answer. So actually, HBM will not solve the memory-wall problem. Memory-wall problem is an inherent AI problem, which you cannot avoid." "So we try our best to provide higher bandwidth with this memory, with HBM technology even though it has some limitation, as Pat already mentioned, with thermal, IOs, cost, a lot of stuff. "But that's the only thing we got right now to help ease this memory-wall problem, memory-bottleneck problem. But at the same time, we are working in different directions to solve this memory-wall problem with different solutions."

Fireside Alpha

140,200 Aufrufe • vor 11 Tagen

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."
2:08

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,110 Aufrufe • vor 2 Monaten

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.
0:23

A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces. So you add KV cache compression and evict 90% of the cached tokens. VRAM usage stays as is and GPU still runs out of memory. Why? (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. This sounds counterintuitive, but it follows directly from how production servers store the cache today. The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost for reasoning models. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention, which is the memory manager behind vLLM and most production servers. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when every slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with at least some survivor tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds. This introduces another bookkeeping cost that an in-order layout inherently avoids. So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server. There's another problem. Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected). But fast attention kernels used in production, like FlashAttention, never save those scores. They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast. So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide. NVIDIA published a method called TriAttention to solve both these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA write-up here: I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching. Read it below.

Avi Chawla

270,018 Aufrufe • vor 1 Monat

WE COULD NOT BELIEVE IT WHEN WE SAW THIS We were curious about Kimi K3's massive jump in consumed thinking tokens After taking a closer look, we found that Kimi K3 devotes 10x more thinking tokens to coding during reasoning than any other model we've seen before Not only that, but it has basically memorized all of the image IDs from Unsplash and, without a web search tool call, can reason about images with a higher accuracy than we've seen from any other model The image and video below are visualizations of its (very long) thinking trace for single-turn generation The resulting website selects images with greater relevance and accuracy than we've ever seen on DesignArena Forgive my all-caps heading, but I am excited about this and wanted you to read it too bc this is pretty gnarly Huge congratulations to the Kimi.ai team for this breakthrough and becoming the 1st-place model on Design Arena!
0:20

WE COULD NOT BELIEVE IT WHEN WE SAW THIS We were curious about Kimi K3's massive jump in consumed thinking tokens After taking a closer look, we found that Kimi K3 devotes 10x more thinking tokens to coding during reasoning than any other model we've seen before Not only that, but it has basically memorized all of the image IDs from Unsplash and, without a web search tool call, can reason about images with a higher accuracy than we've seen from any other model The image and video below are visualizations of its (very long) thinking trace for single-turn generation The resulting website selects images with greater relevance and accuracy than we've ever seen on DesignArena Forgive my all-caps heading, but I am excited about this and wanted you to read it too bc this is pretty gnarly Huge congratulations to the Kimi.ai team for this breakthrough and becoming the 1st-place model on Design Arena!

Grace Li

56,796 Aufrufe • vor 13 Tagen

Announcing Metropolis Part 1 – enabling users to immediately trade new tokens and markets. This is a major upgrade to the way Jupiter manages new tokens/markets and the token list. We are launching the new T0 API for new tokens, which allows you to query the chain to find the market with the most liquidity to swap directly. Tokens will be available using this feature immediately after they are launched. You can try this on today. This feature currently supports Raydium’s CPMM pools and Meteora Dynamic Pools/DLMM as well as Orca Whirlpools. Direct routing is enabled, so you might need SOL or USDC to trade these new tokens. We also shipped a few upgrades to that aims to reduce loading times. When you open the UI it will only load the strict list, instead of the entire 80K tokens on the all-list. We then use Typesense, a performant search engine to serve results from the all-list. If the all-list doesnt contain your desired token, it will then query the chain directly using the new T0 API and swap using direct routing. You should see improved loading times on the front-end and an overall much smoother experience looking for your tokens now. We will be rolling out Metropolis Part 2 and Ecosystem Token Lists next. For now, try out the T0 functionality and leave us feedback in Discord!
0:26

Announcing Metropolis Part 1 – enabling users to immediately trade new tokens and markets. This is a major upgrade to the way Jupiter manages new tokens/markets and the token list. We are launching the new T0 API for new tokens, which allows you to query the chain to find the market with the most liquidity to swap directly. Tokens will be available using this feature immediately after they are launched. You can try this on today. This feature currently supports Raydium’s CPMM pools and Meteora Dynamic Pools/DLMM as well as Orca Whirlpools. Direct routing is enabled, so you might need SOL or USDC to trade these new tokens. We also shipped a few upgrades to that aims to reduce loading times. When you open the UI it will only load the strict list, instead of the entire 80K tokens on the all-list. We then use Typesense, a performant search engine to serve results from the all-list. If the all-list doesnt contain your desired token, it will then query the chain directly using the new T0 API and swap using direct routing. You should see improved loading times on the front-end and an overall much smoother experience looking for your tokens now. We will be rolling out Metropolis Part 2 and Ecosystem Token Lists next. For now, try out the T0 functionality and leave us feedback in Discord!

Jupiter

249,148 Aufrufe • vor 2 Jahren

"I burned roughly $5,000 of API tokens in a few hours for $200. It's 100x subsidized. This is insanity" Kain on running 500+ agents to decompose an entire codebase on Claude Opus 4.8 "I set up my own personal account to test this on Opus 4.8. I know enough that I can come up with a task that's gonna burn a fuckload. In the course of four hours I used something like 200 million tokens and didn't hit any limits" "It was hundreds of agents, like 500 or 600 agents decomposing a codebase. It would have been somewhere on the order of $5,000 with API metered tokens, and I know how much tokens cost because that's where I live" "I ran this overnight multiple times. It's very possible I could have burned through $20,000 worth of API credits for $200. I knew it was subsidized, but it's subsidized to a level that's crazy. It's 100x subsidized"
2:06

"I burned roughly $5,000 of API tokens in a few hours for $200. It's 100x subsidized. This is insanity" Kain on running 500+ agents to decompose an entire codebase on Claude Opus 4.8 "I set up my own personal account to test this on Opus 4.8. I know enough that I can come up with a task that's gonna burn a fuckload. In the course of four hours I used something like 200 million tokens and didn't hit any limits" "It was hundreds of agents, like 500 or 600 agents decomposing a codebase. It would have been somewhere on the order of $5,000 with API metered tokens, and I know how much tokens cost because that's where I live" "I ran this overnight multiple times. It's very possible I could have burned through $20,000 worth of API credits for $200. I knew it was subsidized, but it's subsidized to a level that's crazy. It's 100x subsidized"

Laura Shin

85,115 Aufrufe • vor 1 Monat