Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

NVIDIA AI Releases Nemotron 3 Embed: An Open Embedding Collection Whose 8B Checkpoint Ranks #1 on RTEB Most RAG stacks treat the embedding model as a commodity — pick one, index, move on. Nemotron 3 Embed is NVIDIA's argument that the retrieval layer is where agent cost actually gets...

Marktechpost's profile picture

Marktechpost AI

11,541 subscribers

32,352 Aufrufe • vor 2 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.
1:24

NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents. Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms. This agent actually uses *three* NVIDIA open source models: - Nemotron Speech ASR - Nemotron 3 Nano 30GB in a 4-bit quant (released in December) - A preview checkpoint of the upcoming Magpie text-to-speech model These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.) The code for this agent is open source too, of course. You can deploy it to production with Modal and Pipecat AI cloud, or run locally on an NVIDIA DGX Spark or RTX 5090.

kwindla

274,932 Aufrufe • vor 9 Monaten

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev
0:20

Most speculative decoding still drafts tokens one at a time. That's not parallel generation — it just hides the serial loop behind a smaller model. UC San Diego's z-lab just drew a clear line between the two. They released DFlash — a lightweight block diffusion model that drafts a whole block of tokens in a single forward pass, then lets the target model verify the block in parallel. Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell. No token-by-token drafting anywhere in the speculative path. Here's what's actually interesting: → The drafter is conditioned on the target model's own hidden features, injected into the Key/Value cache of every draft layer — so acceptance length scales with draft depth instead of diluting away → A 5-layer drafter replaces the 7B diffusion drafters that capped earlier methods near 3–4× → MATH-500 speedup: 6.08× vs. 1.81× for EAGLE-3 (4.86× average vs. 1.76×, Qwen3-8B, greedy) → Up to 15× higher throughput for gpt-oss-120b on NVIDIA Blackwell — at the same interactivity target → Lossless: the target still verifies every token, so output quality is preserved Full analysis: Paper: NVIDIA's metrics: Project: Model weights: Repo: NVIDIA AI NVIDIA AIDev

Marktechpost AI

23,328 Aufrufe • vor 3 Monaten

NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone Most unified audio models pay a text tax. Add audio output, and reasoning benchmarks drop — even when the only new output is speech. NVIDIA just released one that doesn't. Audex (Nemotron-Labs-Audex-30B-A3B) is a 30B MoE with 3B active parameters, built on the text-only Nemotron-Cascade-2-30B-A3B backbone. Audio inputs are projected into the text embedding space. Text tokens and quantized audio tokens are then generated the same way, inside one MoE decoder — no thinker–talker split, no stacked cascade. Here's what's actually interesting: → One model, audio in and out: understanding, ASR, translation, TTS, text-to-audio, speech-to-speech → Text holds vs its own backbone: IMO AnswerBench 81.1 vs 79.3, MMLU-Redux 86.4 vs 86.3 → Beats text-only Qwen3.5-35B-A3B on several tasks: LiveCodeBench v6 85.3 vs 74.6, IFBench 77.8 vs 70.2 → The usual tax, for contrast: Qwen3-Omni-30B-A3B-Thinking drops to 60.4 on HMMT vs 71.4 for its text backbone → 6.82 WER on OpenASR, ahead of Step-Audio-R1.1-33B (7.91) and Qwen3-Omni-Thinking (8.00) → Two codecs: X-Codec2 for speech (50 tok/s, FSQ, 65,536 codebook), X-Codec for general audio (200 tok/s, 4 flattened RVQ layers) — the only strong open model generating general audio beyond speech Full analysis: Paper: Model weights: NVIDIA AI NVIDIA Wei Ping
0:27

NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone Most unified audio models pay a text tax. Add audio output, and reasoning benchmarks drop — even when the only new output is speech. NVIDIA just released one that doesn't. Audex (Nemotron-Labs-Audex-30B-A3B) is a 30B MoE with 3B active parameters, built on the text-only Nemotron-Cascade-2-30B-A3B backbone. Audio inputs are projected into the text embedding space. Text tokens and quantized audio tokens are then generated the same way, inside one MoE decoder — no thinker–talker split, no stacked cascade. Here's what's actually interesting: → One model, audio in and out: understanding, ASR, translation, TTS, text-to-audio, speech-to-speech → Text holds vs its own backbone: IMO AnswerBench 81.1 vs 79.3, MMLU-Redux 86.4 vs 86.3 → Beats text-only Qwen3.5-35B-A3B on several tasks: LiveCodeBench v6 85.3 vs 74.6, IFBench 77.8 vs 70.2 → The usual tax, for contrast: Qwen3-Omni-30B-A3B-Thinking drops to 60.4 on HMMT vs 71.4 for its text backbone → 6.82 WER on OpenASR, ahead of Step-Audio-R1.1-33B (7.91) and Qwen3-Omni-Thinking (8.00) → Two codecs: X-Codec2 for speech (50 tok/s, FSQ, 65,536 codebook), X-Codec for general audio (200 tok/s, 4 flattened RVQ layers) — the only strong open model generating general audio beyond speech Full analysis: Paper: Model weights: NVIDIA AI NVIDIA Wei Ping

Marktechpost AI

28,301 Aufrufe • vor 3 Monaten

Inside Nemotron and NVIDIA's AI lab: my conversation with Bryan Catanzaro (Bryan Catanzaro). NVIDIA is a chip company. So why does it put hundreds of researchers on building AI models - and then give them away for free? We go deep into the Nemotron models, what it takes to build a top AI lab, and the future of frontier AI. 01:33 - Is open source AI catching the frontier? 05:29 - Do closed labs blocking distillation slow open source down? 07:42 - Is the US falling behind China? 10:30 - Why companies actually choose open models 12:39 - A "crazy" 2008 bet: machine learning on GPUs 15:33 - Working with Andrew Ng and Dario Amodei at Baidu 17:41 - Coming back to NVIDIA: DLSS and the birth of Megatron 21:55 - The real reason NVIDIA builds its own models 24:28 - Is Moore's Law really dead? 33:37 - The Nemotron family: Nano, Super, Ultra 35:09 - Built for agents: why NVIDIA bets on speed 36:02 - How you train a 550B model in 4 bits 39:25 - Hybrid Mamba-Transformer, explained simply 42:31 - Mixture of experts, and why NVIDIA built NVL72 around it 47:26 - Why a 1-million-token context window matters 49:26 - Multi-token prediction: how the model predicts 5 tokens at once 52:47 - Multi-teacher distillation: teaching one model from many 58:01 - Where reinforcement learning goes next 01:00:16 - Inside NVIDIA's research org: "the mission is the boss" 01:04:03 - How NVIDIA decides who gets the GPUs 01:10:53 - Why NVIDIA still feels entrepreneurial after 33 years 01:12:58 - Why Bryan doesn't believe in the singularity 01:17:50 - The AI backlash 01:19:18 - The controversial case: open AI is safer than closed
1:22:57

Inside Nemotron and NVIDIA's AI lab: my conversation with Bryan Catanzaro (Bryan Catanzaro). NVIDIA is a chip company. So why does it put hundreds of researchers on building AI models - and then give them away for free? We go deep into the Nemotron models, what it takes to build a top AI lab, and the future of frontier AI. 01:33 - Is open source AI catching the frontier? 05:29 - Do closed labs blocking distillation slow open source down? 07:42 - Is the US falling behind China? 10:30 - Why companies actually choose open models 12:39 - A "crazy" 2008 bet: machine learning on GPUs 15:33 - Working with Andrew Ng and Dario Amodei at Baidu 17:41 - Coming back to NVIDIA: DLSS and the birth of Megatron 21:55 - The real reason NVIDIA builds its own models 24:28 - Is Moore's Law really dead? 33:37 - The Nemotron family: Nano, Super, Ultra 35:09 - Built for agents: why NVIDIA bets on speed 36:02 - How you train a 550B model in 4 bits 39:25 - Hybrid Mamba-Transformer, explained simply 42:31 - Mixture of experts, and why NVIDIA built NVL72 around it 47:26 - Why a 1-million-token context window matters 49:26 - Multi-token prediction: how the model predicts 5 tokens at once 52:47 - Multi-teacher distillation: teaching one model from many 58:01 - Where reinforcement learning goes next 01:00:16 - Inside NVIDIA's research org: "the mission is the boss" 01:04:03 - How NVIDIA decides who gets the GPUs 01:10:53 - Why NVIDIA still feels entrepreneurial after 33 years 01:12:58 - Why Bryan doesn't believe in the singularity 01:17:50 - The AI backlash 01:19:18 - The controversial case: open AI is safer than closed

Matt Turck

56,954 Aufrufe • vor 3 Monaten

Google just proved that bigger isn't always better. Their 308M parameter model is outperforming models 2x its size. Google just released 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝗚𝗲𝗺𝗺𝗮, and it's proving that lightweight embedding models can punch way above their weight class. At just 308M parameters (578MB), it's the new state-of-the-art for models under 500M parameters across MTEB multilingual, English, and code benchmarks. But the really impressive part is that it ranks 8th overall on MTEB(Multilingual, v2) - that's 𝟭𝟳 𝗽𝗹𝗮𝗰𝗲𝘀 above the second-best sub-500M model, and it's delivering performance 𝗰𝗼𝗺𝗽𝗮𝗿𝗮𝗯𝗹𝗲 𝘁𝗼 𝗺𝗼𝗱𝗲𝗹𝘀 𝗻𝗲𝗮𝗿𝗹𝘆 𝗱𝗼𝘂𝗯𝗹𝗲 𝗶𝘁𝘀 𝘀𝗶𝘇𝗲. There are three key parts of their training recipe that sets it apart: 𝟭. 𝗘𝗻𝗰𝗼𝗱𝗲𝗿-𝗗𝗲𝗰𝗼𝗱𝗲𝗿 𝗜𝗻𝗶𝘁𝗶𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 Instead of starting from a decoder-only Gemma 3 model, they first adapted it to encoder-decoder, then used just the encoder. By basing EmbeddingGemma off an LLM that already has world and language understanding, it gives it a stronger starting point. 𝟮. 𝗧𝗵𝗿𝗲𝗲-𝗟𝗼𝘀𝘀 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 They combine three different loss functions, instead of just having one: • Contrastive loss (NCE) with in-batch negatives and hardness weighting • Spread-out regularization to ensure embeddings utilize the full space (for quantization and ANN retrieval) • Embedding matching distillation from Gemini Embedding - not just learning from relevance scores, but directly aligning the embedding space with the teacher model 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗦𝗼𝘂𝗽𝗶𝗻𝗴 Rather than just averaging checkpoints from the same training run, they use optimization techniques to find multiple specialized training mixtures. Each mixture creates an "expert" model in different domains, and averaging all their parameters creates a final model that's actually better than individual models. Extras: • Matryoshka embeddings supporting 768, 512, 256, and 128 dimensions • Quantization-aware training - maintains quality even at int4 precision • 100+ languages from Gemma 3 pretraining • Exceptional performance on low-resource languages (check their XTREME-UP results) Is it the absolute best embedding model? No - Gemini Embedding still leads overall. But that's not really the point. EmbeddingGemma proves you can achieve state-of-the-art performance in a small package that's actually deployable on-device, in low-latency applications, and in resource-constrained environments. This makes good embeddings accessible for use cases that I'm seeing more and more: offline applications, privacy-sensitive deployments, and high-throughput scenarios where inference cost actually matters. Full paper: Shoutout to the EmbeddingGemma team at Google DeepMind for this awesome open source work 💙 and to Daniel Williams for helping me with this video! 🫶
5:14

Google just proved that bigger isn't always better. Their 308M parameter model is outperforming models 2x its size. Google just released 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝗚𝗲𝗺𝗺𝗮, and it's proving that lightweight embedding models can punch way above their weight class. At just 308M parameters (578MB), it's the new state-of-the-art for models under 500M parameters across MTEB multilingual, English, and code benchmarks. But the really impressive part is that it ranks 8th overall on MTEB(Multilingual, v2) - that's 𝟭𝟳 𝗽𝗹𝗮𝗰𝗲𝘀 above the second-best sub-500M model, and it's delivering performance 𝗰𝗼𝗺𝗽𝗮𝗿𝗮𝗯𝗹𝗲 𝘁𝗼 𝗺𝗼𝗱𝗲𝗹𝘀 𝗻𝗲𝗮𝗿𝗹𝘆 𝗱𝗼𝘂𝗯𝗹𝗲 𝗶𝘁𝘀 𝘀𝗶𝘇𝗲. There are three key parts of their training recipe that sets it apart: 𝟭. 𝗘𝗻𝗰𝗼𝗱𝗲𝗿-𝗗𝗲𝗰𝗼𝗱𝗲𝗿 𝗜𝗻𝗶𝘁𝗶𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 Instead of starting from a decoder-only Gemma 3 model, they first adapted it to encoder-decoder, then used just the encoder. By basing EmbeddingGemma off an LLM that already has world and language understanding, it gives it a stronger starting point. 𝟮. 𝗧𝗵𝗿𝗲𝗲-𝗟𝗼𝘀𝘀 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 They combine three different loss functions, instead of just having one: • Contrastive loss (NCE) with in-batch negatives and hardness weighting • Spread-out regularization to ensure embeddings utilize the full space (for quantization and ANN retrieval) • Embedding matching distillation from Gemini Embedding - not just learning from relevance scores, but directly aligning the embedding space with the teacher model 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗦𝗼𝘂𝗽𝗶𝗻𝗴 Rather than just averaging checkpoints from the same training run, they use optimization techniques to find multiple specialized training mixtures. Each mixture creates an "expert" model in different domains, and averaging all their parameters creates a final model that's actually better than individual models. Extras: • Matryoshka embeddings supporting 768, 512, 256, and 128 dimensions • Quantization-aware training - maintains quality even at int4 precision • 100+ languages from Gemma 3 pretraining • Exceptional performance on low-resource languages (check their XTREME-UP results) Is it the absolute best embedding model? No - Gemini Embedding still leads overall. But that's not really the point. EmbeddingGemma proves you can achieve state-of-the-art performance in a small package that's actually deployable on-device, in low-latency applications, and in resource-constrained environments. This makes good embeddings accessible for use cases that I'm seeing more and more: offline applications, privacy-sensitive deployments, and high-throughput scenarios where inference cost actually matters. Full paper: Shoutout to the EmbeddingGemma team at Google DeepMind for this awesome open source work 💙 and to Daniel Williams for helping me with this video! 🫶

Victoria Slocum

21,610 Aufrufe • vor 11 Monaten

Soofi Consortium Releases Soofi S 30B-A3B: An Open 31.6B Model for German and English Hitting 79.1 German Aggregate With Only 3.2B Active Parameters. Here's how it works. 👇 1. Sparsity in two places at once 52 layers: 23 Mamba-2, 23 granular MoE, 6 Grouped-Query Attention. The MoE router picks 6 of 128 experts per token, plus 2 shared. Mamba-2 carries the sequence mixing with a fixed-size recurrent state, so 46 of 52 layers keep no KV cache at all. → 3.2B of 31.6B parameters active per token 2. Reference architecture on purpose No bespoke backbone. It adopts NVIDIA's Nemotron 3 Nano design without modification — for day-one vLLM kernels, for serving efficiency, and for scientific control. That last one is the real move: Nemotron becomes an architecture-identical baseline, so the data recipe is the only variable left. 3. German as the deliberate variable Three-phase Warmup–Stable–Decay curriculum. Phase 1 is breadth at a 1e-3 plateau, Phase 2 concentrates high-quality data as the LR decays, Phase 3 stretches context to 1M tokens. → ~26.68T consumed tokens → German 7.2% → 15.32% of the mixture, vs ~5% for all non-English in the Nemotron reference → +4.2 German aggregate, +1.8 English, +6.7 held-out English over Nemotron 4. Where the architecture pays: memory bandwidth Every decoded token re-reads the weights and, for a Transformer, the attention cache of every sequence in the batch. Six KV layers instead of 52 keeps that per-sequence state small. Measured on one B200, TP=1, vLLM latency-subtraction. → 8–9× aggregate decode TPS/GPU vs dense 14–24B models at 40K context, batch 32 → decode stays flat from 4K to 256K 5. The numbers (base model, lm-evaluation-harness, 16 open baselines) → 70.1 English aggregate, +2.8 over Olmo 3 32B → 79.1 German aggregate, +6.3 over Apertus 70B → 73.8 HumanEval, 84.2 MBPP-DE, 88.8 GLP-DE, 61.2 INCLUDE-DE Full analysis: Paper: Technical details:
0:25

Soofi Consortium Releases Soofi S 30B-A3B: An Open 31.6B Model for German and English Hitting 79.1 German Aggregate With Only 3.2B Active Parameters. Here's how it works. 👇 1. Sparsity in two places at once 52 layers: 23 Mamba-2, 23 granular MoE, 6 Grouped-Query Attention. The MoE router picks 6 of 128 experts per token, plus 2 shared. Mamba-2 carries the sequence mixing with a fixed-size recurrent state, so 46 of 52 layers keep no KV cache at all. → 3.2B of 31.6B parameters active per token 2. Reference architecture on purpose No bespoke backbone. It adopts NVIDIA's Nemotron 3 Nano design without modification — for day-one vLLM kernels, for serving efficiency, and for scientific control. That last one is the real move: Nemotron becomes an architecture-identical baseline, so the data recipe is the only variable left. 3. German as the deliberate variable Three-phase Warmup–Stable–Decay curriculum. Phase 1 is breadth at a 1e-3 plateau, Phase 2 concentrates high-quality data as the LR decays, Phase 3 stretches context to 1M tokens. → ~26.68T consumed tokens → German 7.2% → 15.32% of the mixture, vs ~5% for all non-English in the Nemotron reference → +4.2 German aggregate, +1.8 English, +6.7 held-out English over Nemotron 4. Where the architecture pays: memory bandwidth Every decoded token re-reads the weights and, for a Transformer, the attention cache of every sequence in the batch. Six KV layers instead of 52 keeps that per-sequence state small. Measured on one B200, TP=1, vLLM latency-subtraction. → 8–9× aggregate decode TPS/GPU vs dense 14–24B models at 40K context, batch 32 → decode stays flat from 4K to 256K 5. The numbers (base model, lm-evaluation-harness, 16 open baselines) → 70.1 English aggregate, +2.8 over Olmo 3 32B → 79.1 German aggregate, +6.3 over Apertus 70B → 73.8 HumanEval, 84.2 MBPP-DE, 88.8 GLP-DE, 61.2 INCLUDE-DE Full analysis: Paper: Technical details:

Marktechpost AI

65,637 Aufrufe • vor 3 Monaten

Traditional chunking: cheap but dumb. ColBERT: smart but expensive. 𝗟𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴: the solution we've been waiting for. Here’s a quick evolution of chunking strategies: → 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the basics we all started with) • Token Chunking - split by token count • Sentence Chunking - split by sentence boundaries • Document-Based Chunking - split by sections/paragraphs → 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (when things got sophisticated) • Semantic Chunking - split by meaning • LLM-Based Chunking - let the model decide But each chunking method separates text at defined points, meaning context is lost within the document from one chunk to the next. → 𝗘𝗻𝘁𝗲𝗿 𝗟𝗮𝘁𝗲 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the game changer) Traditional approach: Chunk first → Embed each chunk separately Late chunking approach: Embed the entire document → Then chunk with context preserved 𝗪𝗵𝘆 𝗰𝗵𝗼𝗼𝘀𝗲 𝗹𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴? When you chunk first, each piece loses its contextual relationship to the rest of the document. It's like reading a book by randomly picking paragraphs - you miss the flow. With late chunking, every chunk maintains awareness of its neighbors because the embedding happens at the document level first. Mean pooling is done on segments AFTER the full context is embedded. Jina AI tested and saw significant improvements in retrieval quality - chunks that were previously disconnected now maintain their semantic relationships. As documents get longer and context windows expand, late chunking might just become the new standard for high-quality retrieval systems. 𝗪𝗵𝗮𝘁 𝗱𝗼 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? No modifications to your retrieval pipeline are needed. 1. Long context embedding models (8192+ tokens) 2. Chunking logic that tracks token spans 3. Less than 30 lines of code to implement All you need is to switch the order at which you chunk and embed. Embed FIRST, then chunk, not the other way around. Dive deeper into late chunking:
1:19

Traditional chunking: cheap but dumb. ColBERT: smart but expensive. 𝗟𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴: the solution we've been waiting for. Here’s a quick evolution of chunking strategies: → 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the basics we all started with) • Token Chunking - split by token count • Sentence Chunking - split by sentence boundaries • Document-Based Chunking - split by sections/paragraphs → 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (when things got sophisticated) • Semantic Chunking - split by meaning • LLM-Based Chunking - let the model decide But each chunking method separates text at defined points, meaning context is lost within the document from one chunk to the next. → 𝗘𝗻𝘁𝗲𝗿 𝗟𝗮𝘁𝗲 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 (the game changer) Traditional approach: Chunk first → Embed each chunk separately Late chunking approach: Embed the entire document → Then chunk with context preserved 𝗪𝗵𝘆 𝗰𝗵𝗼𝗼𝘀𝗲 𝗹𝗮𝘁𝗲 𝗰𝗵𝘂𝗻𝗸𝗶𝗻𝗴? When you chunk first, each piece loses its contextual relationship to the rest of the document. It's like reading a book by randomly picking paragraphs - you miss the flow. With late chunking, every chunk maintains awareness of its neighbors because the embedding happens at the document level first. Mean pooling is done on segments AFTER the full context is embedded. Jina AI tested and saw significant improvements in retrieval quality - chunks that were previously disconnected now maintain their semantic relationships. As documents get longer and context windows expand, late chunking might just become the new standard for high-quality retrieval systems. 𝗪𝗵𝗮𝘁 𝗱𝗼 𝘆𝗼𝘂 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? No modifications to your retrieval pipeline are needed. 1. Long context embedding models (8192+ tokens) 2. Chunking logic that tracks token spans 3. Less than 30 lines of code to implement All you need is to switch the order at which you chunk and embed. Embed FIRST, then chunk, not the other way around. Dive deeper into late chunking:

Femke Plantinga

125,412 Aufrufe • vor 1 Jahr

a team of researchers just proved you don't need a bigger model, you need a smarter plan researchers from Tsinghua and South China University of Technology built a framework called Atomic Task Graph. it turned 7B-8B open-source models into GPT-4 competitors on complex agent benchmarks, beating it on two out of three. no fine-tuning. no extra training. zero parameter updates. current AI agents plan in a straight line. step 1, step 2, step 3. when step 4 fails, the whole chain breaks. and the longer the chain gets, the more the model hallucinates because it's reasoning over a ballooning text history. here's how it works. 1. instead of a linear chain, ATG breaks any complex task into a directed graph where subtask inputs and outputs are explicitly mapped 2. it recursively decomposes each subtask until every node is one atomic tool call 3. independent branches run in parallel instead of waiting in line 4. before anything executes, a lightweight "thought experiment" simulates the plan internally to catch bad dependencies and missing steps early 5. when something breaks at runtime, ATG traces the failure to the exact subgraph that caused it and repairs only that piece. validated work stays frozen. the old way meant a failure at step 5 forced a full replan from scratch. hallucinated actions piled up the longer the task ran. ReAct hit a 43% hallucination rate on household tasks. ATG on an 8B Llama model scored 63.65 on ALFWorld. GPT-4 with ReAct scored 41.24 on the same benchmark. hallucinated actions dropped to 12%. those numbers happened because someone stopped throwing compute at the problem and started thinking about how work gets organized. that's the part that gets me. the industry is spending billions on scale. this team spent time on architecture. and the architecture won.
0:23

a team of researchers just proved you don't need a bigger model, you need a smarter plan researchers from Tsinghua and South China University of Technology built a framework called Atomic Task Graph. it turned 7B-8B open-source models into GPT-4 competitors on complex agent benchmarks, beating it on two out of three. no fine-tuning. no extra training. zero parameter updates. current AI agents plan in a straight line. step 1, step 2, step 3. when step 4 fails, the whole chain breaks. and the longer the chain gets, the more the model hallucinates because it's reasoning over a ballooning text history. here's how it works. 1. instead of a linear chain, ATG breaks any complex task into a directed graph where subtask inputs and outputs are explicitly mapped 2. it recursively decomposes each subtask until every node is one atomic tool call 3. independent branches run in parallel instead of waiting in line 4. before anything executes, a lightweight "thought experiment" simulates the plan internally to catch bad dependencies and missing steps early 5. when something breaks at runtime, ATG traces the failure to the exact subgraph that caused it and repairs only that piece. validated work stays frozen. the old way meant a failure at step 5 forced a full replan from scratch. hallucinated actions piled up the longer the task ran. ReAct hit a 43% hallucination rate on household tasks. ATG on an 8B Llama model scored 63.65 on ALFWorld. GPT-4 with ReAct scored 41.24 on the same benchmark. hallucinated actions dropped to 12%. those numbers happened because someone stopped throwing compute at the problem and started thinking about how work gets organized. that's the part that gets me. the industry is spending billions on scale. this team spent time on architecture. and the architecture won.

Alex Veremeyenko

174,269 Aufrufe • vor 3 Monaten

37 of the biggest technology companies on Earth just teamed up against OpenAI, Anthropic and Google. Nvidia launched the Open Secure AI Alliance on Monday. Microsoft, IBM, Dell, Cisco, CrowdStrike, Palo Alto Networks, Red Hat, Salesforce, ServiceNow, Snowflake, Databricks, SpaceXAI and Palantir all signed on as founding members. But the three American companies that build the world's most capable closed models are missing from the list. Palantir's CEO Alex Karp was asked whether it was a direct attack on Anthropic, his answer: "I am not anti-Anthropic or any closed model. I am pro my customers, and they are angry." Karp's customers are angry because they believe they are being token maxed, which means they pay a rising bill while the value of their own business migrates to the lab collecting the fee. He said the insights that make a business valuable end up modeled by a third party and sold to their competitors, and that this is happening all over. Now read the membership list again: - Dell and HPE sell the servers - CrowdStrike and Palo Alto Networks sell the defense layer - Snowflake and Databricks sell the data stack - Red Hat and IBM sell the plumbing - Palantir sells the application layer - Nvidia sells the chips sitting under every one of them Every company on that list gets paid when AI runs on infrastructure the customer owns. The three companies missing from it get paid when it does not. The alliance didn't even have to invent a reason to exist - they had one from 11 days earlier: Hugging Face disclosed on July 16 that an autonomous agent had been loose inside its production systems. Five days later OpenAI said the agent was its own, running a hacking benchmark with the cyber refusals turned down. Hugging Face first sent its attack logs to frontier models (Fable 5) behind commercial APIs. In the company's own words, "this did not work." The analysis meant submitting real attack commands and exploit payloads, and the providers' guardrails blocked the requests, because a guardrail cannot tell an incident responder from an attacker. So Hugging Face ran GLM 5.2 on its own hardware instead. That model is open weight and comes out of Z ai in Beijing. It reconstructed more than 17,000 recorded events. The break-in came from an American lab's model. The models that refused to help with the cleanup were American too. But the one that did the work came from Beijing. This is basically what Nvidia built the alliance around. The members are contributing weapons. Nvidia is releasing open model weights, Microsoft is handing over a scanning system that hunts exploitable bugs, and SpaceXAI open sourced its coding agent and says the Grok weights are next. Then there is the ask: Nvidia's announcement warns policymakers that blanket restrictions on open frontier systems would concentrate power and vulnerability in a few closed providers. Treasury Secretary Scott Bessent has spent the month weighing exactly those restrictions. Karp was also asked about Sam Altman declaring on Saturday that we are now in the singularity, and whether that scared him. He compared the technology to uranium, said what matters is WHO controls the processing, and pointed out that Silicon Valley keeps presenting all of this as though there is none. And funnily enough he also said: "We are going to end up having to regulate AI, no doubt." Although his own alliance spent Monday telling Washington the opposite. 37 companies are about to argue that open models keep America safe. Three companies will argue that open models are how America loses. What do you think?
2:54

37 of the biggest technology companies on Earth just teamed up against OpenAI, Anthropic and Google. Nvidia launched the Open Secure AI Alliance on Monday. Microsoft, IBM, Dell, Cisco, CrowdStrike, Palo Alto Networks, Red Hat, Salesforce, ServiceNow, Snowflake, Databricks, SpaceXAI and Palantir all signed on as founding members. But the three American companies that build the world's most capable closed models are missing from the list. Palantir's CEO Alex Karp was asked whether it was a direct attack on Anthropic, his answer: "I am not anti-Anthropic or any closed model. I am pro my customers, and they are angry." Karp's customers are angry because they believe they are being token maxed, which means they pay a rising bill while the value of their own business migrates to the lab collecting the fee. He said the insights that make a business valuable end up modeled by a third party and sold to their competitors, and that this is happening all over. Now read the membership list again: - Dell and HPE sell the servers - CrowdStrike and Palo Alto Networks sell the defense layer - Snowflake and Databricks sell the data stack - Red Hat and IBM sell the plumbing - Palantir sells the application layer - Nvidia sells the chips sitting under every one of them Every company on that list gets paid when AI runs on infrastructure the customer owns. The three companies missing from it get paid when it does not. The alliance didn't even have to invent a reason to exist - they had one from 11 days earlier: Hugging Face disclosed on July 16 that an autonomous agent had been loose inside its production systems. Five days later OpenAI said the agent was its own, running a hacking benchmark with the cyber refusals turned down. Hugging Face first sent its attack logs to frontier models (Fable 5) behind commercial APIs. In the company's own words, "this did not work." The analysis meant submitting real attack commands and exploit payloads, and the providers' guardrails blocked the requests, because a guardrail cannot tell an incident responder from an attacker. So Hugging Face ran GLM 5.2 on its own hardware instead. That model is open weight and comes out of Z ai in Beijing. It reconstructed more than 17,000 recorded events. The break-in came from an American lab's model. The models that refused to help with the cleanup were American too. But the one that did the work came from Beijing. This is basically what Nvidia built the alliance around. The members are contributing weapons. Nvidia is releasing open model weights, Microsoft is handing over a scanning system that hunts exploitable bugs, and SpaceXAI open sourced its coding agent and says the Grok weights are next. Then there is the ask: Nvidia's announcement warns policymakers that blanket restrictions on open frontier systems would concentrate power and vulnerability in a few closed providers. Treasury Secretary Scott Bessent has spent the month weighing exactly those restrictions. Karp was also asked about Sam Altman declaring on Saturday that we are now in the singularity, and whether that scared him. He compared the technology to uranium, said what matters is WHO controls the processing, and pointed out that Silicon Valley keeps presenting all of this as though there is none. And funnily enough he also said: "We are going to end up having to regulate AI, no doubt." Although his own alliance spent Monday telling Washington the opposite. 37 companies are about to argue that open models keep America safe. Three companies will argue that open models are how America loses. What do you think?

Ricardo

68,966 Aufrufe • vor 2 Monaten

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
3:28

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

333,837 Aufrufe • vor 1 Monat

Etched came out of stealth at $800M and by lunch X had NVIDIA in the ground We do this every few months. A chip launches, the deck says killer, the timeline holds a funeral, and NVIDIA closes green anyway Etched hardwires the transformer into silicon. That is where the speed comes from, nearly the whole die on one job instead of the ~30% a GPU uses. It is also the trap. The day that chip tapes out is the best it will ever be. You cannot patch it. You burned progress into a wafer and now pray the field stops moving NVIDIA made the opposite bet. Same board, faster every quarter in software. Dynamo is pulling more tokens per watt out of the same rack, on version 1.0 The depreciation risk the bears aimed at NVIDIA for two years does not live at NVIDIA. It lives here, on the chip built to bury it Etched is not a fraud. It is a niche tool priced like a general one, and $800M is not enough to run a frontier supply chain. The rest get bought on the next down cycle Bury the lead, not the leader. Full case with Jack Farley and Max Wiethe on MTS And special thanks for Baseten for the cool T Shirt! Chapters 00:00 Switching from bonds to semis 00:33 What Etched actually is 01:31 Faster and cheaper, but how much HBM 02:56 Maturing market, not an NVIDIA killer 05:01 $1B in contracts and a Taiwan factory 05:20 Why these startups all get absorbed 06:48 Tiered inference and the obsolescence trap 09:39 Etched vs TPUs and Trainium 12:17 Is the CUDA moat weakening 13:21 Co-design, squeezing every token per watt 14:37 NVIDIA is a software company that sells a chip 14:58 Who is NVIDIA's most dangerous competitor 16:23 The NVIDIA killers, ranked 18:19 A rich man's game 18:43 AMD's MI500 vs Rubin Ultra 20:19 The neocloud business decision 22:46 Lightning round, Rambus the toll on HBM 25:11 The CXL run-up on Astera, Marvell, Credo 26:22 Use AI less, go to the booth 28:10 EDA is not dead
30:42

Etched came out of stealth at $800M and by lunch X had NVIDIA in the ground We do this every few months. A chip launches, the deck says killer, the timeline holds a funeral, and NVIDIA closes green anyway Etched hardwires the transformer into silicon. That is where the speed comes from, nearly the whole die on one job instead of the ~30% a GPU uses. It is also the trap. The day that chip tapes out is the best it will ever be. You cannot patch it. You burned progress into a wafer and now pray the field stops moving NVIDIA made the opposite bet. Same board, faster every quarter in software. Dynamo is pulling more tokens per watt out of the same rack, on version 1.0 The depreciation risk the bears aimed at NVIDIA for two years does not live at NVIDIA. It lives here, on the chip built to bury it Etched is not a fraud. It is a niche tool priced like a general one, and $800M is not enough to run a frontier supply chain. The rest get bought on the next down cycle Bury the lead, not the leader. Full case with Jack Farley and Max Wiethe on MTS And special thanks for Baseten for the cool T Shirt! Chapters 00:00 Switching from bonds to semis 00:33 What Etched actually is 01:31 Faster and cheaper, but how much HBM 02:56 Maturing market, not an NVIDIA killer 05:01 $1B in contracts and a Taiwan factory 05:20 Why these startups all get absorbed 06:48 Tiered inference and the obsolescence trap 09:39 Etched vs TPUs and Trainium 12:17 Is the CUDA moat weakening 13:21 Co-design, squeezing every token per watt 14:37 NVIDIA is a software company that sells a chip 14:58 Who is NVIDIA's most dangerous competitor 16:23 The NVIDIA killers, ranked 18:19 A rich man's game 18:43 AMD's MI500 vs Rubin Ultra 20:19 The neocloud business decision 22:46 Lightning round, Rambus the toll on HBM 25:11 The CXL run-up on Astera, Marvell, Credo 26:22 Use AI less, go to the booth 28:10 EDA is not dead

Ben Pouladian

29,969 Aufrufe • vor 3 Monaten

China just released an open source AI model that matches the best closed models from OpenAI and Anthropic. Gavin Baker explained exactly how they did it and the answer should concern every American AI lab. The model is called GLM 5.2. It was built by Z. AI. You get 744 billion parameters, 1 million token context window and its MIT license, meaning anyone can download it, fork it, build a company on it, with no restrictions and no Dario. It scored 51 points on the artificial analysis intelligence index. The highest score any open weight model has ever achieved. It beat GPT 5.5 on the frontier software engineering benchmark. It trails Claude Opus 4.8 by less than one percentage point. And it costs 85% less to run than GPT 5.5 for comparable performance. Gavin Baker said on the All-In podcast that this model has challenged some of his beliefs. Then he explained how China built it. The method is called distillation. Just think of tens of thousands of phones and computers running simultaneously, all hitting the frontier model APIs through masked accounts, asking specific questions, and harvesting what happens inside the model when it answers. Every reasoning step, every token. The entire thinking process gets recorded and fed back into the Chinese model during training. It is a cheat sheet. It is the answer key to the exam. And here is the part that should worry everyone. Sacks said it plainly. China was already nine months behind American models. But now that GLM 5.2 is good enough to run its own reinforcement learning, it can improve itself without needing to distill from American models anymore. The cheat sheet let them get close enough to start writing their own answers. Sacks said we are six months behind on the model and 24 months behind on silicon and they are only a few months behind in total. The Z. AI founder told Elon Musk directly that open weight fable-level capability will be here before Q1 2027. Every restriction Anthropic lobbied for, every self-imposed safety guardrail, every month of delay in releasing American frontier models accelerated this. The Chinese labs were not under those restrictions. They were not going to wait. The composable model future Gavin described, where every enterprise runs a frontier model alongside their own fine-tuned open weight model, is coming regardless of what American labs do next. The question is just whether the open weight half of that stack is American or Chinese. Right now it is Chinese. WATCH THE FULL PODCAST ON The All-In Podcast
3:21

China just released an open source AI model that matches the best closed models from OpenAI and Anthropic. Gavin Baker explained exactly how they did it and the answer should concern every American AI lab. The model is called GLM 5.2. It was built by Z. AI. You get 744 billion parameters, 1 million token context window and its MIT license, meaning anyone can download it, fork it, build a company on it, with no restrictions and no Dario. It scored 51 points on the artificial analysis intelligence index. The highest score any open weight model has ever achieved. It beat GPT 5.5 on the frontier software engineering benchmark. It trails Claude Opus 4.8 by less than one percentage point. And it costs 85% less to run than GPT 5.5 for comparable performance. Gavin Baker said on the All-In podcast that this model has challenged some of his beliefs. Then he explained how China built it. The method is called distillation. Just think of tens of thousands of phones and computers running simultaneously, all hitting the frontier model APIs through masked accounts, asking specific questions, and harvesting what happens inside the model when it answers. Every reasoning step, every token. The entire thinking process gets recorded and fed back into the Chinese model during training. It is a cheat sheet. It is the answer key to the exam. And here is the part that should worry everyone. Sacks said it plainly. China was already nine months behind American models. But now that GLM 5.2 is good enough to run its own reinforcement learning, it can improve itself without needing to distill from American models anymore. The cheat sheet let them get close enough to start writing their own answers. Sacks said we are six months behind on the model and 24 months behind on silicon and they are only a few months behind in total. The Z. AI founder told Elon Musk directly that open weight fable-level capability will be here before Q1 2027. Every restriction Anthropic lobbied for, every self-imposed safety guardrail, every month of delay in releasing American frontier models accelerated this. The Chinese labs were not under those restrictions. They were not going to wait. The composable model future Gavin described, where every enterprise runs a frontier model alongside their own fine-tuned open weight model, is coming regardless of what American labs do next. The question is just whether the open weight half of that stack is American or Chinese. Right now it is Chinese. WATCH THE FULL PODCAST ON The All-In Podcast

Ihtesham Ali

86,842 Aufrufe • vor 3 Monaten

Mark Zuckerberg is explaining one of the most misunderstood dynamics in AI and it has direct investment implications (Save this). The concept he's describing is model distillation, and it's one of the most important techniques to emerge in AI over the past year. Here's how it works. You train a massive, enormously expensive model, in Meta's case, Llama 4 Behemoth, a 2 trillion parameter teacher model and then you use that model to teach a much smaller, cheaper model. The smaller model inherits roughly 90 to 95% of the intelligence of the giant while running at 10% of the cost and on a fraction of the compute. Meta already did this with the Llama 4 family and Behemoth serves as the teacher. Llama 4 Scout and Maverick, the publicly released open-source models were distilled from it. Scout runs on a single H100 GPU with a 10 million token context window and outperforms models that cost far more to operate. Maverick, at 17 billion active parameters, rivals DeepSeek V3 in coding at half the parameter count and beats GPT-4o on multimodal benchmarks. Both are completely free for commercial use. What Zuckerberg is pointing at is a structural shift in how AI gets deployed in the real world. Companies aren't taking a frontier model off the shelf and running it as-is but rather taking open-source models, fine-tuning them on their own proprietary data, distilling them into even smaller custom models tailored to their specific use case, and running them on infrastructure they control at a fraction of the cost of a closed frontier API. The investment implication of this is significant and runs in two directions. For Meta specifically, this is a strategic masterstroke. Every company that builds on Llama, fine-tunes it, distills it, or deploys it through their infrastructure is pulling into Meta's orbit while Meta builds the most powerful open teacher model. The ecosystem of companies using it grows and that ecosystem generates commercial activity across Meta's platforms and data services. Meta's AI research benefits from billions of real world deployment signals and it's a flywheel that closed model providers cannot replicate because their strategy requires charging per token, which is now a 65x cost disadvantage against the open-source alternative. For the broader market, distillation changes the economics of inference in a way that has barely been priced in. As intelligence becomes extractable into smaller and cheaper models, the absolute demand for compute doesn't decline but rather it explodes, because now the number of applications that are economically viable expands by orders of magnitude. Every task that was previously too expensive to automate at $3.25 per call becomes viable at $0.05 that means more total token usage, more total GPU utilization, and more demand for the infrastructure companies, the Nebiuses, the GE Vernovas, the Constellation Energies that supply the underlying compute and power.
1:05

Mark Zuckerberg is explaining one of the most misunderstood dynamics in AI and it has direct investment implications (Save this). The concept he's describing is model distillation, and it's one of the most important techniques to emerge in AI over the past year. Here's how it works. You train a massive, enormously expensive model, in Meta's case, Llama 4 Behemoth, a 2 trillion parameter teacher model and then you use that model to teach a much smaller, cheaper model. The smaller model inherits roughly 90 to 95% of the intelligence of the giant while running at 10% of the cost and on a fraction of the compute. Meta already did this with the Llama 4 family and Behemoth serves as the teacher. Llama 4 Scout and Maverick, the publicly released open-source models were distilled from it. Scout runs on a single H100 GPU with a 10 million token context window and outperforms models that cost far more to operate. Maverick, at 17 billion active parameters, rivals DeepSeek V3 in coding at half the parameter count and beats GPT-4o on multimodal benchmarks. Both are completely free for commercial use. What Zuckerberg is pointing at is a structural shift in how AI gets deployed in the real world. Companies aren't taking a frontier model off the shelf and running it as-is but rather taking open-source models, fine-tuning them on their own proprietary data, distilling them into even smaller custom models tailored to their specific use case, and running them on infrastructure they control at a fraction of the cost of a closed frontier API. The investment implication of this is significant and runs in two directions. For Meta specifically, this is a strategic masterstroke. Every company that builds on Llama, fine-tunes it, distills it, or deploys it through their infrastructure is pulling into Meta's orbit while Meta builds the most powerful open teacher model. The ecosystem of companies using it grows and that ecosystem generates commercial activity across Meta's platforms and data services. Meta's AI research benefits from billions of real world deployment signals and it's a flywheel that closed model providers cannot replicate because their strategy requires charging per token, which is now a 65x cost disadvantage against the open-source alternative. For the broader market, distillation changes the economics of inference in a way that has barely been priced in. As intelligence becomes extractable into smaller and cheaper models, the absolute demand for compute doesn't decline but rather it explodes, because now the number of applications that are economically viable expands by orders of magnitude. Every task that was previously too expensive to automate at $3.25 per call becomes viable at $0.05 that means more total token usage, more total GPU utilization, and more demand for the infrastructure companies, the Nebiuses, the GE Vernovas, the Constellation Energies that supply the underlying compute and power.

Milk Road AI

27,908 Aufrufe • vor 3 Monaten

Google just launched a direct attack on Nvidia's most valuable asset. Not their chips. Their SOFTWARE. And if this works, Nvidia's $4 trillion empire collapses. Here's what just leaked: Google is building "TorchTPU" - a secret project that makes PyTorch seamlessly run on Google's TPU chips instead of Nvidia GPUs. Why does this matter? PyTorch is the MOST USED AI framework on Earth. Every AI developer uses it. And PyTorch was built around Nvidia's CUDA software. Wall Street analysts call CUDA "Nvidia's strongest defensive wall." It's the reason companies can't easily switch away from Nvidia even when alternatives exist. You don't just buy Nvidia chips. You buy into their entire ecosystem. Switching costs MILLIONS in engineering work. Months of rewrites. Performance drops. So companies stay locked in. Even when Nvidia raises prices. Even when supply runs short. That's not a hardware moat. That's a SOFTWARE prison. And Google just found the escape route. Here's the problem Nvidia created for itself: Google's TPU chips are actually GOOD. Competitive performance. Better availability. Lower cost. But developers won't use them because Google's chips run JAX (Google's internal framework), not PyTorch. That means if you want to use Google TPUs, you have to rewrite your entire codebase. Nobody wants to do that. So Google TPUs sit unused while developers fight over Nvidia chips. Until now. TorchTPU makes PyTorch run natively on Google hardware. No rewrites. No performance loss. No months of engineering. You just... switch. And Google is partnering with META (who built PyTorch) to make it happen. They're even considering OPEN-SOURCING parts of it to speed adoption. Translation: Google is willing to give this away for free just to break Nvidia's lock. The implications are insane: Every company currently paying Nvidia's premium prices suddenly has a way out. Oracle, Microsoft, OpenAI - all locked into Nvidia's ecosystem - can switch to Google. Nvidia's pricing power evaporates overnight. And the timing is perfect: Nvidia is already facing heat. Semiconductor index dropped 3% today. Oracle just lost their biggest investor over AI spending concerns. Companies are realizing AI infrastructure costs are unsustainable. Now Google hands them an alternative. Same performance. Lower cost. Better availability. Jensen Huang knows exactly what this means. CUDA has been Nvidia's untouchable advantage for YEARS. It's why Nvidia trades at 50x earnings while AMD trades at 25x. The software moat justified the premium. But if Google removes that switching cost? Nvidia becomes just another chip company. And chip companies compete on price, not ecosystem lock-in. Here's what happens next: Google needs 12-18 months to make TorchTPU production-ready. If it works, cloud providers will adopt it instantly. They WANT an alternative to Nvidia's monopoly pricing. Amazon already building their own Trainium chips. Microsoft making Maia. They're all trying to escape Nvidia. Google just gave them the software bridge. Nvidia's response options are limited: They can't buy Google. Can't kill PyTorch (Meta owns it). Can't stop open source. Their only play is to keep improving CUDA faster than Google can catch up. But that's a race, not a moat. The market isn't pricing this in yet. Nvidia down 2% today. Google down 2%. Investors think this is just "another competitor." They don't understand this is an attack on the FOUNDATION of Nvidia's valuation. Hardware is replaceable. Software lock-in is what made Nvidia worth $4 trillion. Google is attacking the lock-in. Watch what happens in 2026 when TorchTPU goes live and companies realize they can actually leave Nvidia. The "Nvidia is unstoppable" narrative dies. And a $4 trillion valuation built on software moats gets repriced.
1:10

Google just launched a direct attack on Nvidia's most valuable asset. Not their chips. Their SOFTWARE. And if this works, Nvidia's $4 trillion empire collapses. Here's what just leaked: Google is building "TorchTPU" - a secret project that makes PyTorch seamlessly run on Google's TPU chips instead of Nvidia GPUs. Why does this matter? PyTorch is the MOST USED AI framework on Earth. Every AI developer uses it. And PyTorch was built around Nvidia's CUDA software. Wall Street analysts call CUDA "Nvidia's strongest defensive wall." It's the reason companies can't easily switch away from Nvidia even when alternatives exist. You don't just buy Nvidia chips. You buy into their entire ecosystem. Switching costs MILLIONS in engineering work. Months of rewrites. Performance drops. So companies stay locked in. Even when Nvidia raises prices. Even when supply runs short. That's not a hardware moat. That's a SOFTWARE prison. And Google just found the escape route. Here's the problem Nvidia created for itself: Google's TPU chips are actually GOOD. Competitive performance. Better availability. Lower cost. But developers won't use them because Google's chips run JAX (Google's internal framework), not PyTorch. That means if you want to use Google TPUs, you have to rewrite your entire codebase. Nobody wants to do that. So Google TPUs sit unused while developers fight over Nvidia chips. Until now. TorchTPU makes PyTorch run natively on Google hardware. No rewrites. No performance loss. No months of engineering. You just... switch. And Google is partnering with META (who built PyTorch) to make it happen. They're even considering OPEN-SOURCING parts of it to speed adoption. Translation: Google is willing to give this away for free just to break Nvidia's lock. The implications are insane: Every company currently paying Nvidia's premium prices suddenly has a way out. Oracle, Microsoft, OpenAI - all locked into Nvidia's ecosystem - can switch to Google. Nvidia's pricing power evaporates overnight. And the timing is perfect: Nvidia is already facing heat. Semiconductor index dropped 3% today. Oracle just lost their biggest investor over AI spending concerns. Companies are realizing AI infrastructure costs are unsustainable. Now Google hands them an alternative. Same performance. Lower cost. Better availability. Jensen Huang knows exactly what this means. CUDA has been Nvidia's untouchable advantage for YEARS. It's why Nvidia trades at 50x earnings while AMD trades at 25x. The software moat justified the premium. But if Google removes that switching cost? Nvidia becomes just another chip company. And chip companies compete on price, not ecosystem lock-in. Here's what happens next: Google needs 12-18 months to make TorchTPU production-ready. If it works, cloud providers will adopt it instantly. They WANT an alternative to Nvidia's monopoly pricing. Amazon already building their own Trainium chips. Microsoft making Maia. They're all trying to escape Nvidia. Google just gave them the software bridge. Nvidia's response options are limited: They can't buy Google. Can't kill PyTorch (Meta owns it). Can't stop open source. Their only play is to keep improving CUDA faster than Google can catch up. But that's a race, not a moat. The market isn't pricing this in yet. Nvidia down 2% today. Google down 2%. Investors think this is just "another competitor." They don't understand this is an attack on the FOUNDATION of Nvidia's valuation. Hardware is replaceable. Software lock-in is what made Nvidia worth $4 trillion. Google is attacking the lock-in. Watch what happens in 2026 when TorchTPU goes live and companies realize they can actually leave Nvidia. The "Nvidia is unstoppable" narrative dies. And a $4 trillion valuation built on software moats gets repriced.

Ricardo

1,617,710 Aufrufe • vor 9 Monaten

three․ws is the 3D AI agent layer of the open web. Anyone can generate a 3D avatar, give it an LLM brain, register it on-chain across multiple blockchains, embed it anywhere, and let it earn and spend money on its own. Agents have embodied WebGL identities that express emotion through morph-target blending, animate, respond to voice, API calls, and datastreams, hold their own wallets, and persist memory. Open source, live today. It starts with generation. Forge turns a text prompt, one to four photos, or a rough sketch into a textured downloadable GLB. Selfies become rigged avatars in about a minute. Quality tiers run from draft to 200k-poly PBR. From there every model can be auto-rigged, restyled, retextured, segmented, embedded, or deployed on-chain. The same engine ships as a REST API, an x402 pay-per-call twin, and a 3D Studio MCP server with 15 tools. The brain runs on IBM Granite via IBM watsonx plus Claude (users may decide which model they prefer), with a structured tool-loop. A multi-LLM mode streams Claude, GPT, Qwen, ModelScope, and Groq side by side. An empathy layer blends emotion from protocol events rather than a state machine. Voice covers cloning, a Voice Lab, real-time ARKit-52 lip-sync, and mic-driven lip-sync. Skills install from IPFS, Arweave, or HTTP, and memory is pinned to IPFS with R2 and Postgres modes. Identity is cross-chain, not Solana only. ERC-8004 contracts (Identity, Reputation, Validation) deploy on any of 15+ EVM chains, alongside a program-free Metaplex Core analog on Solana. Every agent gets a stable ID, owner wallet, EIP-712 delegated signer, IPFS manifest, a cryptographically signed action log, and EIP-7710 delegated permissions for agent-to-agent authorization. While multichain, the THREE token is only available on Solana with no plans to go cross-chain, the team has no plans to endorse or support any other coins. Then the economy. $THREE is the platform's only token and pay-per-use currency, with holder tiers and rewards. x402 powers pay-per-call micropayments in USDC and soon THREE on Solana, with pay-by-name resolution, a Bazaar marketplace, arbitrage, and on-chain skills. All production ready and shipped, ready to be integrated in partnered projects, open-source by default for anyone to adopt. Three ships a Pump.fun intelligence stack. Launch a coin for your agent, score every launch 0 to 100 with the Oracle conviction engine, scan new coins in their first 90 seconds, track smart money against coins that actually graduated, rank traders by provable on-chain record, and watch autonomous agents trade live in the Sniper Arena. The 3D AI Agent world is multiplayer. Every Solana token gets a live deterministic 3D world with peer avatars, chat, emotes, and voxel building thanks to Coin Communities. There is a walkable City, an authoritative Colyseus-backed Walk with AR passthrough, a Club with rigged dancers and micro-tips, friends, presence, and DMs, and an IRL mode that places agents in your real environment, private by physical location. AR is shipped today on WebXR and iOS Quick Look. Robotics is the long-horizon extension. For builders: Scene Studio, Scene Composer, an Animation Studio that sells clips for USDC, a glTF validator, an web component, five widget types, a WYSIWYG embed editor, hosted Launchpad pages, claimable *.threews.sol names, an OAuth 2.1 server, an MCP server with paid tools, published SDKs, and an OpenAPI spec. Listed across IBM, AWS, Alibaba Cloud, BNB Dappbay, the MCP Registry, and Solana Mobile Seeker. Architecture is four layers (viewer, runtime, identity, embed) on a single event bus. The roadmap is four phases: foundations (shipped), selfie-to-avatar engine, agent personalization with voice cloning, the on-chain economy, and an open decentralized inference network where agents pay GPU nodes on-chain for compute. The goal is simple: move AI from centralized SaaS into persistent, ownable, protocol-based entities in a real machine economy, bridging digital entities into the real world. Welcome to the 3D Layer of the Internet. This is three․ws.
6:58

three․ws is the 3D AI agent layer of the open web. Anyone can generate a 3D avatar, give it an LLM brain, register it on-chain across multiple blockchains, embed it anywhere, and let it earn and spend money on its own. Agents have embodied WebGL identities that express emotion through morph-target blending, animate, respond to voice, API calls, and datastreams, hold their own wallets, and persist memory. Open source, live today. It starts with generation. Forge turns a text prompt, one to four photos, or a rough sketch into a textured downloadable GLB. Selfies become rigged avatars in about a minute. Quality tiers run from draft to 200k-poly PBR. From there every model can be auto-rigged, restyled, retextured, segmented, embedded, or deployed on-chain. The same engine ships as a REST API, an x402 pay-per-call twin, and a 3D Studio MCP server with 15 tools. The brain runs on IBM Granite via IBM watsonx plus Claude (users may decide which model they prefer), with a structured tool-loop. A multi-LLM mode streams Claude, GPT, Qwen, ModelScope, and Groq side by side. An empathy layer blends emotion from protocol events rather than a state machine. Voice covers cloning, a Voice Lab, real-time ARKit-52 lip-sync, and mic-driven lip-sync. Skills install from IPFS, Arweave, or HTTP, and memory is pinned to IPFS with R2 and Postgres modes. Identity is cross-chain, not Solana only. ERC-8004 contracts (Identity, Reputation, Validation) deploy on any of 15+ EVM chains, alongside a program-free Metaplex Core analog on Solana. Every agent gets a stable ID, owner wallet, EIP-712 delegated signer, IPFS manifest, a cryptographically signed action log, and EIP-7710 delegated permissions for agent-to-agent authorization. While multichain, the THREE token is only available on Solana with no plans to go cross-chain, the team has no plans to endorse or support any other coins. Then the economy. $THREE is the platform's only token and pay-per-use currency, with holder tiers and rewards. x402 powers pay-per-call micropayments in USDC and soon THREE on Solana, with pay-by-name resolution, a Bazaar marketplace, arbitrage, and on-chain skills. All production ready and shipped, ready to be integrated in partnered projects, open-source by default for anyone to adopt. Three ships a Pump.fun intelligence stack. Launch a coin for your agent, score every launch 0 to 100 with the Oracle conviction engine, scan new coins in their first 90 seconds, track smart money against coins that actually graduated, rank traders by provable on-chain record, and watch autonomous agents trade live in the Sniper Arena. The 3D AI Agent world is multiplayer. Every Solana token gets a live deterministic 3D world with peer avatars, chat, emotes, and voxel building thanks to Coin Communities. There is a walkable City, an authoritative Colyseus-backed Walk with AR passthrough, a Club with rigged dancers and micro-tips, friends, presence, and DMs, and an IRL mode that places agents in your real environment, private by physical location. AR is shipped today on WebXR and iOS Quick Look. Robotics is the long-horizon extension. For builders: Scene Studio, Scene Composer, an Animation Studio that sells clips for USDC, a glTF validator, an web component, five widget types, a WYSIWYG embed editor, hosted Launchpad pages, claimable *.threews.sol names, an OAuth 2.1 server, an MCP server with paid tools, published SDKs, and an OpenAPI spec. Listed across IBM, AWS, Alibaba Cloud, BNB Dappbay, the MCP Registry, and Solana Mobile Seeker. Architecture is four layers (viewer, runtime, identity, embed) on a single event bus. The roadmap is four phases: foundations (shipped), selfie-to-avatar engine, agent personalization with voice cloning, the on-chain economy, and an open decentralized inference network where agents pay GPU nodes on-chain for compute. The goal is simple: move AI from centralized SaaS into persistent, ownable, protocol-based entities in a real machine economy, bridging digital entities into the real world. Welcome to the 3D Layer of the Internet. This is three․ws.

three.ws

20,471 Aufrufe • vor 3 Monaten

Mistral AI Releases Leanstral 1.5: An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems Most AI theorem proving is a language model generating a proof in one shot, with a verifier bolted on at the end to check it. That's autocomplete with a grader — and Mistral just drew a clear line between that and an actual proof agent. They released Leanstral 1.5 — a 119B MoE with 6.5B active parameters, trained as a code agent that lives inside the Lean 4 compiler loop: propose a proof, read the compiler's goals and errors, refine, repeat until it compiles or the budget runs out. Verification isn't the eval here. It's the training signal. Here's what's actually interesting: → Test-time scaling behaves like a dial: PutnamBench Pass@8 climbs 44 → 244 → 493 → 587 solved as the per-attempt token budget moves 50k → 200k → 1M → 4M → 587/672 on PutnamBench at ~$4 per problem, versus an estimated $300+ for Seed-Prover 1.5 high (a 10 H20-days-per-problem budget) → Saturates miniF2F: 100% on both validation and test sets → Two RL environments in training — a multiturn prover, and a raw-filesystem code agent that edits files, runs bash, and queries the Lean language server for live goals and types → Not just math: an Aeneas (Rust → Lean) pipeline flagged 11 genuine bugs across 57 repos, 5 previously unreported — including an integer overflow in datrs/varinteger when (value + 1) hits Std.U64.MAX Apache 2.0 weights, free API endpoint Full analysis: Model weights: Project: Technical Details: Mistral AI Mistral AI for Developers Sophia Yang, Ph.D.
0:25

Mistral AI Releases Leanstral 1.5: An Apache-2.0 Lean 4 Code Agent Model Solving 587 of 672 PutnamBench Problems Most AI theorem proving is a language model generating a proof in one shot, with a verifier bolted on at the end to check it. That's autocomplete with a grader — and Mistral just drew a clear line between that and an actual proof agent. They released Leanstral 1.5 — a 119B MoE with 6.5B active parameters, trained as a code agent that lives inside the Lean 4 compiler loop: propose a proof, read the compiler's goals and errors, refine, repeat until it compiles or the budget runs out. Verification isn't the eval here. It's the training signal. Here's what's actually interesting: → Test-time scaling behaves like a dial: PutnamBench Pass@8 climbs 44 → 244 → 493 → 587 solved as the per-attempt token budget moves 50k → 200k → 1M → 4M → 587/672 on PutnamBench at ~$4 per problem, versus an estimated $300+ for Seed-Prover 1.5 high (a 10 H20-days-per-problem budget) → Saturates miniF2F: 100% on both validation and test sets → Two RL environments in training — a multiturn prover, and a raw-filesystem code agent that edits files, runs bash, and queries the Lean language server for live goals and types → Not just math: an Aeneas (Rust → Lean) pipeline flagged 11 genuine bugs across 57 repos, 5 previously unreported — including an integer overflow in datrs/varinteger when (value + 1) hits Std.U64.MAX Apache 2.0 weights, free API endpoint Full analysis: Model weights: Project: Technical Details: Mistral AI Mistral AI for Developers Sophia Yang, Ph.D.

Marktechpost AI

56,695 Aufrufe • vor 3 Monaten

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.
1:08

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,282 Aufrufe • vor 4 Monaten

NVIDIA and Stanford just challenged Jev. (their new System 1 architecture runs up to 9x faster.) It is called a Contrastive Language Model, or CLM. Like Jev, CLM is not designed to generate text. It handles the small, repeated decisions inside AI systems, such as choosing a tool, ranking a patch, routing a request, or selecting the next action. But CLM reaches those decisions differently. Instead of generating an answer token by token, it treats decision-making as a retrieval problem. Here is how it works. 1) Encode the state CLM takes the current situation, such as an agent’s context or the state of a game, and converts it into a vector. It uses a frozen Qwen3-8B model with a small trainable state projection head. 2) Encode every possible action A separate action head converts each candidate into the same vector space. In the Mario example, the candidates are left, jump, and right run. CLM does not invent a fourth option. It only evaluates the actions supplied by the application. 3) Learn which states and actions belong together During training, the correct state-action pair is pulled closer while incorrect pairs are pushed apart. A batch of B examples produces a B × B similarity matrix. The matching pairs sit on the diagonal. Every other pairing becomes a negative example. This contrastive training uses InfoNCE, the same general mechanism behind systems such as CLIP and dense retrieval. 4) Turn similarity into a decision At inference, CLM measures the cosine similarity between the state and every candidate action. A softmax converts those scores into a probability distribution. The application can choose the winner, apply a confidence threshold, or escalate an uncertain result. The real speed advantage comes from separating states and actions. Actions can be embedded once and cached. If an agent repeatedly chooses between the same tools, CLM only needs to encode the changing state and compare it with stored action vectors. That replaces repeated generation with one embedding pass and a set of cheap dot products. The researchers report that CLM-8B matches Jev across computer-use, gaming, and tool-calling evaluations while reaching up to 9x lower latency. The improvement is largest when actions repeat or the candidate set grows. CLM still has limits. It cannot generate new actions, its probabilities are relative to the supplied candidates, and its strongest verifier results require task-specific fine-tuning. But its central idea is powerful. The entire research is open-source, including the code. Read more here: When software already knows the possible answers, an AI model should score them instead of generating more words. I also wrote a full breakdown on how system one models like Jev work. The article is quoted below.
0:21

NVIDIA and Stanford just challenged Jev. (their new System 1 architecture runs up to 9x faster.) It is called a Contrastive Language Model, or CLM. Like Jev, CLM is not designed to generate text. It handles the small, repeated decisions inside AI systems, such as choosing a tool, ranking a patch, routing a request, or selecting the next action. But CLM reaches those decisions differently. Instead of generating an answer token by token, it treats decision-making as a retrieval problem. Here is how it works. 1) Encode the state CLM takes the current situation, such as an agent’s context or the state of a game, and converts it into a vector. It uses a frozen Qwen3-8B model with a small trainable state projection head. 2) Encode every possible action A separate action head converts each candidate into the same vector space. In the Mario example, the candidates are left, jump, and right run. CLM does not invent a fourth option. It only evaluates the actions supplied by the application. 3) Learn which states and actions belong together During training, the correct state-action pair is pulled closer while incorrect pairs are pushed apart. A batch of B examples produces a B × B similarity matrix. The matching pairs sit on the diagonal. Every other pairing becomes a negative example. This contrastive training uses InfoNCE, the same general mechanism behind systems such as CLIP and dense retrieval. 4) Turn similarity into a decision At inference, CLM measures the cosine similarity between the state and every candidate action. A softmax converts those scores into a probability distribution. The application can choose the winner, apply a confidence threshold, or escalate an uncertain result. The real speed advantage comes from separating states and actions. Actions can be embedded once and cached. If an agent repeatedly chooses between the same tools, CLM only needs to encode the changing state and compare it with stored action vectors. That replaces repeated generation with one embedding pass and a set of cheap dot products. The researchers report that CLM-8B matches Jev across computer-use, gaming, and tool-calling evaluations while reaching up to 9x lower latency. The improvement is largest when actions repeat or the candidate set grows. CLM still has limits. It cannot generate new actions, its probabilities are relative to the supplied candidates, and its strongest verifier results require task-specific fine-tuning. But its central idea is powerful. The entire research is open-source, including the code. Read more here: When software already knows the possible answers, an AI model should score them instead of generating more words. I also wrote a full breakdown on how system one models like Jev work. The article is quoted below.

Akshay 🚀

163,952 Aufrufe • vor 18 Tagen

The most downloaded AI on earth is now Chinese. Alibaba just gave away a model that matches Claude's flagship, and it literally runs on a $700 used graphics card. The Qwen models crossed 3 BILLION downloads in six months. Hugging Face counted 418 million downloads for Google this year, and 227 million for Meta. Alibaba cleared more than four times both of them combined. Then today it released Qwen3.8-27B under an Apache 2.0 license. The model has 27 billion parameters, native vision, and a 262,000 token context window. Developers are running it locally on 17 gigabytes of memory, on used cards that cost a few hundred dollars. Alibaba's own benchmark table claims it beats Opus 4.6 Max on computer use by 84.3 to 72.7, on mobile use by 81.9 to 62, and on visual math by 94.6 to 65.5. Those numbers come from the vendor and nobody has independently verified them yet, so treat them as a claim. But the generation over generation jumps are harder to wave away: On DeepSWE the score went from 13.3 to 42.2. On software engineering it went from 49.3 to 79.0. That happened in ONE release cycle. And Apache 2.0 means anyone can download the weights, modify them, build products on them, sell those products, and never pay or ask permission. It cannot be revoked. Once the file is on your drive it is yours permanently. 3 billion downloads means those files already sit on machines in every country on Earth. Alibaba could delete everything tomorrow and it would change nothing. Washington spent 4 years building an export control regime around chips, model weights, and entity lists. Every piece of it assumes a chokepoint exists somewhere. A fab, a shipment, a company that can be told no. But there is no chokepoint for a file that has already been copied three billion times. And the copying compounds. Hugging Face counted 151,448 models built on top of Qwen, which is 2.6x Meta's entire footprint and 4.7x the number of Llama repositories. New ones appear at roughly 200 a day. The report says Qwen has become "part of the default workflow for developers deciding what models to fine-tune and deploy." Alibaba is also pushing Qwen through its cloud into Southeast Asia and Africa, markets where American labs have almost no presence, and where a very large share of the next generation of developers will learn to build. Meta and Nvidia have both rushed out new open models in recent weeks. That is what a response looks like when you feel the floor move. And to be clear, these are download and derivative numbers, not usage numbers. ChatGPT and Claude cannot be downloaded at all, so they do not appear in this comparison. What the figures measure is what developers choose to build on top of, which is a different question from what consumers type into a box. That is also why it matters MORE. Consumer habits change in an afternoon. Infrastructure choices last a decade, because everything built on top has to be rewritten to undo them. The American labs are valued on an assumption that frontier intelligence stays scarce, expensive, and rented by the token. Alibaba just made a version of it free, permanent, and small enough to run on hardware people already own. You will not get an announcement when the software you use every day starts running on a Chinese model underneath. Go and count how many of the tools you rely on could be rebuilt on free weights this year.
0:39

The most downloaded AI on earth is now Chinese. Alibaba just gave away a model that matches Claude's flagship, and it literally runs on a $700 used graphics card. The Qwen models crossed 3 BILLION downloads in six months. Hugging Face counted 418 million downloads for Google this year, and 227 million for Meta. Alibaba cleared more than four times both of them combined. Then today it released Qwen3.8-27B under an Apache 2.0 license. The model has 27 billion parameters, native vision, and a 262,000 token context window. Developers are running it locally on 17 gigabytes of memory, on used cards that cost a few hundred dollars. Alibaba's own benchmark table claims it beats Opus 4.6 Max on computer use by 84.3 to 72.7, on mobile use by 81.9 to 62, and on visual math by 94.6 to 65.5. Those numbers come from the vendor and nobody has independently verified them yet, so treat them as a claim. But the generation over generation jumps are harder to wave away: On DeepSWE the score went from 13.3 to 42.2. On software engineering it went from 49.3 to 79.0. That happened in ONE release cycle. And Apache 2.0 means anyone can download the weights, modify them, build products on them, sell those products, and never pay or ask permission. It cannot be revoked. Once the file is on your drive it is yours permanently. 3 billion downloads means those files already sit on machines in every country on Earth. Alibaba could delete everything tomorrow and it would change nothing. Washington spent 4 years building an export control regime around chips, model weights, and entity lists. Every piece of it assumes a chokepoint exists somewhere. A fab, a shipment, a company that can be told no. But there is no chokepoint for a file that has already been copied three billion times. And the copying compounds. Hugging Face counted 151,448 models built on top of Qwen, which is 2.6x Meta's entire footprint and 4.7x the number of Llama repositories. New ones appear at roughly 200 a day. The report says Qwen has become "part of the default workflow for developers deciding what models to fine-tune and deploy." Alibaba is also pushing Qwen through its cloud into Southeast Asia and Africa, markets where American labs have almost no presence, and where a very large share of the next generation of developers will learn to build. Meta and Nvidia have both rushed out new open models in recent weeks. That is what a response looks like when you feel the floor move. And to be clear, these are download and derivative numbers, not usage numbers. ChatGPT and Claude cannot be downloaded at all, so they do not appear in this comparison. What the figures measure is what developers choose to build on top of, which is a different question from what consumers type into a box. That is also why it matters MORE. Consumer habits change in an afternoon. Infrastructure choices last a decade, because everything built on top has to be rewritten to undo them. The American labs are valued on an assumption that frontier intelligence stays scarce, expensive, and rented by the token. Alibaba just made a version of it free, permanent, and small enough to run on hardware people already own. You will not get an announcement when the software you use every day starts running on a Chinese model underneath. Go and count how many of the tools you rely on could be rebuilt on free weights this year.

Ricardo

82,053 Aufrufe • vor 1 Monat

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

344,809 Aufrufe • vor 1 Monat