正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

another 5 days later, zml/llmd runs fully on Metal, serving 8 simultaneous requests at full bf16 zml/llmd is our LLM serving software that all the modern niceties (continuous batching etc..) ps: "full bf16", what a time to be alive

Steeve Morin

9,490 subscribers

27,579 次观看 • 4 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 条评论

RetroRanger Anonymous 的头像

RetroRanger Anonymous4 个月前

what a time to be alive indeed

Vito Botta 的头像

Vito Botta4 个月前

Full bf16 on Metal handling 8 concurrent requests is impressive. What's the memory bandwidth like under load?

Julien Jerphanion 的头像

Julien Jerphanion4 个月前

Congrats! 🎉

Thomas Huber 的头像

Thomas Huber4 个月前

awsome.

Austin 的头像

Austin4 个月前

What is this @grok

Gregor 的头像

Gregor4 个月前

ran into this with mlx on my m2 kept getting silent f32 fallbacks while thinking i had bf16, took like 5hrs to even realize. did you hit precision drift issues during the metal port or did zml surface those cleanly?

相关视频

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Laguna S 2.1 is built to run locally. We heard the ask for a serious open model in the 80–120B range. Laguna S 2.1 is our answer, and we want it to redefine what this size class can do. Weights ship in BF16, FP8, INT4 and NVFP4, with official GGUF and MLX conversions, plus official DFlash draft models for faster inference. Day-one support across llama.cpp, ollama, Valerie Lee Make Up, SGLang, NVIDIA TRT-LLM, Hugging Face transformers and ZML. Watch it run on an NVIDIA AI DGX Spark.

Poolside

31,242 次观看 • 2 个月前

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 次观看 • 2 年前

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,349 次观看 • 6 个月前

Yesterday, I asked the Government whether grooming gang perpetrators would serve their full sentences. The minister seemed to say that we should be grateful that these people are serving prison sentences *at all*. What on earth?!

Yesterday, I asked the Government whether grooming gang perpetrators would serve their full sentences. The minister seemed to say that we should be grateful that these people are serving prison sentences at all. What on earth?!

Katie Lam

85,008 次观看 • 3 个月前

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Continuous batching in LLMs, clearly explained: (a popular LLM interview question; bookmark this) In traditional ML inference, a batch is a matrix. Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment. LLM decoding does not work that way. One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token. Under static batching, membership is fixed when the batch starts. A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400. The GPU keeps paying the full weight read for a batch that is mostly empty. Loading model weights out of HBM costs the same whether four slots are producing tokens or one. Continuous batching moves the decision boundary. Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again. A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done. Anyscale benchmarked both OPT-13B on a single A100. With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving. Variance drives the entire gap. Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst. None of this alters the model. vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA ships the same mechanism under the name in-flight batching. The animation below runs both policies on the same 16 requests and the same 4 slots, stepping in lockstep. The only difference is when a new request is allowed in. To dive deeper into continuous batching specifically, I wrote a full breakdown of the scheduler underneath it. It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation. Read it below.

Avi Chawla

16,140 次观看 • 2 个月前

It's time you all know exactly what we're building here at Opacity. We've been hard at work building a visual-first, full-stack software development platform, that gives you a team of autonomous AI agents to build and grow your software products and businesses. The future of software development is here. 😏

It's time you all know exactly what we're building here at Opacity. We've been hard at work building a visual-first, full-stack software development platform, that gives you a team of autonomous AI agents to build and grow your software products and businesses. The future of software development is here. 😏

Zoah (formerly Opacity)

16,275 次观看 • 2 个月前

Today I celebrated Thanksgiving by serving meals to Coloradans in need at the Denver Rescue Mission. Thanksgiving is a time for good food, friends, and family, and I encourage all Coloradans to reflect on what they are thankful for and lend a helping hand to our fellow Coloradans.

Today I celebrated Thanksgiving by serving meals to Coloradans in need at the Denver Rescue Mission. Thanksgiving is a time for good food, friends, and family, and I encourage all Coloradans to reflect on what they are thankful for and lend a helping hand to our fellow Coloradans.

Governor Jared Polis

15,808 次观看 • 1 年前

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

Eyal Toledano

10,743 次观看 • 1 个月前

✏️What skills are most important when serving as a manuscript peer reviewer? Our editors weigh in 👉 This is the final topic of the 2nd season of HemaSphere’s ‘Meet the Editors’ series, offering short, casual anecdotes from our experts. We hope you have enjoyed meeting our team! The full Meet the Editors seasons 1 and 2 on our YouTube channel and can be rewatched in full at any time. Our YouTube channel link is in the comments. #HemaSphereHumans #hematology #HemeTwitter European Hematology Association Jan Cools David Kent @AdeleKFielding Claire Harrison

✏️What skills are most important when serving as a manuscript peer reviewer? Our editors weigh in 👉 This is the final topic of the 2nd season of HemaSphere’s ‘Meet the Editors’ series, offering short, casual anecdotes from our experts. We hope you have enjoyed meeting our team! The full Meet the Editors seasons 1 and 2 on our YouTube channel and can be rewatched in full at any time. Our YouTube channel link is in the comments. #HemaSphereHumans #hematology #HemeTwitter European Hematology Association Jan Cools David Kent @AdeleKFielding Claire Harrison

HemaSphere Journal (IF = 14.6)

4,866,871 次观看 • 2 年前

Let’s check in and see how the City of Medicine Hat is doing, shall we? Counsellor Bill Cocks, currently serving his 5th term on City Council (and hopefully getting help for his TDS), suggests at the Sept. 8 meeting that it may be time to start wearing black arm bands. “We are a nation at war.” This is quite something to watch.

Let’s check in and see how the City of Medicine Hat is doing, shall we? Counsellor Bill Cocks, currently serving his 5th term on City Council (and hopefully getting help for his TDS), suggests at the Sept. 8 meeting that it may be time to start wearing black arm bands. “We are a nation at war.” This is quite something to watch.

Tamara Lich 🇨🇦

108,923 次观看 • 1 个月前

Here's where feminism lied to you! It told you that serving is slavery; that your role as a wife, being a helper to your husband: that's bondage. Since when did we take on the demonic idea that serving is slavery. SERVING IS CHRISTLIKE. "For even the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many" Mark 10:45 Paul introduced himself as a "doulos" AKA a SERVANT of God. Husbands and wives are both called to serve, but our serving looks DIFFERENT. The husband serves us by providing and protecting. They serve by carrying the ultimate responsibility of making sure there's food on the table every day. They serve us by protecting us from evil - both physical evil and evil ideas. They protect us against false ideas. They are responsible for the spiritual condition of the home. Wives are called to serve by honoring his leadership, showing gratitude, creating a house full of peace of warmth, putting food in his belly and raising children. One way of serving is not inferior to the other. Let's JOYFULLY serve our husbands. SERVING ISN'T SLAVERY... SERVING IS A PRIVILEGE. LET'S REPLACE COMPLAINING WITH GRATITUDE. IT'S AN HONOR TO SERVE. IT'S AN HONOR TO BE MY HUSBAND'S HELPER. WHEN I SERVE MY HUSBAND, I HONOR GOD.

Here's where feminism lied to you! It told you that serving is slavery; that your role as a wife, being a helper to your husband: that's bondage. Since when did we take on the demonic idea that serving is slavery. SERVING IS CHRISTLIKE. "For even the Son of Man did not come to be served, but to serve, and to give his life as a ransom for many" Mark 10:45 Paul introduced himself as a "doulos" AKA a SERVANT of God. Husbands and wives are both called to serve, but our serving looks DIFFERENT. The husband serves us by providing and protecting. They serve by carrying the ultimate responsibility of making sure there's food on the table every day. They serve us by protecting us from evil - both physical evil and evil ideas. They protect us against false ideas. They are responsible for the spiritual condition of the home. Wives are called to serve by honoring his leadership, showing gratitude, creating a house full of peace of warmth, putting food in his belly and raising children. One way of serving is not inferior to the other. Let's JOYFULLY serve our husbands. SERVING ISN'T SLAVERY... SERVING IS A PRIVILEGE. LET'S REPLACE COMPLAINING WITH GRATITUDE. IT'S AN HONOR TO SERVE. IT'S AN HONOR TO BE MY HUSBAND'S HELPER. WHEN I SERVE MY HUSBAND, I HONOR GOD.

Millicent Sedra

61,955 次观看 • 9 个月前

“We recently deployed our first AIP workflow, bringing us from analog to AI in 18 months. Now, every single encounter that we do is documented digitally, and new employees to the department will never see a paper chart.” With Palantir, the U.S. Department of State can reduce the time it takes to clear candidates to the Foreign Service from 60 days down to 12. The time saved allows a Western standard of care to reach all patients, regardless of what corner of the world they’re serving our country. At AIPCon 7, Alan Lewis, IT Director at the Bureau of Medical Services, U.S. Department of State, describes how Palantir helps the U.S. save lives. Watch full keynote here:

“We recently deployed our first AIP workflow, bringing us from analog to AI in 18 months. Now, every single encounter that we do is documented digitally, and new employees to the department will never see a paper chart.” With Palantir, the U.S. Department of State can reduce the time it takes to clear candidates to the Foreign Service from 60 days down to 12. The time saved allows a Western standard of care to reach all patients, regardless of what corner of the world they’re serving our country. At AIPCon 7, Alan Lewis, IT Director at the Bureau of Medical Services, U.S. Department of State, describes how Palantir helps the U.S. save lives. Watch full keynote here:

Palantir

55,336 次观看 • 1 年前

North Carolina HC Scott Forbes on what this Tar Heels team means to him. “I couldn't be prouder of a group. I just talked to them in the locker room. We talk all the time about service, about serving one another. I don't tell them this, but I write it down every day for myself, in Galatians 5:13: True service, serving one another through love, humbly in love and how powerful love is. This group loved each other all season. They took us on a ride that was unbelievable. We came up a little bit short. But I would take that ride again every day of the year.” “The thing I'm going to miss the most is just being with them. That's what I'm going to miss the most. Obviously, it gets emotional back there. But I'm thankful, too. I'm thankful that I've been given the opportunity. It's an honor and a privilege to be the head coach at University of North Carolina. I don't take that lightly.”

North Carolina HC Scott Forbes on what this Tar Heels team means to him. “I couldn't be prouder of a group. I just talked to them in the locker room. We talk all the time about service, about serving one another. I don't tell them this, but I write it down every day for myself, in Galatians 5:13: True service, serving one another through love, humbly in love and how powerful love is. This group loved each other all season. They took us on a ride that was unbelievable. We came up a little bit short. But I would take that ride again every day of the year.” “The thing I'm going to miss the most is just being with them. That's what I'm going to miss the most. Obviously, it gets emotional back there. But I'm thankful, too. I'm thankful that I've been given the opportunity. It's an honor and a privilege to be the head coach at University of North Carolina. I don't take that lightly.”

Harrison Cordell Fant

27,157 次观看 • 3 个月前

What a time to be alive! Hard to believe with everything that's gone on there was a sleaze probe vote at the start of the week. That was before we even knew about our Ang getting trollied. Luckily for the PM, all his favourite Us Labours came through for him:

What a time to be alive! Hard to believe with everything that's gone on there was a sleaze probe vote at the start of the week. That was before we even knew about our Ang getting trollied. Luckily for the PM, all his favourite Us Labours came through for him:

Intel Lady

22,540 次观看 • 5 个月前

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

35,910 次观看 • 1 个月前

NVIDIA AI Releases Nemotron 3 Embed: An Open Embedding Collection Whose 8B Checkpoint Ranks #1 on RTEB Most RAG stacks treat the embedding model as a commodity — pick one, index, move on. Nemotron 3 Embed is NVIDIA's argument that the retrieval layer is where agent cost actually gets set. They released three open checkpoints — Nemotron-3-Embed-8B-BF16, 1B-BF16, and 1B-NVFP4 — built on Ministral bases, trained with bidirectional attention masking, pooled by averaging token-level representations, all taking 32,768-token inputs under OpenMDW-1.1. Here's what's actually interesting: → The 8B ranks #1 overall on RTEB: 78.46 avg NDCG@10, alongside 75.45 on MMTEB Retrieval and 60.60 on ViDoRe-V3 text → The 1B wasn't trained small. It was pruned from a 3B parent with ModelOpt mcore_minitron NAS, then distilled from the 8B teacher on COS + MSE loss — twice → That pipeline lands the 1.14B checkpoint at 72.38 RTEB, up 10.4 points on the prior-generation llama-nemotron-embed-vl-1b-v2 → NVFP4 costs 0.38 RTEB points (72.00 vs 72.38, ~99.5% retention) and buys up to 2x BF16 throughput on Blackwell Full Analysis: Model weight: Technical details: NVIDIA AI NVIDIA

NVIDIA AI Releases Nemotron 3 Embed: An Open Embedding Collection Whose 8B Checkpoint Ranks #1 on RTEB Most RAG stacks treat the embedding model as a commodity — pick one, index, move on. Nemotron 3 Embed is NVIDIA's argument that the retrieval layer is where agent cost actually gets set. They released three open checkpoints — Nemotron-3-Embed-8B-BF16, 1B-BF16, and 1B-NVFP4 — built on Ministral bases, trained with bidirectional attention masking, pooled by averaging token-level representations, all taking 32,768-token inputs under OpenMDW-1.1. Here's what's actually interesting: → The 8B ranks #1 overall on RTEB: 78.46 avg NDCG@10, alongside 75.45 on MMTEB Retrieval and 60.60 on ViDoRe-V3 text → The 1B wasn't trained small. It was pruned from a 3B parent with ModelOpt mcore_minitron NAS, then distilled from the 8B teacher on COS + MSE loss — twice → That pipeline lands the 1.14B checkpoint at 72.38 RTEB, up 10.4 points on the prior-generation llama-nemotron-embed-vl-1b-v2 → NVFP4 costs 0.38 RTEB points (72.00 vs 72.38, ~99.5% retention) and buys up to 2x BF16 throughput on Blackwell Full Analysis: Model weight: Technical details: NVIDIA AI NVIDIA

Marktechpost AI

32,352 次观看 • 2 个月前

The base:0xa4a2e2ca3fbfe21aed83471d28b6f65a233c6e00 launch wallet just woke up after 241 DAYS. At 00:18 UTC, it sent a mysterious code to an empty address. Four hours later, a 3-person vault appeared at that EXACT address across 5 chains in just 4 seconds. All on the morning of Robinhood's big summit. Coincidence? Guys, I traced EVERY transaction. What I found is wild. Full video in the comments. 👇

The base:0xa4a2e2ca3fbfe21aed83471d28b6f65a233c6e00 launch wallet just woke up after 241 DAYS. At 00:18 UTC, it sent a mysterious code to an empty address. Four hours later, a 3-person vault appeared at that EXACT address across 5 chains in just 4 seconds. All on the morning of Robinhood's big summit. Coincidence? Guys, I traced EVERY transaction. What I found is wild. Full video in the comments. 👇

Kyle Chassé 🐸

15,783 次观看 • 14 天前

[WATCH] I would not want to be a part of people who are going to destabilize our movement. When I am done serving in my current role, I want to look back and say that we came in during a very difficult time, and now the ANC is back, and we are witnessing its rise again. Full interview: #LetsDoMoreTogether

[WATCH] I would not want to be a part of people who are going to destabilize our movement. When I am done serving in my current role, I want to look back and say that we came in during a very difficult time, and now the ANC is back, and we are witnessing its rise again. Full interview: #LetsDoMoreTogether

ANC SECRETARY GENERAL | Fikile Mbalula

77,002 次观看 • 2 年前

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 次观看 • 4 个月前