Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

DeepSeek-V4-Flash powering 4 parallel agents on Pi (by Mario Zechner) 🚀 Running on M3 Ultra at ~30-34 tok/s and 160-187GB peak URAM using MLX-LM. Special shoutout to clandestine.eth 🦇🔊, Pedro Cuenca, Tarjei Mandt, Ivan Fioravanti ᯅ and others for helping optimize and shape this PR. PR:

Prince_Canuma's profile picture

Prince Canuma

22,221 subscribers

109,941 görüntüleme • 4 ay önce •via X (Twitter)

SanatSağlık & İyilikBilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.
1:11

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,599 görüntüleme • 6 ay önce

New Audio Language Models on MLX 🔥🚀 - Orpheus - Sesame - Suno Bark Shoutout to Lucas Newman, Ivan Fioravanti ᯅ Pastilhas and Chi Kim for their amazing contributions Get started today: > pip install -U mlx-audio Please leave us a star and send us a PR:
0:29

New Audio Language Models on MLX 🔥🚀 - Orpheus - Sesame - Suno Bark Shoutout to Lucas Newman, Ivan Fioravanti ᯅ Pastilhas and Chi Kim for their amazing contributions Get started today: > pip install -U mlx-audio Please leave us a star and send us a PR:

Prince Canuma

35,688 görüntüleme • 1 yıl önce

DeepSeek-v4 now runs at ~23-26 tok/s on MLX! I made some custom kernels for the sinkhorn and it took gen speeds for 17 -> 26 tok/s. The weights are also significantly smaller thanks to Pedro Cuenca tip about keeping the experts in MXFP4! Now you can use it to power your local coding agents (PI, Open code, Hermes agent or even CC) PR:
0:38

DeepSeek-v4 now runs at ~23-26 tok/s on MLX! I made some custom kernels for the sinkhorn and it took gen speeds for 17 -> 26 tok/s. The weights are also significantly smaller thanks to Pedro Cuenca tip about keeping the experts in MXFP4! Now you can use it to power your local coding agents (PI, Open code, Hermes agent or even CC) PR:

Prince Canuma

58,646 görüntüleme • 4 ay önce

MiniMax M3 support added to mlx-vlm with MSA implementation! 🚀 Tested on M3 Ultra 512GB running at 24 tps with peak memory ~240GB. Now working on optimizing performance and adding ton of tests 💪 Model is here: PR is here:
0:24

MiniMax M3 support added to mlx-vlm with MSA implementation! 🚀 Tested on M3 Ultra 512GB running at 24 tps with peak memory ~240GB. Now working on optimizing performance and adding ton of tests 💪 Model is here: PR is here:

Ivan Fioravanti ᯅ

24,376 görüntüleme • 2 ay önce

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻
0:35

MLX GLM 5.2 Distributed on two M3 Ultra 512GB 🔥 One M3 Ultra: 18.8 tokens/sec Two M3 Ultra: 23.4 tokens/sec Context: - PR by Pedro Cuenca is still open and probably there is room for improvement: - basic generation test to measure decoding performance here, I will do a full context benchmarking once PR is more mature - nvfp4 quantization used - Video alternates standard speed and x20, with one Mac first and distributed later. Enjoy! 🙌🏻

Ivan Fioravanti ᯅ

87,805 görüntüleme • 2 ay önce

DwarfStar DeepSeek v4 Flash 0731 pre-M5 kernel keeps improving 💪🏻 - prefill +11% - decode +10% - single request no DSpark/MTP. - 100% same logics between baseline and now. Today is QA like crazy. I hope to file PR by EOD antirez Here are results on the M3 Ultra and a video now vs baseline using ds4-eval.
0:42

DwarfStar DeepSeek v4 Flash 0731 pre-M5 kernel keeps improving 💪🏻 - prefill +11% - decode +10% - single request no DSpark/MTP. - 100% same logics between baseline and now. Today is QA like crazy. I hope to file PR by EOD antirez Here are results on the M3 Ultra and a video now vs baseline using ds4-eval.

Ivan Fioravanti ᯅ

17,121 görüntüleme • 20 gün önce

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!
0:52

Next mlx-vlm release will ship with continuous batching support on the server 🚀 What's coming: → Continuous batching — new requests join the active batch immediately, no waiting. Mixed image + text batches supported → OpenAI-compatible API — field-for-field match with mlx-lm, reasoning/content split for thinking models, tag-aware streaming → Multi-turn tool calling — full tool use support across streaming and non-streaming, works with Gemma4 and other templates → Vision feature caching — cache image embeddings across turns. Gemma4: 228x speedup, Qwen3.5: 23x on cache hit All running locally on Apple Silicon. Check our this demo running 4 concurrent requests (mixed image + text) to gemma-4-26B-A4B-IT by Google Gemma in bf16 using Pi + MLX-VLM server on my M3 Ultra. One of the requests ingests a 8K resolution image!

Prince Canuma

82,318 görüntüleme • 4 ay önce

Congratulations to the Cohere team on releasing North Mini Code 1.0 🔥 We provide day-0 support on MLX thanks to our close collaboration with the Cohere team. North Mini Code 1.0 is a 30B param MoE model with 3B active. It runs at ~66 tok/s in BF16, truly impressive speeds before any compression. Here is a quick demo I built using Pi by Mario Zechner & Armin Ronacher ⇌ as my agent hardness. Get started today: 1. Install mlx-vlm from source: > uv pip install git+ 2. Start the server and power your favourite agents (pi, opencode, hermes and etc) > uv run mlx_vlm.server --model CohereLabs/North-Mini-Code-1.0 New release tomorrow
4:34

Congratulations to the Cohere team on releasing North Mini Code 1.0 🔥 We provide day-0 support on MLX thanks to our close collaboration with the Cohere team. North Mini Code 1.0 is a 30B param MoE model with 3B active. It runs at ~66 tok/s in BF16, truly impressive speeds before any compression. Here is a quick demo I built using Pi by Mario Zechner & Armin Ronacher ⇌ as my agent hardness. Get started today: 1. Install mlx-vlm from source: > uv pip install git+ 2. Start the server and power your favourite agents (pi, opencode, hermes and etc) > uv run mlx_vlm.server --model CohereLabs/North-Mini-Code-1.0 New release tomorrow

Prince Canuma

35,367 görüntüleme • 2 ay önce

hy3 vs mimo-v2.5 vs deepseek v4 flash vs minimax m3 the four models on top of the openrouter leaderboard by tokens this week: #1 hy3 (Tencent Hy) – 7.5t #2 mimo-v2.5 (Xiaomi MiMo) – 6.56t #3 deepseek v4 flash (DeepSeek) – 5.24t #4 minimax m3 (MiniMax (official)) – 4.21t so we tested them. 3 prompts, single-file html, Three.js from a cdn, fully procedural, no external assets. all run via AI/ML API each prompt is a transparent cutaway machine that has to be mechanically correct, not decorative: • 4-stroke engine with full oil circulation – slider-crank kinematics, cam at 2:1, valve lift driven by lobes, oil loop from sump to gallery to big-end • watt walking-beam steam engine – four-bar vector-loop closure, eccentric-driven slide valve, steam events synced to real port position • francis reaction water turbine – 20 guide vanes on a regulating ring, 17 lofted runner blades, gpu particle advection, precessing vortex rope at part load the takeaway up front: none of the four cleared all three scenes on the first attempt. but the price spread between them is roughly 70x – hy3 fixed included costs less than two cents overall results (summed across all 3 scenes): cost #1 hy3 – $0.016 #2 deepseek v4 flash – $0.025 #3 mimo-v2.5 – $0.97 #4 minimax m3 – $1.17 tokens #1 hy3 – 19,326 #2 deepseek v4 flash – 63,126 #3 mimo-v2.5 – 322,523 #4 minimax m3 – 702,900 lines of code #1 hy3 – 1,047 #2 mimo-v2.5 – 2,759 #3 deepseek v4 flash – 3,273 #4 minimax m3 – 3,354 scenes needing a second attempt #1 hy3 – 1 (engine) #1 mimo-v2.5 – 1 (turbine) #1 minimax m3 – 1 (turbine) #4 deepseek v4 flash – 2 (steam engine, turbine) observations: 1. the token spread is the real story – minimax burns 36x hy3's tokens and lands in the same place, one retry, ~3.3k lines 2. hy3 is the outlier on density: 1,047 lines total, fewest tokens, cheapest run, and only one scene needed a second pass. deepseek is the opposite trade – near-hy3 pricing but the most retries 3. mimo and minimax seem to overthink instead of writing the code. minimax spent 359.1k tokens on the steam engine and produced 1,346 lines – the tokens are going somewhere other than the file 4. the francis turbine broke three of the four. the spec that separates them is the one with 20 linked guide vanes and gpu particle advection, not the one with the most parts overall impression: none of these models excelled at any of the tasks we gave them. but they were close, and they were extremely cheap. the gap that matters isn't quality anymore – it's that hy3 ran all three scenes for less than two cents while the frontier labs charge dollars for the same work right now you pick these because they're good for the zero price you pay. soon that's something openai and anthropic will have to think about follow thehype. for 24/7 ai news, analysis and breakdowns
0:55

hy3 vs mimo-v2.5 vs deepseek v4 flash vs minimax m3 the four models on top of the openrouter leaderboard by tokens this week: #1 hy3 (Tencent Hy) – 7.5t #2 mimo-v2.5 (Xiaomi MiMo) – 6.56t #3 deepseek v4 flash (DeepSeek) – 5.24t #4 minimax m3 (MiniMax (official)) – 4.21t so we tested them. 3 prompts, single-file html, Three.js from a cdn, fully procedural, no external assets. all run via AI/ML API each prompt is a transparent cutaway machine that has to be mechanically correct, not decorative: • 4-stroke engine with full oil circulation – slider-crank kinematics, cam at 2:1, valve lift driven by lobes, oil loop from sump to gallery to big-end • watt walking-beam steam engine – four-bar vector-loop closure, eccentric-driven slide valve, steam events synced to real port position • francis reaction water turbine – 20 guide vanes on a regulating ring, 17 lofted runner blades, gpu particle advection, precessing vortex rope at part load the takeaway up front: none of the four cleared all three scenes on the first attempt. but the price spread between them is roughly 70x – hy3 fixed included costs less than two cents overall results (summed across all 3 scenes): cost #1 hy3 – $0.016 #2 deepseek v4 flash – $0.025 #3 mimo-v2.5 – $0.97 #4 minimax m3 – $1.17 tokens #1 hy3 – 19,326 #2 deepseek v4 flash – 63,126 #3 mimo-v2.5 – 322,523 #4 minimax m3 – 702,900 lines of code #1 hy3 – 1,047 #2 mimo-v2.5 – 2,759 #3 deepseek v4 flash – 3,273 #4 minimax m3 – 3,354 scenes needing a second attempt #1 hy3 – 1 (engine) #1 mimo-v2.5 – 1 (turbine) #1 minimax m3 – 1 (turbine) #4 deepseek v4 flash – 2 (steam engine, turbine) observations: 1. the token spread is the real story – minimax burns 36x hy3's tokens and lands in the same place, one retry, ~3.3k lines 2. hy3 is the outlier on density: 1,047 lines total, fewest tokens, cheapest run, and only one scene needed a second pass. deepseek is the opposite trade – near-hy3 pricing but the most retries 3. mimo and minimax seem to overthink instead of writing the code. minimax spent 359.1k tokens on the steam engine and produced 1,346 lines – the tokens are going somewhere other than the file 4. the francis turbine broke three of the four. the spec that separates them is the one with 20 linked guide vanes and gpu particle advection, not the one with the most parts overall impression: none of these models excelled at any of the tasks we gave them. but they were close, and they were extremely cheap. the gap that matters isn't quality anymore – it's that hy3 ran all three scenes for less than two cents while the frontier labs charge dollars for the same work right now you pick these because they're good for the zero price you pay. soon that's something openai and anthropic will have to think about follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

17,145 görüntüleme • 1 ay önce

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:
0:36

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:

Ivan Fioravanti ᯅ

18,381 görüntüleme • 19 gün önce

I have been testing DeepSeek-V4-Pro with the Pi coding agent. I am mindblown by how well it works out of the box. A few notes: I spent a few hours building an LLM wiki with an agent powered entirely by DeepSeek-V4-Pro on Fireworks inference. This is the first time I feel like there is an open-weight model that can reason at the level of Claude and Codex. And it does this in a cost-effective way with support for 1M context length. To be clear, I am using DeepSeek-V4-Pro inside of Pi without any special configuration. It works out of the box. It's exciting that there is a model that can just be plugged into a basic harness like Pi, and it just works. I've never seen that before. Most models require lots of configuration and setup. DeepSeek's DeepSeek-V4-Pro is clearly good at agentic coding (probably the best from the open-weight models), but the model is also great on knowledge-intensive tasks where reasoning matters. The agent pulled agentic engineering best practices from different company docs (Anthropic, OpenAI, Google, Stripe, Meta, Modal, DeepSeek, Mistral, Cohere), searched and digested Reddit and HN threads, summarized arxiv papers, and surfaced trending GitHub repos. Then it distilled everything into actionable tips across categories. I love the Wiki it built. The quality is really good. Here is a snapshot of what the wiki looks like: DeepSeek-V4-Pro handled the task without breaking stride. Multi-step research queries, code generation for scaffolding, context-heavy reasoning across disparate sources. For coding specifically, this is the first open-weight model that genuinely feels like a Codex or Claude Code experience. It compares in capability and actual multi-turn agentic work. What made the loop feel so responsive was Fireworks' inference speed (the fastest in the market) and the fact that they actually validate models at the systems level before shipping. No corrupted reasoning traces. Just fast, reliable iteration. The hybrid CSA and HCA attention design cuts KV cache to just 10% and inference FLOPs by nearly 4x at 1M-token context. This is what makes the agent loop actually fast and cheap enough to run in practice. For devs who've been watching open-weight models close the gap but haven't found one that actually delivers in practice, this is the closest I've seen. Try it here:
0:31

I have been testing DeepSeek-V4-Pro with the Pi coding agent. I am mindblown by how well it works out of the box. A few notes: I spent a few hours building an LLM wiki with an agent powered entirely by DeepSeek-V4-Pro on Fireworks inference. This is the first time I feel like there is an open-weight model that can reason at the level of Claude and Codex. And it does this in a cost-effective way with support for 1M context length. To be clear, I am using DeepSeek-V4-Pro inside of Pi without any special configuration. It works out of the box. It's exciting that there is a model that can just be plugged into a basic harness like Pi, and it just works. I've never seen that before. Most models require lots of configuration and setup. DeepSeek's DeepSeek-V4-Pro is clearly good at agentic coding (probably the best from the open-weight models), but the model is also great on knowledge-intensive tasks where reasoning matters. The agent pulled agentic engineering best practices from different company docs (Anthropic, OpenAI, Google, Stripe, Meta, Modal, DeepSeek, Mistral, Cohere), searched and digested Reddit and HN threads, summarized arxiv papers, and surfaced trending GitHub repos. Then it distilled everything into actionable tips across categories. I love the Wiki it built. The quality is really good. Here is a snapshot of what the wiki looks like: DeepSeek-V4-Pro handled the task without breaking stride. Multi-step research queries, code generation for scaffolding, context-heavy reasoning across disparate sources. For coding specifically, this is the first open-weight model that genuinely feels like a Codex or Claude Code experience. It compares in capability and actual multi-turn agentic work. What made the loop feel so responsive was Fireworks' inference speed (the fastest in the market) and the fact that they actually validate models at the systems level before shipping. No corrupted reasoning traces. Just fast, reliable iteration. The hybrid CSA and HCA attention design cuts KV cache to just 10% and inference FLOPs by nearly 4x at 1M-token context. This is what makes the agent loop actually fast and cheap enough to run in practice. For devs who've been watching open-weight models close the gap but haven't found one that actually delivers in practice, this is the closest I've seen. Try it here:

elvis

60,091 görüntüleme • 3 ay önce

Today we're shipping our biggest MLX-VLM release yet: v0.6.0 ...and we are raising 💸 This one's about turning your Apple devices into real local agent machines. From your desk to your pocket. What's new: ⚡ Speculative decoding everywhere — Gemma 4 EAGLE3 + DFlash, Qwen MTP, DeepSeek V4 MTP. Faster tokens, less waiting. 🤖 Agent-ready server — native Anthropic /v1/messages API, stateful /v1/responses, tool calls, Codex context budgets. Plug Claude Code & Codex straight into local models. 👁️ New models galore — DeepSeek V4, ZAYA1-VL, MiniCPM-V 4.6, LFM2 MoE, Step-3.7 Flash, Laguna + more. 🎨 Image gen & editing — FLUX.2 (base + klein), PrismML Bonsai. 🔊 Audio in — Qwen3 Omni, Gemma 4 audio, base64 chat audio. 🧮 TurboQuant KV cache — RHT-correct fast paths for leaner memory. 📦 Modular server, better metrics, cleaner streaming. Run real agents on the hardware already in your hands. Github:
6:47

Today we're shipping our biggest MLX-VLM release yet: v0.6.0 ...and we are raising 💸 This one's about turning your Apple devices into real local agent machines. From your desk to your pocket. What's new: ⚡ Speculative decoding everywhere — Gemma 4 EAGLE3 + DFlash, Qwen MTP, DeepSeek V4 MTP. Faster tokens, less waiting. 🤖 Agent-ready server — native Anthropic /v1/messages API, stateful /v1/responses, tool calls, Codex context budgets. Plug Claude Code & Codex straight into local models. 👁️ New models galore — DeepSeek V4, ZAYA1-VL, MiniCPM-V 4.6, LFM2 MoE, Step-3.7 Flash, Laguna + more. 🎨 Image gen & editing — FLUX.2 (base + klein), PrismML Bonsai. 🔊 Audio in — Qwen3 Omni, Gemma 4 audio, base64 chat audio. 🧮 TurboQuant KV cache — RHT-correct fast paths for leaner memory. 📦 Modular server, better metrics, cleaner streaming. Run real agents on the hardware already in your hands. Github:

Prince Canuma

65,868 görüntüleme • 2 ay önce

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!
0:21

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

45,767 görüntüleme • 23 gün önce

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.
1:40

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.

Sudo su

30,228 görüntüleme • 3 ay önce

THIS FRENCH GUY RUNS OPENCLAW AND HERMES AI AGENTS ON A HOMELAB THAT REPLACED GOOGLE DRIVE AND ICLOUD ENTIRELY ON JUST 10 WATTS The project: host his own local AI agents, OpenClaw and Hermes, ditch cloud storage subscriptions entirely by running his own NAS, and use the whole build to prep for his K8s certification – all running on 4 Raspberry Pis. Today's upgrade: a PoE module. One Ethernet cable now does two jobs – powers the Raspberry Pi AND gives it network access at the same time. No separate power adapters, no cable clutter, no juggling connections during maintenance. Total power draw for all 4 Raspberry Pis combined: max 10 watts. He's not renting cloud storage anymore. He's not paying monthly subscriptions for space he already owns the hardware for. His files, his AI agents, his data – all sitting on hardware in his own home, on a fraction of the power a lightbulb uses. Next up: blocking ads network-wide across his entire home, using one of the automation services running on the same setup. This is what "own your data" actually looks like when someone builds it instead of just talking about it. Bookmark this post. Full build in the video below.
0:45

THIS FRENCH GUY RUNS OPENCLAW AND HERMES AI AGENTS ON A HOMELAB THAT REPLACED GOOGLE DRIVE AND ICLOUD ENTIRELY ON JUST 10 WATTS The project: host his own local AI agents, OpenClaw and Hermes, ditch cloud storage subscriptions entirely by running his own NAS, and use the whole build to prep for his K8s certification – all running on 4 Raspberry Pis. Today's upgrade: a PoE module. One Ethernet cable now does two jobs – powers the Raspberry Pi AND gives it network access at the same time. No separate power adapters, no cable clutter, no juggling connections during maintenance. Total power draw for all 4 Raspberry Pis combined: max 10 watts. He's not renting cloud storage anymore. He's not paying monthly subscriptions for space he already owns the hardware for. His files, his AI agents, his data – all sitting on hardware in his own home, on a fraction of the power a lightbulb uses. Next up: blocking ads network-wide across his entire home, using one of the automation services running on the same setup. This is what "own your data" actually looks like when someone builds it instead of just talking about it. Bookmark this post. Full build in the video below.

SCOTTY BEAM

18,175 görüntüleme • 1 ay önce

This is a very nice description of the problem. I think the first part of the solution is to not give into the hyper productivity hype: running a gazillion things in parallel. There's likely only a handful of humans that have the mental make-up to survive the permanent context switches. You can train your brain to become better at it, but there's a limit to that, and I found that it will lead to much worse outcomes in terms of output quality of both you and your clanker. I'm no neuroscientist, but I would assume there's a hard limit on how many things your brain can juggle at a time, probably made worse by the complexity that is code. I generally don't do more than 2-3 things in parallel. I use virtual desktops as "bundle of things that belong to a task", so terminals + editor + browser. Works fine, doesn't need any special OS support, expose on the virtual desktop let's me quickly switch to the task specific app I need. I also have a little pi extension that shows me what GH issue/PR the clanker session is about, with a link I can click to open the issue/PR in the browser. Plus 1 virtual desktop for entertainment, so the brain can get some rest. (better alternative: get up and walk a bit every 45 minutes) Video: - First desktop: pi session for a PR, extension shows me what PR I work on, can click to open in browser, aux terminal for random BS, editor, browser. Yes, I could split instead of having two terminal windows. I'm old. - Second desktop: clawdbot pi session, aux terminal for testing, editor, browser - Third desktop: entertainment booth
0:23

This is a very nice description of the problem. I think the first part of the solution is to not give into the hyper productivity hype: running a gazillion things in parallel. There's likely only a handful of humans that have the mental make-up to survive the permanent context switches. You can train your brain to become better at it, but there's a limit to that, and I found that it will lead to much worse outcomes in terms of output quality of both you and your clanker. I'm no neuroscientist, but I would assume there's a hard limit on how many things your brain can juggle at a time, probably made worse by the complexity that is code. I generally don't do more than 2-3 things in parallel. I use virtual desktops as "bundle of things that belong to a task", so terminals + editor + browser. Works fine, doesn't need any special OS support, expose on the virtual desktop let's me quickly switch to the task specific app I need. I also have a little pi extension that shows me what GH issue/PR the clanker session is about, with a link I can click to open the issue/PR in the browser. Plus 1 virtual desktop for entertainment, so the brain can get some rest. (better alternative: get up and walk a bit every 45 minutes) Video: - First desktop: pi session for a PR, extension shows me what PR I work on, can click to open in browser, aux terminal for random BS, editor, browser. Yes, I could split instead of having two terminal windows. I'm old. - Second desktop: clawdbot pi session, aux terminal for testing, editor, browser - Third desktop: entertainment booth

Mario Zechner

40,284 görüntüleme • 6 ay önce

OpenClaw - the agentic software spreading like wildfire - was built on top of Pi, a minimalist, self-modifying agent. I sat down with Pi's creator, Mario Zechner and longtime Pi user (+ the creator of Flask) Armin Ronacher ⇌ to talk Pi, and their (very grounded!) takes on building with AI. Timestamps: 00:00 Intro 07:30 How Mario, Armin, and Peter Steinberger met 15:15 How 30 dev teams use AI agents: learnings 21:50 The importance of judgment 24:26 Challenges when non-engineers write code 28:30 Downsides of over-automation 32:18 Pi 48:09 OpenClaw + Pi 50:54 “Clankers” 57:32 Open source and AI 1:00:22 Complexity as the enemy 1:02:50 Building an AI-native startup 1:11:52 “Slow the F down” 1:16:40 MCPs vs. CLI 1:25:03 Predictions and staying up to date • YouTube: • Spotify: • Apple: Brought to you by: • Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. • Sonar — The makers of SonarQube, the industry standard for code verification and automated code review. Try it out for yourself. • WorkOS – WorkOS gives you APIs to ship enterprise features – SSO, directory sync, RBAC, audit logs – in days, not months. Visit learn more. --- Three parts I found especially interesting in this discussion: 1. New trend: AI makes it harder for senior engineers to reject pointless complexity. Historically, senior engineers kept software complexity at bay simply by saying “no” a lot. But Armin observes that these days, more junior engineers and product managers deploy agent-scripted counterarguments when a senior colleague kicks an idea to the curb. This makes decision-making exhausting, and more bad ideas make it into production as a result. 2. It should be MUCH easier to build specialized tools for specific tasks. Different projects need different harness types because, as Mario points out, the same hammer is not ideal for every single construction job. As such, Pi is built with the goal of allowing the creation of specialized harnesses. It can modify itself so that a user can create the bespoke harness needed for any task. Mario believes it’s a preview of how self-modifiable software might look in the future. 3. Automation bias is one of the biggest risks of working with AI agents. Once devs confirm that an AI agent can produce acceptable code, they start to review its output less often, even though agents can – and do! – produce slop. Mario advises being far more sceptical with agents, and cautions that the quality of their output isn’t guaranteed, however well they performed previously.
1:33:55

OpenClaw - the agentic software spreading like wildfire - was built on top of Pi, a minimalist, self-modifying agent. I sat down with Pi's creator, Mario Zechner and longtime Pi user (+ the creator of Flask) Armin Ronacher ⇌ to talk Pi, and their (very grounded!) takes on building with AI. Timestamps: 00:00 Intro 07:30 How Mario, Armin, and Peter Steinberger met 15:15 How 30 dev teams use AI agents: learnings 21:50 The importance of judgment 24:26 Challenges when non-engineers write code 28:30 Downsides of over-automation 32:18 Pi 48:09 OpenClaw + Pi 50:54 “Clankers” 57:32 Open source and AI 1:00:22 Complexity as the enemy 1:02:50 Building an AI-native startup 1:11:52 “Slow the F down” 1:16:40 MCPs vs. CLI 1:25:03 Predictions and staying up to date • YouTube: • Spotify: • Apple: Brought to you by: • Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. • Sonar — The makers of SonarQube, the industry standard for code verification and automated code review. Try it out for yourself. • WorkOS – WorkOS gives you APIs to ship enterprise features – SSO, directory sync, RBAC, audit logs – in days, not months. Visit learn more. --- Three parts I found especially interesting in this discussion: 1. New trend: AI makes it harder for senior engineers to reject pointless complexity. Historically, senior engineers kept software complexity at bay simply by saying “no” a lot. But Armin observes that these days, more junior engineers and product managers deploy agent-scripted counterarguments when a senior colleague kicks an idea to the curb. This makes decision-making exhausting, and more bad ideas make it into production as a result. 2. It should be MUCH easier to build specialized tools for specific tasks. Different projects need different harness types because, as Mario points out, the same hammer is not ideal for every single construction job. As such, Pi is built with the goal of allowing the creation of specialized harnesses. It can modify itself so that a user can create the bespoke harness needed for any task. Mario believes it’s a preview of how self-modifiable software might look in the future. 3. Automation bias is one of the biggest risks of working with AI agents. Once devs confirm that an AI agent can produce acceptable code, they start to review its output less often, even though agents can – and do! – produce slop. Mario advises being far more sceptical with agents, and cautions that the quality of their output isn’t guaranteed, however well they performed previously.

Gergely Orosz

172,936 görüntüleme • 3 ay önce

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?
0:31

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 görüntüleme • 2 ay önce

🚨 OpenAI just launched Codex, a brand-new autonomous coding agent that can build features and fix bugs on its own. We’ve been using it Every 📧 for a few days, and I’m impressed. I invited Alexander Embiricos (ben davies), a member of the product staff responsible for Codex, to demo Codex and talk about it live on a special edition of AI & I: What Codex is and how it works Codex is designed to be used by senior engineers—it performs coding tasks like adding features or fixing bugs autonomously. It's built to allow you to start many sessions at once, so you can have multiple agents working in parallel. Codex is built to have "taste" OpenAI trained Codex to have the taste of a senior software engineer. It knows how big codebases work, how to write a good PR, and uses clean, minimal code. Why an “abundance mindset” is best for interacting with agents Codex is designed to allow users to delegate many tasks at once without getting caught up in the details. This lets you point an abundance of agents at a specific task like a difficult bug—it’s worth it even if only one of them succeeds. How OpenAI is thinking about agents Codex is one piece of a unified super-assistant OpenAI wants to eventually build—an agent that helps users easily get things done by selecting the right tools for them behind the scenes. OpenAI’s vision for the future of programming In the future developers will probably spend less time writing routine code and more time guiding agents, reviewing their work, and making strategy decisions. Programming will become more social, letting teams easily delegate multiple tasks at once, allowing people to focus on ideas and collaboration instead of routine coding. Watch below!
44:33

🚨 OpenAI just launched Codex, a brand-new autonomous coding agent that can build features and fix bugs on its own. We’ve been using it Every 📧 for a few days, and I’m impressed. I invited Alexander Embiricos (ben davies), a member of the product staff responsible for Codex, to demo Codex and talk about it live on a special edition of AI & I: What Codex is and how it works Codex is designed to be used by senior engineers—it performs coding tasks like adding features or fixing bugs autonomously. It's built to allow you to start many sessions at once, so you can have multiple agents working in parallel. Codex is built to have "taste" OpenAI trained Codex to have the taste of a senior software engineer. It knows how big codebases work, how to write a good PR, and uses clean, minimal code. Why an “abundance mindset” is best for interacting with agents Codex is designed to allow users to delegate many tasks at once without getting caught up in the details. This lets you point an abundance of agents at a specific task like a difficult bug—it’s worth it even if only one of them succeeds. How OpenAI is thinking about agents Codex is one piece of a unified super-assistant OpenAI wants to eventually build—an agent that helps users easily get things done by selecting the right tools for them behind the scenes. OpenAI’s vision for the future of programming In the future developers will probably spend less time writing routine code and more time guiding agents, reviewing their work, and making strategy decisions. Programming will become more social, letting teams easily delegate multiple tasks at once, allowing people to focus on ideas and collaboration instead of routine coding. Watch below!

Dan Shipper 📧

145,487 görüntüleme • 1 yıl önce