正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

DwarfStar DeepSeek v4 Flash 0731 pre-M5 kernel keeps improving 💪🏻 - prefill +11% - decode +10% - single request no DSpark/MTP. - 100% same logics between baseline and now. Today is QA like crazy. I hope to file PR by EOD antirez Here are results on the M3 Ultra... show more

Ivan Fioravanti ᯅ

40,412 subscribers

17,121 次观看 • 1 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

I didn't expect DeepSeek v4 PRO (not Flash) to run well on the Mac Studio M3 Ultra with 512GB of RAM. This is 2 bit quantized with the same DwarfStar recipe used for Flash. 433GB GGUF file. 130 t/s prefill, 13 t/s generation. Prefill in the video is low because small prompt.

antirez

172,816 次观看 • 4 个月前

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:

Where there's a will, there's a way 😎 DwarfStar DeepSeek V4 Flash 0731 mxfp4 on M3 Ultra single request no MPT/DSpark ~41 toks/s decode!!! 🚀 40 tps barrier broken! Same logits, 100% same result: this is The Mandatory Rule for any agents improving kernels that I think everyone should follow. Speed without mathematical precision is useless. Thanks 20% Opus 5 and 80% Kimi.ai K3 (what a model!) And just looking at the thinking trace of K3 you can learn tons of stuff! Here usually I have a separate chat with K3 and when I see something I want to dig deeper or ask about from the main thinking trace I do and in some cases I stop the experiment and pivot immediately to something else. Now let's see if I can apply learned lessons to the M5 kernel! 🚀 Branch for any braves soul willing to test it:

Ivan Fioravanti ᯅ

19,434 次观看 • 1 个月前

Deepseek V4 Flash 0731 (Q2) - 12 tokens/sec - Single RTX 4090 - 650+ tokens/sec prefill - 250k context - no kv cache quantization! DeepSeek just dropped the official V4 Flash 0731 two days ago with a massive agent capabilities upgrade. The official benchmarks are literally crushing their own V4-Pro-Preview on agentic tasks like Terminal Bench 2.1 and DeepSWE. Unsloth AI said they couldn't wait to bring it to local devices, and they delivered. If you thought my 118B Poolside Laguna S 2.1 MoE run last week on a single GPU was wild, hold onto your hardware. I just successfully ran Unsloth’s brand new 91GB DeepSeek-V4-Flash-0731 (UD-IQ2_M) GGUF entirely locally. And I pushed it to a mind-bending 250,000 context window. The VRAM ceiling is an illusion if you know how to optimize llama.cpp. Here are the benchmarks and the cheat codes to run a local frontier class model yourself. For the hardware and setup, I used a single NVIDIA RTX 4090 (24GB VRAM) hooked up via a PCIe 4 bus, running Ubuntu 22.04 LTS and CUDA 13.0. You don't need a massive enterprise server for this, if you have more than 80 GB of standard DDR4 RAM and a 24GB card like an RTX 3090 or 4090, you can run this exact stack yourself. All benchmarks were run using a massive 28k token prompt to truly stress test the prefill limits. no kv cache quantization THE BENCHMARKS (Scaling Context): # 80k Context (Baseline: -b 2048 -ub 2048): Prefill: 465.43 t/s | Decode: 13.00 t/s | VRAM: 22.87 GB # 80k Context (Optimized: -b 4096 -ub 4096): Prefill: 643.15 t/s | Decode: 12.20 t/s | VRAM: 23.00 GB (Notice how doubling the batch flags spiked my prefill throughput by nearly 200 t/s with almost zero VRAM penalty) # 180k Context (-b 4096 -ub 4096): Prefill: 629.18 t/s | Decode: 11.92 t/s | VRAM: 23.40 GB # 250k Context MAXIMUM (-b 4096 -ub 4096): Prefill: 619.02 t/s | Decode: 11.54 t/s | VRAM: 23.40 GB # THE SECRET SAUCE (Why this works): Unsloth’s UD-IQ2_M quant is ~91GB across 3 files. Since I only have 24GB of VRAM, the PCIe 4 bus and system RAM have to do the heavy lifting. The magic bullet is the --no-mmap flag. By completely bypassing OS disk paging, I forced llama.cpp to load the massive model weights directly into the system RAM upfront. Combined with Flash Attention (-fa on) and exactly 12 CPU threads (--threads 12), I maintained an incredibly stable 11.5+ tokens/sec decode speed even at a quarter million token context. # THE EXACT COMMAND: ./build/bin/llama-server -m /workspace/models/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf -c 250000 -fa on --port 8080 --threads 12 -b 4096 -ub 4096 --no-mmap -v Local conversational and agentic coding AI is fully here. You don’t need an API or an H100 cluster. Qwen 3.8 27b drops next week making the 24GB VRAM tier even more worthwhile. What does your current local AI rig look like, and what's the craziest model you've managed to squeeze into it? Official huggingface GGUF links from Unsloth and performance graphs are dropped in the replies below!

Alok

46,100 次观看 • 1 个月前

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Final results for DeepSeek-V4-Flash 284B 3107 on 4x3090 (2-bit quant, my llama.cpp fork) after all optimizations. This morning: 433 tok/s prefill. Right now: 1906 tok/s. At 98K depth, not empty context. Yes I was not mistaken. Its x4.3 Decode untouched, 37-40 tok/s. We did not change a single model weight. One day of removing three walls between the CPU and the cards. Chronology in the thread, raw server log in the video.

Alexey Fateev

19,218 次观看 • 1 个月前

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.

FREE DeepSeek V4 Pro and V4 Flash API. Runtime by Bad Theory Labs just dropped it. Same platform, new free model, same 2-minute setup. > Runtime already gave you free Claude Opus 4.8 and GPT 5.5 access. Now they added DeepSeek V4 Pro and V4 Flash to the free tier. Absolutely FREE, YES. Same exact flow as before: → Go to runtime(.)badtheorylabs(.)com → Sign up with Google → Fill in your details during onboarding → Free balance lands automatically(But you don't need it to use DeepSeek - it's completely free right now) → Grab your API key from the dashboard → Plug into OpenCode, OpenClaw, or Hermes - done DeepSeek V4 Pro is one of the strongest coding and reasoning models available right now. Free tier also includes btl-2 - auto-routes between GLM, OpenAI, Anthropic, and Gemma. 10 million tokens per month at zero cost. 344 models total. One OpenAI-compatible endpoint. No code rebuild, no card, no limits hunting. >> Just start using the API, and when choosing a provider, select DeepSeek I tested it myself - fully working. Bookmark this before the free tier fills up.

Atenov int.

27,487 次观看 • 3 个月前

DeepSeek v4 Flash (0731) vs Kimi K3 (10x more weights) i gave both the exact same prompt, in the exact same harness: one-shot a luxury product landing page ($10,000 "ARC-01" concept object). results in the video. my take: Kimi K3: wins on polish. Better typography hierarchy, more confident use of negative space, and the hero product shot integration is genuinely beautiful DeepSeek v4 Flash: very close. The structure is right, the copy is right, the layout logic is right. It loses on finish, not on fundamentals. a 10x smaller model getting ~90% of the way there. a year ago this gap would have been embarrassing. mind you it cost less than $0.10 for 6 build iterations and 8 QA passes. small models aren't "good for their size" anymore. they're just good.

DeepSeek v4 Flash (0731) vs Kimi K3 (10x more weights) i gave both the exact same prompt, in the exact same harness: one-shot a luxury product landing page ($10,000 "ARC-01" concept object). results in the video. my take: Kimi K3: wins on polish. Better typography hierarchy, more confident use of negative space, and the hero product shot integration is genuinely beautiful DeepSeek v4 Flash: very close. The structure is right, the copy is right, the layout logic is right. It loses on finish, not on fundamentals. a 10x smaller model getting ~90% of the way there. a year ago this gap would have been embarrassing. mind you it cost less than $0.10 for 6 build iterations and 8 QA passes. small models aren't "good for their size" anymore. they're just good.

filipe

26,329 次观看 • 2 个月前

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Google's Gemma 4 26B A4B QAT hits 25+ tokens/sec and 320+ tokens/sec prefill on 8 GB VRAM (RTX 4060) + 16 GB RAM using TurboQuant Prefill just went from 200 → 320+ tok/s on the same 8GB card. 1.6x, no new hardware, no new quant, just a KV cache trick stacked on top of the Gemma 4 26B MoE setup from a few days ago. A few days ago I posted Gemma 4 26B A4B hitting 28 tok/s decode on 8GB VRAM using native MTP. prefill was stuck around 200 tok/s. fair callout by the community. So today I tested something I'd already been meaning to try: TheTom/llama-cpp-turboquant, the TurboQuant KV cache fork by Tom Turney (Tom Turney). (github link in the comments) thanks to him, the fork just got resynced to mainline, so MTP + TurboQuant now run together cleanly (I didnt see any meaningful gains by using MTP with this setup though but you can try). The flags (No MTP): -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -c 64000 --cache-type-k q8_0 --cache-type-v turbo3 Results on the same RTX 4060 8GB, tested with a 27k token prompt at 64k context loaded: Prefill: 200 tok/s → 320+ tok/s Decode: stayed above 25 tok/s (without MTP) Why it works: TurboQuant uses walsh hadamard rotation + polar quantization on the KV cache. keys are sensitive to compression, values aren't much, so it splits the difference: K stays at q8_0, V drops to turbo3 (~3 bits). bonus from the memory savings: same 8GB card can now stretch to 100-120k context with minimal decode penalty. It should now be snappier with any agent harness such as hermes agent without compromise on intelligence. If you're already running Gemma 4 on a small card, this stacks on top for free. Try --cache-type-k q8_0 --cache-type-v turbo3 on your setup and report back what your prefill/decode split looks like. unsloth model gguf and llama.cpp turboquant fork links in the comments. what's your prefill number before vs after?

Alok

119,821 次观看 • 3 个月前

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,110 次观看 • 2 个月前

Run Updated Gemma 4 26B A4B QAT (MoE) with Vision at 25 tokens/sec and massive 120k context window on a single RTX 4060 (8 GB VRAM + 16 GB RAM Only!!) Yesterday I pushed Gemma 4 26B A4B QAT to 250k context on a single RTX 4060 using nothing but Q8 KV cache and optimized -b and -ub flags for higher prefill throughput. Today I stacked Multi Token Prediction (MTP) self speculative decoding AND the vision projector (mmproj) on top of that same card, same batch size optimization, same $250 GPU and pushed it until it broke, then found the fix. All text only runs consist of a 28k prompt. vision runs consist of 28k text prompt + an image. # 1. MTP alone. near free decode speed, no catch MTP draft assistant is a separate small model (MTP heads are backed into the main model itself for the qwen 3.5+ models but its a separate small model for gemma 4 series), 240 MB gguf 80k ctx: Prefill 510 t/s | Decode 29.5 t/s 120k ctx: Prefill 433 t/s | Decode 29 t/s 180k ctx: Prefill 240 t/s | Decode 24.9 t/s 250k ctx: Prefill 63 t/s | Decode 13 t/s llama.cpp flags: m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf-c 180000 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 # 2. Add vision on top. the tax you actually pay the vision projector gguf is about 1.1 GBs 80k ctx: Prefill 360 t/s | Decode 25.4 t/s 120k ctx: Prefill 230 t/s | Decode 23.8 t/s 180k ctx (Q8 KV): Prefill 75 t/s | Decode 12.5 t/s - cliff flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf -c 80000 --port 8080 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 --mmproj mmproj-F16.gguf # 3. The fix if you want to run vision over 120k context: swap Q8 KV for Q4 KV past 120k Stack MTP + vision + Q8 KV past 120k context and you hit a wall. draft model overhead plus KV pressure tanks everything. Drop to Q4 KV and the wall disappears: 180k ctx (Q4 KV): Prefill 220 t/s | Decode 25.5 t/s -ctk q4_0 -ctv q4_0 --mmproj mmproj-F16.gguf (rest same as above) Bottom line: MTP gives you a near free +20-30% decode boost up to 120k context. Past that, it's fighting your VRAM, not helping and if vision is loaded too, Q4 KV isn't optional past 120k, it's mandatory. 30% boost is model and card specific, MTP boosted decode 2x for gemma 4 31b on a single rtx 4090. Same 8GB card. Same $250 GPU. Multimodal, speculative decoding, 180k usable context, zero upgrades. You gotta try this if you have a single NVIDIA RTX 3050, 3060, 3070, 4050, 4060, 5050 or 5060. You can try it with a 6 GB VRAM card as well but you will have to lower the context window. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. Which models are you running on your 6/8/12GB cards with MTP?

Run Updated Gemma 4 26B A4B QAT (MoE) with Vision at 25 tokens/sec and massive 120k context window on a single RTX 4060 (8 GB VRAM + 16 GB RAM Only!!) Yesterday I pushed Gemma 4 26B A4B QAT to 250k context on a single RTX 4060 using nothing but Q8 KV cache and optimized -b and -ub flags for higher prefill throughput. Today I stacked Multi Token Prediction (MTP) self speculative decoding AND the vision projector (mmproj) on top of that same card, same batch size optimization, same $250 GPU and pushed it until it broke, then found the fix. All text only runs consist of a 28k prompt. vision runs consist of 28k text prompt + an image. # 1. MTP alone. near free decode speed, no catch MTP draft assistant is a separate small model (MTP heads are backed into the main model itself for the qwen 3.5+ models but its a separate small model for gemma 4 series), 240 MB gguf 80k ctx: Prefill 510 t/s | Decode 29.5 t/s 120k ctx: Prefill 433 t/s | Decode 29 t/s 180k ctx: Prefill 240 t/s | Decode 24.9 t/s 250k ctx: Prefill 63 t/s | Decode 13 t/s llama.cpp flags: m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf-c 180000 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 # 2. Add vision on top. the tax you actually pay the vision projector gguf is about 1.1 GBs 80k ctx: Prefill 360 t/s | Decode 25.4 t/s 120k ctx: Prefill 230 t/s | Decode 23.8 t/s 180k ctx (Q8 KV): Prefill 75 t/s | Decode 12.5 t/s - cliff flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp -md mtp-gemma-4-26B-A4B-it.gguf -c 80000 --port 8080 -b 1024 -ub 1024 --spec-draft-n-max 6 --spec-draft-p-min 0.7 -ctk q8_0 -ctv q8_0 --mmproj mmproj-F16.gguf # 3. The fix if you want to run vision over 120k context: swap Q8 KV for Q4 KV past 120k Stack MTP + vision + Q8 KV past 120k context and you hit a wall. draft model overhead plus KV pressure tanks everything. Drop to Q4 KV and the wall disappears: 180k ctx (Q4 KV): Prefill 220 t/s | Decode 25.5 t/s -ctk q4_0 -ctv q4_0 --mmproj mmproj-F16.gguf (rest same as above) Bottom line: MTP gives you a near free +20-30% decode boost up to 120k context. Past that, it's fighting your VRAM, not helping and if vision is loaded too, Q4 KV isn't optional past 120k, it's mandatory. 30% boost is model and card specific, MTP boosted decode 2x for gemma 4 31b on a single rtx 4090. Same 8GB card. Same $250 GPU. Multimodal, speculative decoding, 180k usable context, zero upgrades. You gotta try this if you have a single NVIDIA RTX 3050, 3060, 3070, 4050, 4060, 5050 or 5060. You can try it with a 6 GB VRAM card as well but you will have to lower the context window. Hugging Face links to the updated Unsloth's QAT quants and performance graph are in the replies below. Which models are you running on your 6/8/12GB cards with MTP?

Alok

16,405 次观看 • 2 个月前

A video showing impact of max frequency change on my 2 x DGX Spark cluster with DeepSeek V4 Flash 0731 nvfp4 running like crazy at c4 using Mia recipe! Power consumption at wall is surely higher than the one reported by nvtop, but it gives you an idea: - 2455 MHz ~ 47W ~72º C - 2300 MHz ~34W ~67º C - 2200 MHz ~32W ~66º C - 2000 MHz ~27W ~63º C - 1800 MHz ~23W ~61º C This has minimal impact on decode speed that is memory bound, but there is impact on prefill and other compute bound activities. set it with: sudo nvidia-smi -lgc 0,2200 reset it with: sudo nvidia-smi -rgc I need to thank Paul to let me discover this magical trick!

A video showing impact of max frequency change on my 2 x DGX Spark cluster with DeepSeek V4 Flash 0731 nvfp4 running like crazy at c4 using Mia recipe! Power consumption at wall is surely higher than the one reported by nvtop, but it gives you an idea: - 2455 MHz ~ 47W ~72º C - 2300 MHz ~34W ~67º C - 2200 MHz ~32W ~66º C - 2000 MHz ~27W ~63º C - 1800 MHz ~23W ~61º C This has minimal impact on decode speed that is memory bound, but there is impact on prefill and other compute bound activities. set it with: sudo nvidia-smi -lgc 0,2200 reset it with: sudo nvidia-smi -rgc I need to thank Paul to let me discover this magical trick!

Ivan Fioravanti ᯅ

76,968 次观看 • 1 个月前

I'm really really impressed with Qwen3.8-Flash-Next😍 It is better than Qwen3.8-27B !! Below is a screen-recording from a Three.js FPS game made by Qwen3.8-Flash-Next running on my 3090. Everything here is generated at runtime: textures, sounds, music, models and the rigid-body solver. No image, audio or mesh files are downloaded. three.js is loaded from a CDN. atomic.chat Qwen3.8-Flash-Next IQ4_XS👌 DeepSeek Harness 👌 Single RTX 3090 + 64GB RAM (Total 88 GB) Stats: - ~ 10 - 11 hrs in total - 22M tokens, 94% cache - Avg decode 20t/s - Avg prefill 200t/s - TTFT avg 9.6s It is slow, but faster than Unsloth IQ3_XXS !

I'm really really impressed with Qwen3.8-Flash-Next😍 It is better than Qwen3.8-27B !! Below is a screen-recording from a Three.js FPS game made by Qwen3.8-Flash-Next running on my 3090. Everything here is generated at runtime: textures, sounds, music, models and the rigid-body solver. No image, audio or mesh files are downloaded. three.js is loaded from a CDN. atomic.chat Qwen3.8-Flash-Next IQ4_XS👌 DeepSeek Harness 👌 Single RTX 3090 + 64GB RAM (Total 88 GB) Stats: - ~ 10 - 11 hrs in total - 22M tokens, 94% cache - Avg decode 20t/s - Avg prefill 200t/s - TTFT avg 9.6s It is slow, but faster than Unsloth IQ3_XXS !

AJ

18,718 次观看 • 1 个月前

hy3 vs mimo-v2.5 vs deepseek v4 flash vs minimax m3 the four models on top of the openrouter leaderboard by tokens this week: #1 hy3 (Tencent Hy) – 7.5t #2 mimo-v2.5 (Xiaomi MiMo) – 6.56t #3 deepseek v4 flash (DeepSeek) – 5.24t #4 minimax m3 (MiniMax (official)) – 4.21t so we tested them. 3 prompts, single-file html, Three.js from a cdn, fully procedural, no external assets. all run via AI/ML API each prompt is a transparent cutaway machine that has to be mechanically correct, not decorative: • 4-stroke engine with full oil circulation – slider-crank kinematics, cam at 2:1, valve lift driven by lobes, oil loop from sump to gallery to big-end • watt walking-beam steam engine – four-bar vector-loop closure, eccentric-driven slide valve, steam events synced to real port position • francis reaction water turbine – 20 guide vanes on a regulating ring, 17 lofted runner blades, gpu particle advection, precessing vortex rope at part load the takeaway up front: none of the four cleared all three scenes on the first attempt. but the price spread between them is roughly 70x – hy3 fixed included costs less than two cents overall results (summed across all 3 scenes): cost #1 hy3 – $0.016 #2 deepseek v4 flash – $0.025 #3 mimo-v2.5 – $0.97 #4 minimax m3 – $1.17 tokens #1 hy3 – 19,326 #2 deepseek v4 flash – 63,126 #3 mimo-v2.5 – 322,523 #4 minimax m3 – 702,900 lines of code #1 hy3 – 1,047 #2 mimo-v2.5 – 2,759 #3 deepseek v4 flash – 3,273 #4 minimax m3 – 3,354 scenes needing a second attempt #1 hy3 – 1 (engine) #1 mimo-v2.5 – 1 (turbine) #1 minimax m3 – 1 (turbine) #4 deepseek v4 flash – 2 (steam engine, turbine) observations: 1. the token spread is the real story – minimax burns 36x hy3's tokens and lands in the same place, one retry, ~3.3k lines 2. hy3 is the outlier on density: 1,047 lines total, fewest tokens, cheapest run, and only one scene needed a second pass. deepseek is the opposite trade – near-hy3 pricing but the most retries 3. mimo and minimax seem to overthink instead of writing the code. minimax spent 359.1k tokens on the steam engine and produced 1,346 lines – the tokens are going somewhere other than the file 4. the francis turbine broke three of the four. the spec that separates them is the one with 20 linked guide vanes and gpu particle advection, not the one with the most parts overall impression: none of these models excelled at any of the tasks we gave them. but they were close, and they were extremely cheap. the gap that matters isn't quality anymore – it's that hy3 ran all three scenes for less than two cents while the frontier labs charge dollars for the same work right now you pick these because they're good for the zero price you pay. soon that's something openai and anthropic will have to think about follow thehype. for 24/7 ai news, analysis and breakdowns

hy3 vs mimo-v2.5 vs deepseek v4 flash vs minimax m3 the four models on top of the openrouter leaderboard by tokens this week: #1 hy3 (Tencent Hy) – 7.5t #2 mimo-v2.5 (Xiaomi MiMo) – 6.56t #3 deepseek v4 flash (DeepSeek) – 5.24t #4 minimax m3 (MiniMax (official)) – 4.21t so we tested them. 3 prompts, single-file html, Three.js from a cdn, fully procedural, no external assets. all run via AI/ML API each prompt is a transparent cutaway machine that has to be mechanically correct, not decorative: • 4-stroke engine with full oil circulation – slider-crank kinematics, cam at 2:1, valve lift driven by lobes, oil loop from sump to gallery to big-end • watt walking-beam steam engine – four-bar vector-loop closure, eccentric-driven slide valve, steam events synced to real port position • francis reaction water turbine – 20 guide vanes on a regulating ring, 17 lofted runner blades, gpu particle advection, precessing vortex rope at part load the takeaway up front: none of the four cleared all three scenes on the first attempt. but the price spread between them is roughly 70x – hy3 fixed included costs less than two cents overall results (summed across all 3 scenes): cost #1 hy3 – $0.016 #2 deepseek v4 flash – $0.025 #3 mimo-v2.5 – $0.97 #4 minimax m3 – $1.17 tokens #1 hy3 – 19,326 #2 deepseek v4 flash – 63,126 #3 mimo-v2.5 – 322,523 #4 minimax m3 – 702,900 lines of code #1 hy3 – 1,047 #2 mimo-v2.5 – 2,759 #3 deepseek v4 flash – 3,273 #4 minimax m3 – 3,354 scenes needing a second attempt #1 hy3 – 1 (engine) #1 mimo-v2.5 – 1 (turbine) #1 minimax m3 – 1 (turbine) #4 deepseek v4 flash – 2 (steam engine, turbine) observations: 1. the token spread is the real story – minimax burns 36x hy3's tokens and lands in the same place, one retry, ~3.3k lines 2. hy3 is the outlier on density: 1,047 lines total, fewest tokens, cheapest run, and only one scene needed a second pass. deepseek is the opposite trade – near-hy3 pricing but the most retries 3. mimo and minimax seem to overthink instead of writing the code. minimax spent 359.1k tokens on the steam engine and produced 1,346 lines – the tokens are going somewhere other than the file 4. the francis turbine broke three of the four. the spec that separates them is the one with 20 linked guide vanes and gpu particle advection, not the one with the most parts overall impression: none of these models excelled at any of the tasks we gave them. but they were close, and they were extremely cheap. the gap that matters isn't quality anymore – it's that hy3 ran all three scenes for less than two cents while the frontier labs charge dollars for the same work right now you pick these because they're good for the zero price you pay. soon that's something openai and anthropic will have to think about follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

17,145 次观看 • 2 个月前

Qwen3.8 Flash Next is starting to look ridiculous on Apple Silicon. I’m running the 4-bit MTP build locally on an M3 Ultra Studio, and the latest OMP run hit: 97.1 tok/s decode 23K context ~1,131 tok/s uncached prompt processing That first number is the one that caught my attention. Nearly 100 tokens per second from a local Qwen3.8 Flash Next setup is already fast enough that the usual “local models are slow” argument starts feeling pretty outdated. And the prompt processing speed is even crazier. Over 1,100 tok/s on an uncached prompt means the model can chew through a large amount of context before generation even starts. The setup matters here. This isn’t just Qwen3.8 Flash Next running untouched. It’s a 4-bit quantized build with MTP, and the inference stack is clearly doing a lot of work behind the scenes to make the hardware perform like this. There’s already a PR open for the implementation on oMLX, so this isn’t just a one-off local experiment either. If these optimizations make their way into the broader MLX ecosystem, running large models locally on Apple Silicon gets even more interesting. The other thing I like about numbers like this is that they put the focus back on the entire inference stack. Model size is one variable. Quantization is another. Then you have MTP, KV cache configuration, runtime optimizations and the hardware itself. Change the recipe and the same model can feel completely different. Qwen3.8 Flash Next at ~97 tok/s on an M3 Ultra is a pretty good demonstration of that.

Qwen3.8 Flash Next is starting to look ridiculous on Apple Silicon. I’m running the 4-bit MTP build locally on an M3 Ultra Studio, and the latest OMP run hit: 97.1 tok/s decode 23K context ~1,131 tok/s uncached prompt processing That first number is the one that caught my attention. Nearly 100 tokens per second from a local Qwen3.8 Flash Next setup is already fast enough that the usual “local models are slow” argument starts feeling pretty outdated. And the prompt processing speed is even crazier. Over 1,100 tok/s on an uncached prompt means the model can chew through a large amount of context before generation even starts. The setup matters here. This isn’t just Qwen3.8 Flash Next running untouched. It’s a 4-bit quantized build with MTP, and the inference stack is clearly doing a lot of work behind the scenes to make the hardware perform like this. There’s already a PR open for the implementation on oMLX, so this isn’t just a one-off local experiment either. If these optimizations make their way into the broader MLX ecosystem, running large models locally on Apple Silicon gets even more interesting. The other thing I like about numbers like this is that they put the focus back on the entire inference stack. Model size is one variable. Quantization is another. Then you have MTP, KV cache configuration, runtime optimizations and the hardware itself. Change the recipe and the same model can feel completely different. Qwen3.8 Flash Next at ~97 tok/s on an M3 Ultra is a pretty good demonstration of that.

FHILY👑

15,378 次观看 • 16 天前

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

glm 5.3 vs qwen 3.8 vs gemini 3.7 vs deepseek v4 flash four models designed and built three structures each on a physics-backed site, with no dimensions anywhere in the brief the setup: our own agent loop on OpenRouter, a construction site as the tool set – footings, walls, arches, roofs, scaffold, a lamp. the site enforces physics and nothing else: unsupported brick falls, a roof needs walls under it, a worker reaches 3.2 m above whatever he stands on, an arch needs centring until the keystone is set, concrete cures before it carries. no budget ceiling – material cost is tallied and reported, never blocked. tasks: 1. house – a plot and a palette, no plan. shape, height and material are the model's call 2. lighthouse – a headland cut by a gully, with a rock stack standing 30 m offshore. the lamp must burn, it must be the highest thing built, and the keeper must be able to walk to it 3. bridge – a river with one islet and banks at different heights. cross it however you want models: Z.ai glm 5.3 flash, Qwen qwen 3.8 flash, Google DeepMind gemini 3.7 flash, DeepSeek v4 flash vision all twelve objects were finished and signed off by the models themselves. tallest lighthouse is qwen's at 38.4 m, planted on the offshore stack with a bridge run out to it – the only model that read the site that way. deepseek signed off its bridge on an empty riverbed: 0 bricks, 107 minutes, $1.16m of material tallied - total cost, three builds #1 glm 5.3 flash – $0.201 #2 gemini 3.7 flash – $0.871 #3 qwen 3.8 flash – $1.058 #4 deepseek v4 flash – $1.567 - wall clock, three builds #1 gemini 3.7 flash – 91m #2 glm 5.3 flash – 228m #3 deepseek v4 flash – 502m #4 qwen 3.8 flash – 912m - total tokens #1 gemini 3.7 flash – 3,567,052 #2 glm 5.3 flash – 4,732,748 #3 qwen 3.8 flash – 13,469,333 #4 deepseek v4 flash – 18,230,076 - defects logged by the site #1 deepseek v4 flash – 59 #2 gemini 3.7 flash – 132 #3 glm 5.3 flash – 221 #4 qwen 3.8 flash – 350 - material tallied across three builds #1 gemini 3.7 flash – $359,884 #2 glm 5.3 flash – $583,358 #3 deepseek v4 flash – $1,327,484 #4 qwen 3.8 flash – $2,188,625 observations: • glm is the cheap one and nothing here is close – $0.201 for three buildings, $0.042 per million tokens, 6x under gemini's rate • what glm spends it on is bulk, not care: 166,228 bricks in one house and 156 defect weight, the worst single object in the set • gemini is the efficiency line – 91 minutes and 3.57m tokens for all three and an eighth of qwen's clock • gemini also builds the smallest of everything. its lighthouse is 22.5 m against qwen's 38.4, its house 6.9 m against 19.3 • qwen is the maximalist: 1.18m bricks, $2.19m of material, tallest on all three tasks, and 912 minutes – 15 hours – to get there conclusion: twelve finished objects for $3.80 all in, and a 7.8x price spread between the cheapest model and the priciest! follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

26,360 次观看 • 1 个月前

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now

Eyal Toledano

10,468 次观看 • 1 个月前

Pi Agent vs OpenCode token usage A lot of people recommended Pi Agent so I decided to check Pi Agent took 1.1k tokens in first turn OpenCode took 11.5k Setup: 1) Trimmed OpenCode (from usual 30k first turn to 11.5k) - 0 MCPs - 2 lightweight plugins (opencode-env-protect and openslimedit) - 8k char AGENTS.md - 11585 deepseek-v4-flash input tokens - for $0.0016 2) Vanilla Pi - 0 MCPs - 0 system prompt - 8k char AGENTS.md - 1114 kimi-k2.6 input tokens - for $0.0008 I think using better models with capped tokens per turn can keep usage nearly the same as uncontrolled DeepSeek V4 Flash? The challenge now is finding the sweet spot. Can't cap tokens if quality drops. We'll see. Video below - OpenCode vs Pi side by side "say hi back" first turns test, with OpenCode Go usage for each

Pi Agent vs OpenCode token usage A lot of people recommended Pi Agent so I decided to check Pi Agent took 1.1k tokens in first turn OpenCode took 11.5k Setup: 1) Trimmed OpenCode (from usual 30k first turn to 11.5k) - 0 MCPs - 2 lightweight plugins (opencode-env-protect and openslimedit) - 8k char AGENTS.md - 11585 deepseek-v4-flash input tokens - for $0.0016 2) Vanilla Pi - 0 MCPs - 0 system prompt - 8k char AGENTS.md - 1114 kimi-k2.6 input tokens - for $0.0008 I think using better models with capped tokens per turn can keep usage nearly the same as uncontrolled DeepSeek V4 Flash? The challenge now is finding the sweet spot. Can't cap tokens if quality drops. We'll see. Video below - OpenCode vs Pi side by side "say hi back" first turns test, with OpenCode Go usage for each

raymel 👋

209,100 次观看 • 4 个月前

qwen 3.8 max vs deepseek v4 flash 0731 vs kimi k3 vs gpt 5.6 sol – on rubik's cube and chess four frontier models built a rubik's cube stand and solved it, then built a chess board and played claude opus 5 on it the setup: Nous Research's hermes agent cli on OpenRouter tasks: 1. cube – build a 3d rubik's cube with a cli and a Three.js viewer, then solve an identical scrambled position on your own stand 2. chess – build a 3d chess stand, then play white against claude opus 5 as black, live, one move at a time. no engine, no solver, no opening book on either side. stockfish depth 14 grades every chess ply afterwards; neither player sees the score models: DeepSeek v4 flash 0731, OpenAI gpt-5.6 sol, Kimi.ai kimi k3, Qwen qwen 3.8 max gpt-5.6 sol and deepseek v4 flash solved their cubes – sol in 24 moves and seventeen seconds, deepseek in 32. qwen and kimi never got there, giving up at 96 and 207 moves then all four built chess stands and played white against claude opus 5 on them, and all four resigned: deepseek on move 13, sol on 19, kimi on 21, qwen holding out longest at 29 - build time, both stands #1 gpt-5.6 sol – 16m 43s #2 deepseek v4 flash – 97m 39s #3 kimi k3 – 166m 09s #4 qwen 3.8 max – 215m 08s - build attempts before a working stand #1 gpt-5.6 sol – 3 #2 qwen 3.8 max – 4 #3 kimi k3 – 4 #4 deepseek v4 flash – 5 - total tokens #1 gpt-5.6 sol – 6,713,754 #2 qwen 3.8 max – 17,272,507 #3 kimi k3 – 22,427,504 #4 deepseek v4 flash – 27,417,442 - total price #1 deepseek v4 flash – $0.557 #2 gpt-5.6 sol – $6.319 #3 qwen 3.8 max – $10.270 #4 kimi k3 – $16.667 observations: • deepseek v4 flash is the cheapest model here by a margin nobody else is near, and it got there while being the least efficient of the four. it burned 27.4m tokens – more than anyone, 5m more than kimi – and still finished both benchmarks for $0.557. that is $0.02 per million tokens against kimi's $0.74. it also needed the most passes to produce working stands, five, and that did not matter: all five deepseek passes together cost a thirtieth of kimi's two • so what deepseek cannot do is get it right the first time. what it can do is get it right the fifth time, for half a dollar. that is a different thing to be buying – not a good first draft, but the option to keep asking • gpt-5.6 sol is the opposite profile and the strongest of the four on pure efficiency. 16m 43s to build both stands, 6.7m tokens, three passes – under 40% of the next lowest token count and a quarter of deepseek's, on an eighth of qwen's clock. it also solved the cube fastest of anyone, 24 moves in seventeen seconds. sol is what you reach for when you want the answer now and can absorb $0.94 per million • sol's weakness is in what it does not check. its chess viewer deleted the capturing piece instead of the captured one, so pieces disappeared off the board mid-game – a defect the fifty-cent deepseek stand did not have. fast and terse turns out to be the same dial as fast and unverified • qwen 3.8 max is not the cheap open-weights option it gets treated as. $10.270 across the two benchmarks, second most expensive of the four, 18x deepseek, and by a distance the slowest – 215 minutes of build time, nearly thirteen times sol's. what the money buys is judgment: it played eighteen moves without a single error worth a hundredth of a pawn, then made exactly one bad move in the whole game, and averaged 44.6 centipawns lost across the longest game any of the four managed. it also could not solve a rubik's cube in 96 tries • kimi k3 is the one line with no reading that flatters it. most expensive at $16.667, last on the cube at 207 moves, last at chess at 478 centipawns lost per move. it is also the model that verified hardest – on the cube it wrote its own integrity check instead of trusting its output. that makes the result worse rather than better: the checking was real, and the reasoning underneath it still was not follow thehype. for 24/7 ai news, analysis and breakdowns

thehype.

84,777 次观看 • 1 个月前

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Very quick comparison between Ornith 1.5 35B MoE and Qwen 3.8 27B Dense. 📣 Clearly it's not a fair one, but let's in any case see how it went. 397B download in progress! In the Videos below: - Brick SA -> Qwen 3.8 - Lego Streets -> Ornith 1.5 Context: - Pi agent used in both cases, prompt below - M3 Ultra 512GB - Ornith-1.5-35B-A3B-oQ8e MTP hosted on oMLX - mlx-community/Qwen3.8-27B-8bit DFlash 2 hosted on mlx-dspark - Ornith has been incredibly fast with speed from 75 t/s to 45 t/s (160K+ context) - Qwen3.8 suffered context much more reaching 5 t/s above 160K context, but it can be due to engine tested still work in progress - Prompt: using threejs, and cdn, create a lego like game that's inspired by gta san andreas, with beautiful aesthetics and graphics and ability to steal cars. it should be 3d and have a nice and large map and areas. the graphics should be decent and nice. it should feature iconic things from gta san andreas Notes: - Qwen 3.8 result is 0-shot, while Ornith 1.5 required 6 iterations - Ornith 1.5 is not at the same level of autonomy as Qwen 3.8 27B honestly. To try getting the same results I'm constantly nudging, steering and 🤬 at it. - Qwen 3.8 took 6 hours to complete, but it was more a problem of timeout of Chrome headless used for testing, real prefill/decode TBD. I'll test again now with oMLX - Pi agent has a nearly perfect Cache Hit ratio that for local models is MEGA important!

Ivan Fioravanti ᯅ

12,150 次观看 • 1 个月前

Anthropic Engineer: “the biggest mistake in AI right now is people are building graphs and loops without a self-improving eval agent. we made that mistake at anthropic. it cost us two years.” here’s how they built what solved it: step 1 → take 50 real user prompts and run your agent. if it passes 80%+, your eval is too easy. the sweet spot is around 50%. step 2 → every failed run becomes a transcript. feed it to haiku: “what went wrong?” your eval set keeps growing on its own. step 3 → score two things: the right answer and the right path. a right answer with the wrong path will break later. step 4 → when a new model drops, run the same eval again. one model looked 9% better. but 6% of that gain came from hiding a bug. the transcript caught it. the dashboard didn’t. step 5 → plug your evals into ci. no green evals = no deploy. that’s how the system keeps improving itself. 99% of people never build this. but it pays for itself on day one. save this and build it today.

Anthropic Engineer: “the biggest mistake in AI right now is people are building graphs and loops without a self-improving eval agent. we made that mistake at anthropic. it cost us two years.” here’s how they built what solved it: step 1 → take 50 real user prompts and run your agent. if it passes 80%+, your eval is too easy. the sweet spot is around 50%. step 2 → every failed run becomes a transcript. feed it to haiku: “what went wrong?” your eval set keeps growing on its own. step 3 → score two things: the right answer and the right path. a right answer with the wrong path will break later. step 4 → when a new model drops, run the same eval again. one model looked 9% better. but 6% of that gain came from hiding a bug. the transcript caught it. the dashboard didn’t. step 5 → plug your evals into ci. no green evals = no deploy. that’s how the system keeps improving itself. 99% of people never build this. but it pays for itself on day one. save this and build it today.

MIKE

102,478 次观看 • 1 个月前

Every big MoE model I test on my 3090 is basically to see how good it is compared to Qwen3.6-27B. "How good is it compared to Qwen3.6-27B?" is the question I get asked the most, and that shows how much people love and, more importantly, trust the 27B 27B is battle tested. It has proven its worth and has nothing left to prove. Now it just needs to make way for the next 27B. Here’s an example video: a Three.js task, same prompt, not oneshot but two shots for all models. All ran on my 3090 + 64GB VRAM Qwen-3.6-27b fully on vram while others have experts offloaded to system ram. Pinned on the left: Qwen3.6-27B-Q5 On the right, i'm switching between Inkling-Small-Q4_K_M DSV4-Flash-0731-UD-IQ1 Ling-3.0-Flash-Q4_K_M Qwen3.6-27B nailed it. The effects and visuals are just spot on.The other models got some things right, but not everything. Second place would have to go to Inkling-Small while DSV4-Flash-0731 and Ling-3.0-Flash failed to capture the full requirements. This is also why we’re all excited for Qwen Qwen3.8-27B!

Every big MoE model I test on my 3090 is basically to see how good it is compared to Qwen3.6-27B. "How good is it compared to Qwen3.6-27B?" is the question I get asked the most, and that shows how much people love and, more importantly, trust the 27B 27B is battle tested. It has proven its worth and has nothing left to prove. Now it just needs to make way for the next 27B. Here’s an example video: a Three.js task, same prompt, not oneshot but two shots for all models. All ran on my 3090 + 64GB VRAM Qwen-3.6-27b fully on vram while others have experts offloaded to system ram. Pinned on the left: Qwen3.6-27B-Q5 On the right, i'm switching between Inkling-Small-Q4_K_M DSV4-Flash-0731-UD-IQ1 Ling-3.0-Flash-Q4_K_M Qwen3.6-27B nailed it. The effects and visuals are just spot on.The other models got some things right, but not everything. Second place would have to go to Inkling-Small while DSV4-Flash-0731 and Ling-3.0-Flash failed to capture the full requirements. This is also why we’re all excited for Qwen Qwen3.8-27B!

AJ

11,670 次观看 • 1 个月前