Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Same Mac. Same model. Same words. My 512 GB M3 Ultra ran GLM-5.3-Flash on MLX last week: 27 tok/s, first token in 0.46 s. Today it runs TensorFold 0.6.2: 61 tok/s, first token in 0.20 s. In the clip it laps the reply twice while the MLX side is... show more

Volatile Markets

2,384 subscribers

26,010 просмотров • 4 дней назад •via X (Twitter)

Здоровье и велнес Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 11

Фото профиля Ash Hart

Ash Hart4 дней назад

Let’s get that to 100tps 👀

Фото профиля matt

matt4 дней назад

61 tok/s on the same box is a nice jump. does it hold once the context gets ugly?

Фото профиля Volatile Markets

Volatile Markets4 дней назад

Like duct tape.

Фото профиля 1x3e2

1x3e24 дней назад

Yep, it almost doubled the speed on my sparks with the same model. And I have already seen improvements with the new updates and recipes. Excited to see where it goes with so many people being able to find these new techniques.

Фото профиля Andrés- e/acc

Andrés- e/acc4 дней назад

The eight-request result is the real systems datapoint: concurrency exposes memory-bandwidth headroom that single-stream tok/s hides.

Фото профиля Mia

Mia4 дней назад

🔥🔥

Фото профиля Drake Stapleton

Drake Stapleton4 дней назад

2.25x on the same box, same model, measured with your own probe, is hard to argue with. The first token drop, 0.46 to 0.20, is the part that gets me. That's where interactive use lives or dies.

Фото профиля Mike Gannotti 🔳

Mike Gannotti 🔳4 дней назад

Love it.

Фото профиля Brandon

Brandon4 дней назад

With vision?

Фото профиля chris_paul_walker

chris_paul_walker4 дней назад

Crushing

Фото профиля Klasta

Klasta4 дней назад

mlx 27 then tf 61 and eight concurrent still matches solo byte for byte

Похожие видео

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Splash is here! A month ago, DFlash 2 ran Qwen3.8-27B at 70 tok/s on an M5 Max. Today, Splash runs it at 144 tok/s on the same Mac ⚡ Getting that speed used to take work. Splash is built around the model, with the DFlash 2 draft already in it. Two commands and you're running.

Zhijian Liu

92,333 просмотров • 18 дней назад

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

39,979 просмотров • 1 месяц назад

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.

I put MCDMA through its paces today and tested Qwen 3.8 Flash Next on my dual Spark / Mac Studio cluster. My first test was disaggregated prefill across my two Sparks, then decode onto my Studio. PP 2,100 tok/s Decode 80 tok/s at 20k context, higher on short replies. That's faster than either machine manages on its own: the Studio alone prefills at about 1,200 tok/s, and the Sparks alone decode at 35 to 65. 🤯 Mac prefill has always been way behind the Sparks; now I get the best of both worlds with the hardware I currently have.

Ash Hart

13,121 просмотров • 21 дней назад

DGX Spark hit 228ms time-to-first-token on Qwen3.6 35B-A3B. That is 4x faster than an RTX 3090 and 45% ahead of the RTX 5090 Raw throughput still belongs to the 5090 at 160 tok/s. Full numbers, same Ollama backend, Q4_K_M across all four: RTX 3090: 49.78 tok/s, 852ms RTX 4090: 118.93 tok/s, 686ms RTX 5090: 160.37 tok/s, 409ms DGX Spark: 59.98 tok/s, 228ms

DGX Spark hit 228ms time-to-first-token on Qwen3.6 35B-A3B. That is 4x faster than an RTX 3090 and 45% ahead of the RTX 5090 Raw throughput still belongs to the 5090 at 160 tok/s. Full numbers, same Ollama backend, Q4_K_M across all four: RTX 3090: 49.78 tok/s, 852ms RTX 4090: 118.93 tok/s, 686ms RTX 5090: 160.37 tok/s, 409ms DGX Spark: 59.98 tok/s, 228ms

NO1ennn

22,497 просмотров • 2 месяцев назад

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.

GLM-4.7 runs quite well on an M3 Ultra with mlx-lm, even at a near lossless precision (6-bit here). It generated the best space invaders game I've seen yet for a local model (even included sound effects!). Generated 6600 tokens and ran at 16 tok/s.

Awni Hannun

237,191 просмотров • 9 месяцев назад

1-bit GLM-5.2 GGUF vs. Claude 4.8 Opus vs. GPT-5.5 We gave 3 models the same prompt and compared one-shot outputs. The 1-bit GLM-5.2 GGUF ran locally on a Mac Studio M3 Ultra with 256GB RAM at ~21.6 tok/s. Which output do you like best? GGUF:

1-bit GLM-5.2 GGUF vs. Claude 4.8 Opus vs. GPT-5.5 We gave 3 models the same prompt and compared one-shot outputs. The 1-bit GLM-5.2 GGUF ran locally on a Mac Studio M3 Ultra with 256GB RAM at ~21.6 tok/s. Which output do you like best? GGUF:

Unsloth AI

1,544,807 просмотров • 3 месяцев назад

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

Sudo su

31,174 просмотров • 18 дней назад

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

Qwen3.6 27B landed yesterday, so I ran it on 4 setups side-by-side to see how they stack up: 🔴 RTX 4090 — 45.59 tok/s, TTFT 525ms 🟢 RTX 5090 — 51.83 tok/s, TTFT 752ms ⚫️ M2 Ultra — 22.30 tok/s, TTFT 216ms 🟣 DGX Spark — 11.08 tok/s, TTFT 319ms This is a standard test: no tuning, just the out-of-the-box experience. For the NVIDIA cards I used llama.cpp with Unsloth's UD-Q4_K_XL quant. For the M2 Ultra I used MLX with Unsloth's UD-MLX-4bit quant, since MLX is the native path on Apple Silicon. Please consider this as the baseline, you can definitely squeeze more out of every one of these with fine-tuned settings.

stevibe

104,692 просмотров • 5 месяцев назад

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 просмотров • 7 месяцев назад

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

GLM-5 runs with mlx-lm on a single 512GB M3 Ultra in Q4. It's quite good in my initial testing and pretty fast as well. It generated a highly functional space invaders game using 7.1k tokens at 15.4 tok/s and 419GB memory. Thanks to Gökdeniz Gülmez and Tarjei Mandt for the port.

Awni Hannun

60,694 просмотров • 7 месяцев назад

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro. ⚡ Up to 4.6× the speed of autoregressive decoding, with the same output. This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!

Zhijian Liu

1,157,430 просмотров • 1 месяц назад

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

DFlash makes Qwen 2.2x faster with no quality loss! We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks only differ in one thing - how predictable the next word is: quicksort, describe a file in JSON, a logic puzzle, a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted Baseline writes one token per step. MTP works inside the model itself and guesses 3 tokens ahead. DFlash is a separate small model that writes 15 tokens at once, and the big model only checks them. In JSON the same words repeat all the time, so most guesses were right: 152 tok/s, 3.4x speedup. In the story 9 guesses out of 10 were wrong. DFlash did all that extra work for nothing and became slower than baseline: 42 vs 44 tok/s. MTP guesses only 3 tokens, so a wrong guess costs very little: 46 tok/s and the win in that round. The output is identical in all three modes - DFlash is the pick for tasks with predictable output, like coding, and for chat and creative writing MTP works better. DFlash is now natively integrated into Atomic Chat on llama.cpp - speed up your Qwen models!

atomic.chat

83,196 просмотров • 2 месяцев назад

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.

Sudo su

71,535 просмотров • 6 месяцев назад

the tiebreaker is done. qwen 3.5 27B dense. single RTX 3090. one prompt. zero steering. zero human edits. 1,827 lines across 10 files. 13 minutes. full thinking mode. runs on first load. hermes 4.3 got the same prompt with 2x 3090s and 5x the context it needed. wrote 1,249 lines, left empty files, needed 3 interventions, game was broken on load. same architecture class. same quant. hermes got double the hardware. completely different result. dense wasn't the problem. hermes was. but here's what got me. this model thinks at 27 tok/s. every single token carries 27 billion parameters of reasoning. MoE hit 112 tok/s but only 3B active per token. the dense model is slower and it doesn't matter. watch 13 minutes of autonomous coding on a consumer GPU with zero intervention and tell me speed is what matters. a year ago this wasn't possible. now it runs on hardware you can buy used for $900. no API. no subscription. no cloud. just a 3090 doing what data centers did 18 months ago. full unedited session in the video. every token, every file, every thinking chain. 16 minutes. hit play.

the tiebreaker is done. qwen 3.5 27B dense. single RTX 3090. one prompt. zero steering. zero human edits. 1,827 lines across 10 files. 13 minutes. full thinking mode. runs on first load. hermes 4.3 got the same prompt with 2x 3090s and 5x the context it needed. wrote 1,249 lines, left empty files, needed 3 interventions, game was broken on load. same architecture class. same quant. hermes got double the hardware. completely different result. dense wasn't the problem. hermes was. but here's what got me. this model thinks at 27 tok/s. every single token carries 27 billion parameters of reasoning. MoE hit 112 tok/s but only 3B active per token. the dense model is slower and it doesn't matter. watch 13 minutes of autonomous coding on a consumer GPU with zero intervention and tell me speed is what matters. a year ago this wasn't possible. now it runs on hardware you can buy used for $900. no API. no subscription. no cloud. just a 3090 doing what data centers did 18 months ago. full unedited session in the video. every token, every file, every thinking chain. 16 minutes. hit play.

Sudo su

91,135 просмотров • 7 месяцев назад

I ran a 35-billion parameter AI agent on a $600 Mac mini. Specs: M4 Mac-Mini 16GB RAM The model doesn't fit in RAM. It pages from the SSD at 30 tokens/second. On NVIDIA, the same paging gives you 1.6 tok/s. Apple Silicon gives you 30. That's 18.6x faster. No cloud. No API keys. $0/month. Here's what it can do 🧵

I ran a 35-billion parameter AI agent on a $600 Mac mini. Specs: M4 Mac-Mini 16GB RAM The model doesn't fit in RAM. It pages from the SSD at 30 tokens/second. On NVIDIA, the same paging gives you 1.6 tok/s. Apple Silicon gives you 30. That's 18.6x faster. No cloud. No API keys. $0/month. Here's what it can do 🧵

thestreamingdev()

732,457 просмотров • 6 месяцев назад

🚨 do you understand what this AMD engineer just proved on camera watch the clip, it’s all there in one frame: llama.cpp slot logs on the left, nvitop showing 79% GPU util holding steady, localhost chat answering a 26k token prompt at 711 tok/s prompt eval, the gemma-4-12B Q4_K_XL gguf running clean. no cloud call, no api key, no censorship layer. for 2 years the rule was: frontier models need the cloud, local is too slow for real work, pay $200/month and wait for tokens. today a $1,800 lunch box runs qwen 3 235B at 11 tok/s with zero data leaving the room, and the same box runs gemma 4 12B with multi token prediction at 14+ tok/s on 30k context with no throttling. $440/month subscription stack: dead. $200/month claude tier: dead. local is too slow excuse: dead. one lunch box. $9/month in electricity. two moats buried in one week. the you need the cloud line wasn’t protecting users it was protecting the subscription.

🚨 do you understand what this AMD engineer just proved on camera watch the clip, it’s all there in one frame: llama.cpp slot logs on the left, nvitop showing 79% GPU util holding steady, localhost chat answering a 26k token prompt at 711 tok/s prompt eval, the gemma-4-12B Q4_K_XL gguf running clean. no cloud call, no api key, no censorship layer. for 2 years the rule was: frontier models need the cloud, local is too slow for real work, pay $200/month and wait for tokens. today a $1,800 lunch box runs qwen 3 235B at 11 tok/s with zero data leaving the room, and the same box runs gemma 4 12B with multi token prediction at 14+ tok/s on 30k context with no throttling. $440/month subscription stack: dead. $200/month claude tier: dead. local is too slow excuse: dead. one lunch box. $9/month in electricity. two moats buried in one week. the you need the cloud line wasn’t protecting users it was protecting the subscription.

AdiiX

22,223 просмотров • 3 месяцев назад

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

This doesn't feel like it should be possible. Muse Glimmer 30B can now run locally in your browser on WebGPU... at the same speed as llama.cpp (~25 tok/s on my M4 Max)! Just open a website. No installation required.

Xenova

21,313 просмотров • 1 месяц назад

Been giving my M3 Ultra Studio some love and tinkering with MLX. Started the afternoon at 42 tok/s. Now: 73 tok/s, and there’s still more on the table. Running Qwen3.8-Flash-Next @ 4-bit locally. Now for the crazy bit: Artificial Analysis Intelligence Index v4.3: GPT-5.6 Sol 39 (Medium) GPT-5.6 Sol 42 (High) Qwen3.8-Flash-Next 42 GPT-5.6 Sol 44 (XHigh) GPT-5.6 Sol 47 (Max) Grok 4.5 39 (High) Claude Sonnet 5 38 (Max) *Qwen score currently estimated by Artificial Analysis. A 4-bit open-weights model running at 73 tok/s on a Mac under my desk... with an estimated intelligence score matching GPT-5.6 Sol at High reasoning. Local AI is getting ridiculous. 🔥 Plus, this model does not say no to a bit of red teaming. Once I am happy with it, I will do a pr for Jun Kim's oMLX :)

Been giving my M3 Ultra Studio some love and tinkering with MLX. Started the afternoon at 42 tok/s. Now: 73 tok/s, and there’s still more on the table. Running Qwen3.8-Flash-Next @ 4-bit locally. Now for the crazy bit: Artificial Analysis Intelligence Index v4.3: GPT-5.6 Sol 39 (Medium) GPT-5.6 Sol 42 (High) Qwen3.8-Flash-Next 42 GPT-5.6 Sol 44 (XHigh) GPT-5.6 Sol 47 (Max) Grok 4.5 39 (High) Claude Sonnet 5 38 (Max) *Qwen score currently estimated by Artificial Analysis. A 4-bit open-weights model running at 73 tok/s on a Mac under my desk... with an estimated intelligence score matching GPT-5.6 Sol at High reasoning. Local AI is getting ridiculous. 🔥 Plus, this model does not say no to a bit of red teaming. Once I am happy with it, I will do a pr for Jun Kim's oMLX :)

Ash Hart

18,827 просмотров • 28 дней назад

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job you give it tell it two things: how much context you need, and the slowest speed you'll accept. it reads your mac and picks the quantization plus how many experts stay in ram vs. stream from disk. no need to download a smaller quant or figure out which quant fits your hardware. same model for every use case and the config changes based on what you need I ran this on my own M4 Max and Qwen3.8-Flash-Next splits into two configs. short context, under 32k: full bf16, ~85% of experts in ram, fast at full precision since it fits my ram long context, 64k to 256k: keep ~70% bf16 and stream the rest from SSD at 22 tok/s, or drop to q8 and get 40 tok/s. I pick per task and the model itself never changes. there are many possibilities since I have the ram to spar - if i need speed, Q4 100% resident (74gb ram @ 62 tok/s) - if i need balance, Q8 95% resident (76gb @ 38 tok/s) - if i need accuracy, bf16 70% resident (96gb @ 16 tok/s) given whatever RAM you have (16/32/64/128/256 GB) + your context + your min speed, the engine picks the precision (bf16→q8→q4) and the expert-residency/paging split that fits your box and maximizes quality & speed last thing to optimize is speed. there are so many things we want to power with open models at Hamster and these 180b-300b class models have the potential to play a big role in that

Eyal Toledano

10,590 просмотров • 1 месяц назад

Running a 27B grade model on a 6 GB RTX 4050 laptop !! I kept Bonsai 2’s transformer layers on GPU, moved only the output head to CPU, and sped up its PTQ1 dot product with AVX2. Same answer but with increased throughput of 3.7 → 11.8 tok/s in this screen recorded demo. Built on Prism’s llama.cpp

Running a 27B grade model on a 6 GB RTX 4050 laptop !! I kept Bonsai 2’s transformer layers on GPU, moved only the output head to CPU, and sped up its PTQ1 dot product with AVX2. Same answer but with increased throughput of 3.7 → 11.8 tok/s in this screen recorded demo. Built on Prism’s llama.cpp

cheesecake

18,380 просмотров • 10 дней назад