1,000 tok/s vs 85 tok/s visualized

This is awesome. Thanks Mia for this Was able... to run with the RTX 3090 use this .env config DRAFT=mtp CONTEXT_SIZE=262144 CACHE_QUANT=8,4 GPU_MEM_GB=22 262K context: - Run 1: 65.98 tok/s - Run 2: 63.23 tok/s - Run 3: 64.41 tok/s Average: ~64.5 tok/s.show more

Day-0 Qwen3.8-27B vs Qwen3.6-27B: the voxel pagoda test. Same... prompt, one attempt each, single DGX Spark, both NVFP4. 3.8: 75,291 tok · 52m 56s · 23.7 tok/s (vLLM, native MTP) 3.6: 12,117 tok · 5m 13s · 38.8 tok/s (llama.cpp, DFlash K10) 3.8 thought for 160K characters before writing a line of code.show more

10x speedup: OpenAI's privacy-filter on MLX vs CPU, side... by side in your browser. 1,818 tok/s vs 179 tok/s. Real-time PII redaction across English, German, French. On-device. No cloud. Privacy-first. Which model should we port next?show more

NVIDIA just dropped Nemotron-3-Nano:4b — a tiny 2.8GB model.... Guess whose hardware runs it the fastest? - RTX 4090: 226 tok/s - RTX 3090: 187 tok/s - Mac Studio M2 Ultra: 86 tok/s - Mac Mini M4: 25 tok/s Home court advantage is real. Also trying a new layout with live performance charts. Lmk what you think!show more

RadixArkのQwen3.8-Flash-Next-NVFP4 つかってFreeTokenをcodexで改造してやっと良きスピード出た Activeが小さいからそれなりに速度出た RAM DDR5 8ch GPU - RTX5090... x1 Prefill 277.5 tok/s Decode 59.13 tok/s - RTX5090 x2 Prefill 619.5 tok/s Decode 83.14 tok/sshow more

One task, two models. GLM 5.3 Flash running slower... at 25 tok/s prose Qwen3.8 Flash running faster at 47 tok/s prose Who will finish the task first?show more

Ornith-1.0 35B - MLX Device - 128gb m5 max... mbp Harness - PocketAI Hub 4 bit - 11,504 tokens, 95.98 tok/s, 21.20 GB peak RAM 8 bit - 13,128 tokens, 64.96 tok/s, 38.92 GB peak RAM BF16 - 12,491 tokens, 47.72 tok/s, 70.59 GB peak RAM prompt and link to model belowshow more

Okay, I guess I didn't waste all that money... after all Deepseek V4.1 Flash 304 tok/s code, 135 tok/s prose Zero optimization, will have it max optimized by the morning.show more

Fruit-fly connectome optimized inference for DeepSeek-V4.1-Flash on 2× NVIDIA... GB300 DGX Stations! It impressively hits 56k tok/s prefill and 201.7 tok/s c1 decode. Well done little dude.show more

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6... GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.show more

no one is ready for GPT-5.6 on Cerebras at... 750 tok/sshow more

Making a bop folder on tik tok dm if... interested (『S』 will be included)show more

🏁 BENCHMARKED: webAI's TwIL-8B, an 8.8B open-weight reasoning model,... just went 69/69 on my tests running on my 64GB M5 Pro Mac mini. The 4-bit MLX build runs at 53.8 tok/s from a 4.9 GB file. Full bf16 with the whole 128K context fits too, peaking at 40.5 GB on a 64GB machine. Same scores on both builds, thinking on: ✅ Math 20/20 ✅ Code 10/10 ✅ Instruction following 10/10 ✅ Harder math 8/8 ✅ Harder code 5/5 ✅ Tool calls 5/5 ✅ Hidden fact at 32K and 64K: 8/8 ✅ Hidden fact at full 128K: 3/3 (bf16) 4-bit speed by context: - 512: 53.8 tok/s - 32K: 26.6 tok/s - 128K: 9.2 tok/s at 27.9 GB peak bf16 speed by context: - 512: 16.7 tok/s - 32K: 12.6 tok/s - 128K: 6.6 tok/s David Stout called it "small enough to run on your own hardware." I checked and it runs at full precision with the whole 128K context on a desk I own.show more

Laguna XS 2.1 just matched Qwen 3.6 35B on... a real coding task & ran twice as fast. > Same RTX 3090 > Laguna: 158 tok/s. Qwen: 81 tok/s The local model race isn't about size anymore. It's about who moves faster.show more

Two models. One DGX Spark One identical voxel Eiffel... Tower prompt. Qwen3.8-27B xhigh: 83K tokens, 41m29s, 33.4 tok/s Ornith 1.5 35B-A3B: 47K tokens, 10m35s, 74.0 tok/s Thinking on, no token ceiling, matched camera. Ornith finished much faster; Qwen spent far more tokens.show more

4 DGX Sparks running DeepSeek V4.1 Flash 6 concurrent... sessions, 170.73 tok/s coding throughput. 🔥 1,200 output tokens in 7.028 seconds. Warm single-stream: 💻 Code: 77.61 tok/s 🧮 Math: 74.08 🧠 Reasoning: 48.75 ✍️ Prose: 33.02 6-stream aggregate: 💻 Code: 170.73 tok/s 🔢 Counting: 291.64 tok/s 552B MoE · MXFP4 · TP4 · vLLM + DSpark + CUDA graphs ✅ Tool calling + vision ✅ 127,480-token retrieval test passed ✅ 300K configured context Video replays the actual recorded streams at 1× speed. Thinking OFF, 200-token cap per coding request. Short-prompt coding throughput Credit to Tech2Wild for the Boot9 recipe. NVIDIA AIshow more

what's up tik tok tok tok tok tok tokkk... - #raymcaffey #bootsnetflix #bootsedit #raymcaffeyedit #liamohshow more

tik tok tik tok tik tok

Deepseek v4.1 Flash on 24GB Macbook. I think I... reached my limit, 1.36 tok/s. It is far from being usable but the feeling of having a SOTA model on my consumer machine is awesome. I started from 0.74 tok/s and almost double it. It was a great journey.show more

MiniMax M2.5 casually responding at ~50 tok/s with MLX... (M3 Ultra). The model was released one hour ago 🥳show more