NVIDIA just dropped Nemotron-3-Nano:4b — a tiny 2.8GB model.... Guess whose hardware runs it the fastest? - RTX 4090: 226 tok/s - RTX 3090: 187 tok/s - Mac Studio M2 Ultra: 86 tok/s - Mac Mini M4: 25 tok/s Home court advantage is real. Also trying a new layout with live performance charts. Lmk what you think!show more

Laguna XS 2.1 just matched Qwen 3.6 35B on... a real coding task & ran twice as fast. > Same RTX 3090 > Laguna: 158 tok/s. Qwen: 81 tok/s The local model race isn't about size anymore. It's about who moves faster.show more

BENCHMARKED: LFM3-1.7B by Liquid AI. Liquid AI published 1.6... GB for the KV cache at 128K context. I ran it on my Mac mini and measured 1.6 GB. The whole model at 128K uses 3.15 GB of GPU memory, under 5% of the 64GB. It still generates 80 tok/s. Only 6 of its 24 layers keep a full KV cache. The other 18 use short convolutions or sliding-window attention. llama.cpp (Metal), Q4_K_M, M5 Pro 64GB, median of 3 runs - 512 ctx: 184 tok/s, 0.09s to first token - 4K ctx: 178 tok/s, 0.72s to first token - 32K ctx: 141 tok/s - 128K ctx: 80 tok/s Q8_0 does 125 tok/s at 512, so 4-bit is 47% faster.show more

1,000 tok/s vs 85 tok/s visualized

100 tok/s on GLM-5.3-Flash, GLM-5.3 runs on the 6000s... @ 80-180 tok/s I have not tried the Studio, I’m sure it’s an amazing machine especially if you are in the Mac world already. From a pure logistics POV the sparks use little power, make less noise, serve good model & tokensshow more

now it's live🔥 GLM-5.2 on RTX 5090s: 80-110 tok/s... single stream, up from ~30. 3x faster overnight. come feel it:show more

Im calling everyone with 1 RTX 3090. Qwen3.8 Flash-Next,... 177B. This is absolutely insane model for 1 card. I swear. Here is what 1 used 3090 does with it on Strata: IQ3_S: 70 tok/s, and still 66 tok/s with 261K of context IQ2_XS: 87 tok/s prompts are read at about 1,900 tok/s The card holds the experts it uses most, the rest stays in RAM. So you need RAM for this. AJ says 64GB. My rig has 128GB, so I didnt test the limit. It doesnt have to be fast memory: mine is old DDR4-2400 on 2 channels. What you get for that: a model that scores 61.6 on LiveBench agentic coding, above every GPT, every Gemini and every Grok. 262K context, vision, and I published a version that doesnt refuse. 1 card from 2020 and 64GB of old RAM. If thats the PC under your desk and you still pay for a coding model every month, try it tonight. Links in the reply.show more

Deepseek v4.1 Flash on 24GB Macbook. I think I... reached my limit, 1.36 tok/s. It is far from being usable but the feeling of having a SOTA model on my consumer machine is awesome. I started from 0.74 tok/s and almost double it. It was a great journey.show more

Okay, I guess I didn't waste all that money... after all Deepseek V4.1 Flash 304 tok/s code, 135 tok/s prose Zero optimization, will have it max optimized by the morning.show more

A 27B model in 12 GB of VRAM. Full... 262K context. 70 tok/s. And now it is UNCENSORED. I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file. With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then Palmer took the port and did the hard part. Same card, same file, his serve: - context: 82K -> 262,144 with q8 KV - decode: 38 -> 69.9 tok/s - VRAM peak: 11.7 GB - prefill: 1,124 tok/s He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6. My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag: - harmful prompts refused: 64 of 64 -> 0 of 64 - held-out set: 78 of 82 -> 0 of 82 - decode: 69.9 -> 69.7 tok/s Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K. So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk. Cary did a genius amount of work and almost nobody has seen it. Go follow Palmer right now, I mean it. serve: weights + vector:show more

MiniMax M2.5 casually responding at ~50 tok/s with MLX... (M3 Ultra). The model was released one hour ago 🥳show more

One task, two models. GLM 5.3 Flash running slower... at 25 tok/s prose Qwen3.8 Flash running faster at 47 tok/s prose Who will finish the task first?show more

This is running on a 16gb mac mini and... 3090 Incredible model beats most of the mid stuff, qwen3.8-next-flash in exl3-3bits over half the model is being streamed from NVME it's 31 tok/s that is faster than Codex as of late.. for 2600$ I feel like my mission needs updatingshow more

M4 Mac Mini AI Cluster Uses EXO Labs with... Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 M4 Pro Mac Minis. The cluster is small (iPhone for reference). It’s running Nemotron 70B at 8 tok/sec and scales to Llama 405B (benchmarks soon).show more

454 tok/s on Qwen3.8 27B on single RTX 6000... with +6000 mem overclock Mia’s Qwen3.8-27B NVFP4 W4A4 • SGLang + DFlash2 block-16 drafter • FP8 KVshow more

Ornith-1.0 35B - MLX Device - 128gb m5 max... mbp Harness - PocketAI Hub 4 bit - 11,504 tokens, 95.98 tok/s, 21.20 GB peak RAM 8 bit - 13,128 tokens, 64.96 tok/s, 38.92 GB peak RAM BF16 - 12,491 tokens, 47.72 tok/s, 70.59 GB peak RAM prompt and link to model belowshow more

WhisperKit-0.7.0 is out! Single file inference is several times... faster! The demo below is running distil-whisper large-v3 at 300 tok/s and transcribes 101 seconds of audio in 1 second on an M2 Ultra Mac Studio. Details 🧵 Code (MIT): Demo Audio Input: Demo App: (TestFlight update under review)show more

Day-0 Qwen3.8-27B vs Qwen3.6-27B: the voxel pagoda test. Same... prompt, one attempt each, single DGX Spark, both NVFP4. 3.8: 75,291 tok · 52m 56s · 23.7 tok/s (vLLM, native MTP) 3.6: 12,117 tok · 5m 13s · 38.8 tok/s (llama.cpp, DFlash K10) 3.8 thought for 160K characters before writing a line of code.show more

10x speedup: OpenAI's privacy-filter on MLX vs CPU, side... by side in your browser. 1,818 tok/s vs 179 tok/s. Real-time PII redaction across English, German, French. On-device. No cloud. Privacy-first. Which model should we port next?show more