Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

"which quant should I download?" is a question you may never have to answer again the team Hamster Labs has figured out how to kill it with pMLX. download once at full precision (bf16) and the engine re-fits it to your machine on the fly, based on the job...

EyalToledano's profile picture

Eyal Toledano

18,306 subscribers

10,590 görüntüleme • 26 gün önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

31 Yorum

雪瑜 profil fotoğrafı
雪瑜23 gün önce

@HamsterResearch I'm really looking forward to when pMLX will be released constantly having to calculate memory usage myself is a real pain.

Felix Hesse profil fotoğrafı
Felix Hesse26 gün önce

@HamsterResearch Unbelievable! If you can do that, it's nothing less than the revolution of local AI. Then almost everyone can have their own frontirr at home. A huge potential. When and where can I download pMLX?

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch Not yet released, it’s working fully locally but I need to optimize decode speed a bit before releasing it My goal is q4 at 75% resident decoding at 100+ tok/s Hovering around mid 60’s right now which is 50%+ faster than the current release anyway Let me cook

Felix Hesse profil fotoğrafı
Felix Hesse25 gün önce

@HamsterResearch Sounds great! What do you predict for a mac Studio M2Max 64gb?

Ilya Kaminsky profil fotoğrafı
Ilya Kaminsky25 gün önce

@HamsterResearch This tweet mentions an "engine" that requires two inputs -- desired speed and context length. Where do I get it and how do I run it? Or is everyone waiting for you to publish

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch yep it’s not yet released

Ilya Kaminsky profil fotoğrafı
Ilya Kaminsky25 gün önce

@HamsterResearch Can you hook a brother up with some early access? I need it[1] for Geer[2] to cover all Apple Silicon from 16GB and up. [1] [2]

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch I’m nearly there, will push a first version even if tos benchmark doesn’t pass as it can be improved from there

techntrails profil fotoğrafı
techntrails25 gün önce

@HamsterResearch @EyalToledano please update on status App is down, website is down, email support is down. Lots of small biz owners in limbo that don't know how to get in touch with you

Eyal Toledano profil fotoğrafı
Eyal Toledano24 gün önce

@HamsterResearch 👀 mind dm’ing me?

Doug Hohner profil fotoğrafı
Doug Hohner24 gün önce

@HamsterResearch I assume this will eventually work with larger models like DSv4Flash or GLM 5.3 Flash as well? Very interesting... Trying to decide just how big a Mac to buy, will have to watch your project for a few weeks...

Eyal Toledano profil fotoğrafı
Eyal Toledano24 gün önce

@HamsterResearch gln 5.3 flash is a target right after the qwen3.8 flash next model it’s already loading on the engine at about 55gb resident but the decode is dreadful (but fixable) so odds are contributors could have at it or i’ll get to it next i can’t wait

David King profil fotoğrafı
David King26 gün önce

@HamsterResearch 🤯

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch

Ethpunk profil fotoğrafı
Ethpunk25 gün önce

@HamsterResearch Brilliant

Chris W profil fotoğrafı
Chris W26 gün önce

@HamsterResearch I love that - it is helpful especially for newbies in the game like me.

Afterkind profil fotoğrafı
Afterkind26 gün önce

@HamsterResearch This doesn't work for image models, right?

Eyal Toledano profil fotoğrafı
Eyal Toledano26 gün önce

@HamsterResearch sure it does, qwen3.8-flash-next is multimodal and the vision tower’s unaffected

EddyLeeKhane profil fotoğrafı
EddyLeeKhane25 gün önce

@HamsterResearch is it loading another already downloaded quant on the fly or somehow applying quants in real time???

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch Quantizing on the fly :)

EddyLeeKhane profil fotoğrafı
EddyLeeKhane25 gün önce

@HamsterResearch now if that works in reql time the holy cow dude your team of mad lads are off the charts!!!!!

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch Right now it’s a config flag, meaning server would restart and it would add a bit of latency But I think real time is possible and worth reaching for

Laurent Zuijdwijk profil fotoğrafı
Laurent Zuijdwijk26 gün önce

@HamsterResearch Do you know what disk space costs nowadays?

Eyal Toledano profil fotoğrafı
Eyal Toledano25 gün önce

@HamsterResearch compared to ram though, it’s night and day

SettledScience™️ profil fotoğrafı
SettledScience™️24 gün önce

@laurent_zw @HamsterResearch Any thoughts on ‘fast enough’ storage to not get in the way? I mean, TBs of latest internal SSD on board is great, but if streaming the experts from external drive is good enough, that helps save some $. TB5? TB4? USB3? What is ‘fast enough’?

Eyal Toledano profil fotoğrafı
Eyal Toledano24 gün önce

right now the expectation is the ssd is the nvme that ships on the macbook, which has plenty of bandwidth but does differ across different mac models right now i’m building and testing on an m4 max which has very good ssd bandwidth. the streaming is math exact so only thing limiting decode is hardware and the engine is obviously very young and i’m the only contributor on it for now. it will improve way faster once the community swarms on it (making the tooling to make that possible)

SettledScience™️ profil fotoğrafı
SettledScience™️24 gün önce

@laurent_zw @HamsterResearch Correct to assume the reads are more random than sustained… IOPS more important than sustained read and latencies with external are the big question mark?

Eyal Toledano profil fotoğrafı
Eyal Toledano24 gün önce

yep it’s random, not sustained, so latency matters more than bandwidth but its large-block random each expert is a few mb, not a 4k scattered read it prefetches the next token’s experts and keeps the hot ones resident so most reads are already there. external adds a protocol hop which is why nvme is baseline. so thunderbolt can prob keep up but usb is spicy at least that’s how i understand it

taisei profil fotoğrafı
taisei25 gün önce

@HamsterResearch

Tony Tong | Founder | Ancient Systems x AI profil fotoğrafı
Tony Tong | Founder | Ancient Systems x AI24 gün önce

@HamsterResearch Auto-fitting a model to your hardware beats agonizing over which quant to grab. I sat with a similar kind of indecision last year, weighing whether to leave Singapore for a cheaper cost of living elsewhere, re-running the comparison long after the answer was obvious.

Dr. Cannoli profil fotoğrafı
Dr. Cannoli25 gün önce

@HamsterResearch Wouldn’t you run into naive quantization issues? Not optimized?

Benzer Videolar

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now
2:27

Qwen3.8-Flash-Next now fits on a 48gb macbook air Got q4 running on MLX using only 39gb of memory and decoding at 28 tok/s 🔥 Also q8 running on just 50gb of memory I stacked two tricks so far: 1. REAP pruning cut the expert pool from 68gb to 35gb 2. the 51b n-gram table now lives on the ssd instead of ram My patch memmaps the shards and reads ~100 bytes per lookup straight off disk, dequantized on cpu and verified bit-exact against the gpu path Qwen3.8-Flash-Next at 288 experts with its PLE table on nvme loads in 6 seconds, 39gb resident, 28 tok/s decode, 600 tok/s prefill. Logits identical to the ram-resident version in every way 97gb stock Q4 down to 68gb if pruned to 288 experts down to 39gb if pruned to 288 experts with the table on disk humaneval scoring 91.5% with 288 experts vs. 93.9% for stock I'm not done. next: 1. MTP drafter for speculative decode 2. n-gram lookup drafter both are built, benchmarking them now

Eyal Toledano

10,468 görüntüleme • 28 gün önce

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below
0:37

Anyone with 8GB or 12GB VRAM setups needs to understand that "-ncmoe" is the key flag to boost performance on llama.cpp Here are my results for Qwen3.6 35B A3B, with 64k q8_0 context on a 8GB RTX 3070Ti: ⚪️ no flag → 8.7 tok/s RAM: 13.6GB & VRAM: 7.8GB 🔴 -ncmoe 35 → 27.5 tok/s RAM: 12.1GB & VRAM: 4.3GB 🟢 -ncmoe 30 → 32.5 tok/s RAM: 12GB & VRAM: 5.6GB 🔵 -ncmoe 25 → 40.9 tok/s RAM: 12GB & VRAM: 6.9GB Please note the ram and vram usage you see are total usage of a windows pc, with the model running. My friend's setup: 8GB VRAM and 16GB RAM. You can boost performance by switching to Linux, just something to keep in mind. Basically, this flag keeps the MoE experts in the first X layers on your CPU + RAM, instead of eating all your VRAM straight away. This is a smart hybrid offload way that lets you run bigger models without OOM while keeping the rest on your GPU for speed. As we can see on the data, there's a sweet spot. When we lower it from 35 to 25, speed bumps +50% because there are more layers on your GPU (look at the VRAM usage). The key here is to play around with the number and fit as much as possible on your VRAM, goal is to have 1GB/800MB headroom to avoid stress. ↓ server flags below

left curve dev

166,478 görüntüleme • 4 ay önce

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.
2:29

i spent 3 hours finding the sweet spot for hermes 4.3 36B on a single RTX 3090. saving you the trouble, anon. the model is 21.8GB at Q4_K_M. that leaves 2.2GB free on 24GB VRAM. not much room for KV cache. here's what actually happened: 4K: 35.3 tok/s 8K: 35.2 tok/s 16K: 34.8 tok/s 32K: 34.6 tok/s 64K: 6.4 tok/s 128K: 1.9 tok/s flat from 4K to 32K. then it falls off a cliff at 64K. the trick is quantized KV cache. without it you OOM at 16K. with quantized KV cache you get 32K at full speed. all 64 layers on GPU. at 64K something weird happens. ngl 99 (all layers on GPU) = 3.96 tok/s. the KV cache silently spills to CPU. drop to ngl 55 and speed jumps to 6.37. drop to ngl 48 and it gets worse again (3.46). there's an offload sweet spot where you free just enough VRAM for the cache without losing too much compute to PCIe transfers. 128K works at ngl 32 but you're at 1.95 tok/s. half the model on CPU. usable for batch work, not for interactive. the sweet spot command: llama-server -m hermes-4.3-36b-Q4_K_M.gguf -ngl 99 -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 32K context. 34.6 tok/s. all on GPU. this is where dense 36B lives on 24GB. for comparison, qwen 3.5 (35B MoE, 3B active) holds 112 tok/s from 4K all the way to 262K on the same GPU. no speed drop. same total params, completely different architecture. hybrid linear attention means flat context scaling. dense pays for every token in the KV cache. code and quality comparison coming next. fast vs slow generation side by side in the videos below.

Sudo su

52,268 görüntüleme • 6 ay önce

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.
0:48

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

35,794 görüntüleme • 19 gün önce

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.
1:31

i pointed hermes agent at nvidia's nemotron cascade 2 30B-A3B on a single RTX 3090 24GB. IQ4_XS quant by bartowski, 187 tok/s, 625K context. had it discover its own hardware, create an identity file, then build a full GPU marketplace UI from a single prompt. it one shotted it. first attempt no iteration. qwen 3.5 35B-A3B on the same hardware same 3090 24GB took an iteration to recover from a blank screen on the same type of build. 24 days between these two models releasing. same active parameters, completely different architectures and cascade 2 through hermes agent just keeps going. this model goes on and on. feast your eyes. more iterations and tests dropping soon. nvidia really cooked. no special flags needed. nvidia optimized this mamba MoE so well it just runs. flash attention auto enabled, context auto allocated. the model does the work not the config. but i compiled llama.cpp from source and i'm not sure how it performs on other engines. if you ran nemotron on any hardware drop your numbers below. RTX, AMD, Mac, whatever. model, quant, tok/s, engine. i want to see if it holds everywhere or just on llama.cpp.

Sudo su

71,470 görüntüleme • 6 ay önce

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.
0:50

Qwen3.8-Flash-Next just got a massive speed boost on my old hardware. This guy started with roughly 11 tok/s. After tuning the inference stack, I’m now getting around 24–28 tok/s. That’s roughly a 2× speedup without changing the GPU. The setup: → RTX 3090 24GB → 64GB DDR4 system RAM → Qwen3.8-Flash-Next → Cafe-llama.cpp The interesting part is that Cafe-llama.cpp isn’t just another llama.cpp fork. The optimizations are making a real difference for this particular architecture, especially when you’re dealing with a model that relies heavily on system memory and offloading. Going from 11 tok/s to nearly 30 tok/s completely changes how practical the model feels for local use. And now I’m curious what happens with DDR5. If memory bandwidth becomes another bottleneck, a faster RAM subsystem could squeeze even more performance out of the same 3090. This is why local AI optimization is getting so interesting. Sometimes you don’t need a new GPU. You just need a better inference stack. 11 tok/s → 24–28 tok/s.

FHILY👑

23,243 görüntüleme • 25 gün önce

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.
1:40

nemotron 3 omni q8 on dgx spark 128gb vram cranking via hermes agent at 56 tok/s. first night of real local agentic on this box and local hits harder than i thought it would. q8 (near lossless quant, perplexity loss <1% vs fp16) running 256k context on 33 gb of unified memory, 90+ gb still free. multimodal omni weights included. hermes agent driving from telegram, talking to it from bed. speed: 56 tok/s generation, 1,300 tok/s prefill. for context, qwen 3.6 27b at q4 (heavy quant) on 3090 = 40 tok/s. nemotron at higher precision quant on spark beats qwen at lower precision quant on 3090. moe 3.5b active params architecture earns its keep. what i tested tonight: agentic tool calling works clean. ask it to check disks, it autonomously runs df -h through hermes agent. ask it to set up telegram gateway, it invokes the hermes-agent skill, walks through the prompts, completes the flow. overthinks a bit before tool calls (reasoning model trait) but lands the right move every time. researches api docs, internalizes, tests, documents. completes tasks. current models on dgx spark: 9 gguf files, 305 gb total, mix of qwen 3.6 27b dense (5 quants), nemotron omni (4 quants), deepseek v4-flash 158b q4 (the 112gb flagship test). more data coming this week as i benchmark each.

Sudo su

30,228 görüntüleme • 4 ay önce

I know some bros want to see what happens when we push the value even further, so here we go: ⚪️ -ncmoe 25 → 41.8 tok/s RAM: 12GB & VRAM: 6.9GB 🔴 -ncmoe 23 → 43.8 tok/s RAM: 12.2GB & VRAM: 7.4GB 🟢 -ncmoe 21 → 38.6 tok/s RAM: 12.4GB & VRAM: 7.8GB 🔵 -ncmoe 19 → 19.8 tok/s RAM: 13.8GB & VRAM: 7.8GB As you can see, there's a sweet spot with the VRAM usage. Play around and monitor to find the right value for your setup, you can use llama.cpp web ui to monitor speeds easily Sweet spot seems to be 25-23 for 8GB VRAM ✅ +40tok/s for Qwen3.6 35B with 64k q8_0 context on a 8GB card is very impressive just by using base llama.cpp, and we didn't even try Turboquant, MTP or Dflash yet! I'll focus on these next 👀 (Server flags and setup in the quoted tweet)
0:29

I know some bros want to see what happens when we push the value even further, so here we go: ⚪️ -ncmoe 25 → 41.8 tok/s RAM: 12GB & VRAM: 6.9GB 🔴 -ncmoe 23 → 43.8 tok/s RAM: 12.2GB & VRAM: 7.4GB 🟢 -ncmoe 21 → 38.6 tok/s RAM: 12.4GB & VRAM: 7.8GB 🔵 -ncmoe 19 → 19.8 tok/s RAM: 13.8GB & VRAM: 7.8GB As you can see, there's a sweet spot with the VRAM usage. Play around and monitor to find the right value for your setup, you can use llama.cpp web ui to monitor speeds easily Sweet spot seems to be 25-23 for 8GB VRAM ✅ +40tok/s for Qwen3.6 35B with 64k q8_0 context on a 8GB card is very impressive just by using base llama.cpp, and we didn't even try Turboquant, MTP or Dflash yet! I'll focus on these next 👀 (Server flags and setup in the quoted tweet)

left curve dev

18,049 görüntüleme • 4 ay önce

Which models can your machine actually run? Magnitude is a 100% free, open source desktop app that: - profiles your hardware and runs sample calculations - predicts tok/s for every model before you download - recommends the best models, from fast to smart Pick your models and it handles the rest: - downloads and tunes the models for your hardware - connects agents like pi, opencode, hermes in one click - runs models on demand as your agent works Works on whatever hardware you already own: MacBook (Intel or Apple Silicon), Mac Mini, Mac Studio, DGX Spark, Strix Halo, any NVIDIA/AMD GPU, or just a CPU Download it on macOS, Windows, or Linux: Open source:
0:32

Which models can your machine actually run? Magnitude is a 100% free, open source desktop app that: - profiles your hardware and runs sample calculations - predicts tok/s for every model before you download - recommends the best models, from fast to smart Pick your models and it handles the rest: - downloads and tunes the models for your hardware - connects agents like pi, opencode, hermes in one click - runs models on demand as your agent works Works on whatever hardware you already own: MacBook (Intel or Apple Silicon), Mac Mini, Mac Studio, DGX Spark, Strix Halo, any NVIDIA/AMD GPU, or just a CPU Download it on macOS, Windows, or Linux: Open source:

Tom Greenwald

47,785 görüntüleme • 6 gün önce

Dear Mr. President, Greetings in the Lord. Thank you for your service to our country. As I have noted the various comments you have made about Heaven, I have felt burdened to write to you. I decided on an open letter and there are two reasons I can give for this. In the first place, you have been very open about your questions. This has been a matter of public discussion, and so I don’t feel it is out of place to address them in this way. Second, you are our president, and in this role you are in many ways our representative. I believe there are many Americans who have exactly the same questions that you do. As I write to you, I am writing also to them. And to these reasons, I might also tag on a third, which is that approaching the task this way stands a better chance of actually getting to you, which is actually the central point. I write because I am a minister of the kingdom of Heaven, and when I hear someone talking about Heaven as you have been doing, the only thing that comes to my mind is how I might possibly set out the message of Christ to you in a way that fits. How might I faithfully address the questions that you have been raising about this issue? We should want to do more than just insert the buckle into the seat belt . . . we also need to hear the click.
16:08

Dear Mr. President, Greetings in the Lord. Thank you for your service to our country. As I have noted the various comments you have made about Heaven, I have felt burdened to write to you. I decided on an open letter and there are two reasons I can give for this. In the first place, you have been very open about your questions. This has been a matter of public discussion, and so I don’t feel it is out of place to address them in this way. Second, you are our president, and in this role you are in many ways our representative. I believe there are many Americans who have exactly the same questions that you do. As I write to you, I am writing also to them. And to these reasons, I might also tag on a third, which is that approaching the task this way stands a better chance of actually getting to you, which is actually the central point. I write because I am a minister of the kingdom of Heaven, and when I hear someone talking about Heaven as you have been doing, the only thing that comes to my mind is how I might possibly set out the message of Christ to you in a way that fits. How might I faithfully address the questions that you have been raising about this issue? We should want to do more than just insert the buckle into the seat belt . . . we also need to hear the click.

Douglas Wilson

135,323 görüntüleme • 11 ay önce

been playing with hermes agent paired with qwen 3.5 dense 27B on my single 3090 since last night. there is something about this harness that caught me and i think i know what it is. i've now run five qwen configs on consumer hardware: 35B MoE (3B active) -- 112 tok/s flat across 262K context, 1x 3090 27B dense -- 35 tok/s, zero degradation across the same range, 1x 3090 qwopus 27B (opus distilled) -- 35.7 tok/s, same architecture, different brain 80B coder -- 46 tok/s on 2x 3090s, oneshotted a 564 line particle sim 80B coder -- 1.3 tok/s on 1x 3090, bleeding through RAM because it didn't fit but it still ran with same benchmarks. same prompts. same quant where possible. every config is documented. i know these models. and hermes agent is the first harness that feels like it respects that work. tool calls show inline with execution time. nvidia-smi 0.2s. write_file 0.7s. you see exactly what the agent is doing and how long each step takes. no mystery. no black box. no tool call failures so far and i've been pushing it. most agent frameworks feel like you're watching a spinner and hoping. hermes shows the work. that transparency changes how you trust the output. once you use it you see the UX decisions are not accidental. Teknium 🪽 and the nous team built this like engineers who actually use their own tools. 80 skills. 29 tools. persistent memory. context compression. runs clean on a single consumer GPU.
2:12

been playing with hermes agent paired with qwen 3.5 dense 27B on my single 3090 since last night. there is something about this harness that caught me and i think i know what it is. i've now run five qwen configs on consumer hardware: 35B MoE (3B active) -- 112 tok/s flat across 262K context, 1x 3090 27B dense -- 35 tok/s, zero degradation across the same range, 1x 3090 qwopus 27B (opus distilled) -- 35.7 tok/s, same architecture, different brain 80B coder -- 46 tok/s on 2x 3090s, oneshotted a 564 line particle sim 80B coder -- 1.3 tok/s on 1x 3090, bleeding through RAM because it didn't fit but it still ran with same benchmarks. same prompts. same quant where possible. every config is documented. i know these models. and hermes agent is the first harness that feels like it respects that work. tool calls show inline with execution time. nvidia-smi 0.2s. write_file 0.7s. you see exactly what the agent is doing and how long each step takes. no mystery. no black box. no tool call failures so far and i've been pushing it. most agent frameworks feel like you're watching a spinner and hoping. hermes shows the work. that transparency changes how you trust the output. once you use it you see the UX decisions are not accidental. Teknium 🪽 and the nous team built this like engineers who actually use their own tools. 80 skills. 29 tools. persistent memory. context compression. runs clean on a single consumer GPU.

Sudo su

100,259 görüntüleme • 6 ay önce

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!
2:59

I made the worlds best Clipboard for your AI Agents Its called Bluey and its a dumb pointer which you can use to give 10x better context to your agent And you can point at your screen, talk to it, and it will generate a really good annotated screenshot which you can now give to your AI Agent You can use it while designing your website You can use it while explaining something on your screen. And it understands you and you can be really vague about things like "move this" , modify the color "here" and it will understand what you are pointing at. Im loving using it and I built it for myself as I was tired of clicking screenshots all the time and attaching it to my codex or claude code sessions Now I can just point, speak or type and drag and drop it anywhere I am giving free access to it in the comments and I would love for you to try it out. Join the discord and I am sharing the download link there. By the way everything stays 100 percent local. No screenshots are shared anywhere!

Milind S

12,085 görüntüleme • 3 ay önce

HOW TO DRIFT IN FORZA HORIZON 6 ON CONTROLLER FOR BEGINNERS! Disclaimer, I’m not exactly Drift King, some people make me look like Donkey Kong. I do it well enough for people who are completely new to drifting to ask me how I do it, so I’m going to explain how I do it, as someone who is “decent” at best at it. First of all, you’re going to want to change your transmission to Manual so that you can maintain a consistent drift, as the speed of your vehicle determines the angle of your drift, and with Automatic, you can’t control your speed. It doesn’t matter, just personal preference here, you can do your gear changes with the buttons, but if you own an Elite Controller, you can put the paddles to good use here. I like to equip only two paddles, preferably the larger ones, and map the left to X (Downshift) and map the right to B (Upshift). You also want to turn off your Traction and Stability control assists to allow your car to slide. My vehicle of preference is the Silvia Spec R, and the code for the tune I use is 146 936 201. I am using a RWD Vehicle for reference, but some people prefer AWD. For the most part, you’re going to be driving in third and fourth gear. Third gear is great for small roads and tight, closed corners where you need a sharper drift, and fourth gear is great for wide roads and corners because the extra speed is going to throw your car further forwards and make your drift angle more shallow. It’s important to look at the size of the road and the corner you have to take and judge what gear you need to be in as the speed of your car makes all of the difference. That being said, these are not real cars, if you’re going too fast, I much prefer to hold the gas and brake pedal at the same time whilst taking the drift as opposed to engine braking. It almost makes you float around the corner at a consistent speed and gives you more control. If you are on the inside edge of the corner you’re taking, you can hold your foot brake and handbrake at the same time and your car will almost push back a little giving you more space to angle yourself correctly. If this helped you, let me know 🥹
1:27

HOW TO DRIFT IN FORZA HORIZON 6 ON CONTROLLER FOR BEGINNERS! Disclaimer, I’m not exactly Drift King, some people make me look like Donkey Kong. I do it well enough for people who are completely new to drifting to ask me how I do it, so I’m going to explain how I do it, as someone who is “decent” at best at it. First of all, you’re going to want to change your transmission to Manual so that you can maintain a consistent drift, as the speed of your vehicle determines the angle of your drift, and with Automatic, you can’t control your speed. It doesn’t matter, just personal preference here, you can do your gear changes with the buttons, but if you own an Elite Controller, you can put the paddles to good use here. I like to equip only two paddles, preferably the larger ones, and map the left to X (Downshift) and map the right to B (Upshift). You also want to turn off your Traction and Stability control assists to allow your car to slide. My vehicle of preference is the Silvia Spec R, and the code for the tune I use is 146 936 201. I am using a RWD Vehicle for reference, but some people prefer AWD. For the most part, you’re going to be driving in third and fourth gear. Third gear is great for small roads and tight, closed corners where you need a sharper drift, and fourth gear is great for wide roads and corners because the extra speed is going to throw your car further forwards and make your drift angle more shallow. It’s important to look at the size of the road and the corner you have to take and judge what gear you need to be in as the speed of your car makes all of the difference. That being said, these are not real cars, if you’re going too fast, I much prefer to hold the gas and brake pedal at the same time whilst taking the drift as opposed to engine braking. It almost makes you float around the corner at a consistent speed and gives you more control. If you are on the inside edge of the corner you’re taking, you can hold your foot brake and handbrake at the same time and your car will almost push back a little giving you more space to angle yourself correctly. If this helped you, let me know 🥹

Moki

201,342 görüntüleme • 4 ay önce

EVERYTHING you NEED to know about Jack Dorsey's AI agent "Slack killer" Buzz (set up, use-cases etc in 38 mins) What we get into: 1. What Buzz actually is and should founders switch from Slack? 2. How to swap the model under any agent and keep all your context? 3. How to talk to your agents live with audio huddles? 4. How to get agents to build and deploy real apps for you, like a full CRM from one ask? 5. How to set up the context loop that feeds your live app data back to your agents? 6. How to share compute so a few people split one machine running a local model? 7. Who it's actually for right now, and what's still rough? Full breakdown on the pod The Startup Ideas Podcast (SIP) 🧃. Thanks to Vinny for jumping on and clearly explaining jack and team's latest product. My TLDR take is Buzz is a glimpse into the future of work. Some of you will roll your eyes at that, and I get it, it's alpha software and it's slow in places. But the core idea, that your context is the foundation and agents build out from there, is right, and that's worth seeing early. Watch Curious what you think
38:43

EVERYTHING you NEED to know about Jack Dorsey's AI agent "Slack killer" Buzz (set up, use-cases etc in 38 mins) What we get into: 1. What Buzz actually is and should founders switch from Slack? 2. How to swap the model under any agent and keep all your context? 3. How to talk to your agents live with audio huddles? 4. How to get agents to build and deploy real apps for you, like a full CRM from one ask? 5. How to set up the context loop that feeds your live app data back to your agents? 6. How to share compute so a few people split one machine running a local model? 7. Who it's actually for right now, and what's still rough? Full breakdown on the pod The Startup Ideas Podcast (SIP) 🧃. Thanks to Vinny for jumping on and clearly explaining jack and team's latest product. My TLDR take is Buzz is a glimpse into the future of work. Some of you will roll your eyes at that, and I get it, it's alpha software and it's slow in places. But the core idea, that your context is the foundation and agents build out from there, is right, and that's worth seeing early. Watch Curious what you think

GREG ISENBERG

558,824 görüntüleme • 1 ay önce

hey here is the final result of octopus invaders on nvidia's flagship at full precision. nemotron super 120B on 2x H200 NVL. BF16 unquantized. 287GB of VRAM. hermes agent as the harness. 60 tok/s. first try it autonomously coded for 6 minutes straight. created 11 files. correct project structure. correct load order. started the server. i opened the browser and the result was a blank screen. i did not give up. second try i gave it a precise list of bugs and things to fix. it went back in for another 3 minutes. patched the code. served it again. still blank. so i did what any sane person would do. third try i just said the screen is blank, test it and fix it yourself. and this is where nemotron showed what it actually is. it became a debugger. you can see it in the video. realtime CSS test squares, red screen flashes, hermes agent browser tools, inspecting its own output. it built the parallax background with planets and comets. it rendered a rocket ship that tracks your mouse with fire and bullet physics. the aesthetic is real. but no enemies spawn. no collision. not playable. what surprised me is qwen 27B one shotted this exact game on a single RTX 3090 at Q4 quant. and here is nvidia's flagship at full precision on enterprise hardware needing 3 tries and still not getting there. that makes my hope high for the undisputed qwen 122B which is about to face the same test next. same hardware. same prompt and same harness. lets see if it one shots or not. full session in the video. no cuts. 5x speed.
3:36

hey here is the final result of octopus invaders on nvidia's flagship at full precision. nemotron super 120B on 2x H200 NVL. BF16 unquantized. 287GB of VRAM. hermes agent as the harness. 60 tok/s. first try it autonomously coded for 6 minutes straight. created 11 files. correct project structure. correct load order. started the server. i opened the browser and the result was a blank screen. i did not give up. second try i gave it a precise list of bugs and things to fix. it went back in for another 3 minutes. patched the code. served it again. still blank. so i did what any sane person would do. third try i just said the screen is blank, test it and fix it yourself. and this is where nemotron showed what it actually is. it became a debugger. you can see it in the video. realtime CSS test squares, red screen flashes, hermes agent browser tools, inspecting its own output. it built the parallax background with planets and comets. it rendered a rocket ship that tracks your mouse with fire and bullet physics. the aesthetic is real. but no enemies spawn. no collision. not playable. what surprised me is qwen 27B one shotted this exact game on a single RTX 3090 at Q4 quant. and here is nvidia's flagship at full precision on enterprise hardware needing 3 tries and still not getting there. that makes my hope high for the undisputed qwen 122B which is about to face the same test next. same hardware. same prompt and same harness. lets see if it one shots or not. full session in the video. no cuts. 5x speed.

Sudo su

11,011 görüntüleme • 5 ay önce

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.
4:50

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,510 görüntüleme • 3 yıl önce

Mikel Arteta on how he handles transfer speculation with his players: "Very simple. Whatever you read, if you have a question, come and knock on my door and I will tell you whether it is true or not. If we put now the catalogue on all these walls about all the players that we wanted in the last three months, I think we need the whole building to put it on. So I think they know that there's so much speculation with a lot of players. "If ever they feel the doubt of what we think of them and if we're going to recruit a player, I tell them straight away. Normally I do it - so, 'we're going to sign somebody in their position', and normally tell them that these are the plans and these are the reasons why we have the plans. 'We still believe the same in you, but the competition is going to increase.' I prefer to do it that way."
0:49

Mikel Arteta on how he handles transfer speculation with his players: "Very simple. Whatever you read, if you have a question, come and knock on my door and I will tell you whether it is true or not. If we put now the catalogue on all these walls about all the players that we wanted in the last three months, I think we need the whole building to put it on. So I think they know that there's so much speculation with a lot of players. "If ever they feel the doubt of what we think of them and if we're going to recruit a player, I tell them straight away. Normally I do it - so, 'we're going to sign somebody in their position', and normally tell them that these are the plans and these are the reasons why we have the plans. 'We still believe the same in you, but the competition is going to increase.' I prefer to do it that way."

Connor Humm

179,646 görüntüleme • 28 gün önce

I’ve been waiting for something like this. Using multiple AI agents sounds great until you have to keep giving each one the same context. Jarvix Jarvix is built to solve that： It carries your selected context across the AI tools you already use and brings everything together in one place. #JarvixOS You can use Jarvix Home to see your context and connections, Jarvix Search to find past information without remembering exact file names, and Self Reflection to spot patterns in the way you work. There’s also Jarvix Radio for personalized content discovery and Next Move to surface a useful direction based on your context. And the part I find most interesting: Agent Team Mode. Give Jarvix one goal, choose the agents you want to use, and it divides the work while keeping everyone on the same project context. You can watch the progress, review the result, and approve what happens next. No need to keep briefing every agent from scratch. One goal. Shared context. Multiple agents. You stay in control. That’s a much cleaner way to work with AI.
0:59

I’ve been waiting for something like this. Using multiple AI agents sounds great until you have to keep giving each one the same context. Jarvix Jarvix is built to solve that： It carries your selected context across the AI tools you already use and brings everything together in one place. #JarvixOS You can use Jarvix Home to see your context and connections, Jarvix Search to find past information without remembering exact file names, and Self Reflection to spot patterns in the way you work. There’s also Jarvix Radio for personalized content discovery and Next Move to surface a useful direction based on your context. And the part I find most interesting: Agent Team Mode. Give Jarvix one goal, choose the agents you want to use, and it divides the work while keeping everyone on the same project context. You can watch the progress, review the result, and approve what happens next. No need to keep briefing every agent from scratch. One goal. Shared context. Multiple agents. You stay in control. That’s a much cleaner way to work with AI.

Aaliya

23,225 görüntüleme • 1 ay önce

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.
1:24

Qwen3.6 35B-A3B dropped yesterday, so I ran it on 4 GPUs to see how it performs: 🟣 RTX 3090 — 49.78 tok/s, TTFT 852ms 🟡 RTX 4090 — 118.93 tok/s, TTFT 686ms 🟢 RTX 5090 — 160.37 tok/s, TTFT 409ms 🔵 DGX Spark — 59.98 tok/s, TTFT 228ms I went with ollama as the backend because honestly, it's the easiest way for most people to get started. One command, model pulled, done. I used Q4_K_M (24GB) across all four cards. The reason is the 3090 and 4090 don't support NVFP4 (only the 5090 and DGX Spark could use it). Keeping the same quant everywhere felt like the fairest way to compare. And yes, you can absolutely squeeze more performance out of every card with vLLM, SGLang, or TensorRT-LLM. But that's not what this test is about. This is just the out-of-the-box experience for folks who own a GPU and want to try the new model tonight.

stevibe

403,049 görüntüleme • 5 ay önce