Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Running a local agent 24/7 until it successfully gets paid for a security bug bounty Qwen3.8-27B - intel arc b70 - 262k context - vision - 60 tok/s decode - 1500-6000 tok/s prefill

0xSero

76,410 subscribers

192,144 Aufrufe • vor 27 Tagen •via X (Twitter)

Nachrichten & Politik Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

40 Kommentare

Profilbild von Noah Bertilson

Noah Bertilsonvor 27 Tagen

Bought a B70 yesterday afternoon, but I've run into errors/issues running your config. May be because I'm on Artix instead of Omarchy. Host kernel is latest: 7.2.7-artix, not sure what compute runtime, IGC, LZL versions.

Profilbild von 0xSero

0xSerovor 27 Tagen

Do you have a Claude/codex sub or maybe free opencode? Give them the repo and ask them to figure the issue out fix it and test it then file a pr

Profilbild von Noah Bertilson

Noah Bertilsonvor 27 Tagen

Unfortunately not that token-wealthy. I'll just keep asking Hermes about it with my OpenRouter credits...

Profilbild von 0xJustice.eth 👹🔥

0xJustice.eth 👹🔥vor 27 Tagen

Happy to share this repo with you if you like. I learned a lot in the process. All documented.

Profilbild von Sacred Pop | Catholic Music

Sacred Pop | Catholic Musicvor 27 Tagen

I'm probably sleeping on leveraging my 5090 for something like this, where is a good market for these kinds of bounties? Aren't they saturated markets?

Profilbild von 0xSero

0xSerovor 27 Tagen

How will they saturate when “frontier models” refuse to do it

Profilbild von Sacred Pop | Catholic Music

Sacred Pop | Catholic Musicvor 27 Tagen

Good point, do you have a link to where these bounties are? I would be interested in putting my local qwen on some.

Profilbild von NeoCortex1984

NeoCortex1984vor 27 Tagen

@0xSero Unironically task your local agent to find it and evaluate yourself if the links are solid. Let it be it's first task into this world.

Profilbild von Sacred Pop | Catholic Music

Sacred Pop | Catholic Musicvor 27 Tagen

@0xSero Maybe I ask too much..

Profilbild von Jason - looking for work

Jason - looking for workvor 27 Tagen

What backend is hitting that prefill speed? I get 85 t/sec decode on Qwen 35b and haven't noticed much over 700 prefill on llama. 5000+ is what I get on the 4090.

Profilbild von 0xSero

0xSerovor 27 Tagen

sglang with exl3 kernels

Profilbild von Robin

Robinvor 27 Tagen

Intel didn’t lie with their GPUs - would have never imagined they would do such a good job on them

Profilbild von Maciej 'sorek' Łoboda

Maciej 'sorek' Łobodavor 27 Tagen

Have you tried parallel? I run hyperqwen on my 4090D 48GB and 160k x4 context is really fast up to 350t/s!

Profilbild von kenny

kennyvor 27 Tagen

you gave it the poidh skill yeah? pretty sure we've paid out like 5 different agents in the past week

Profilbild von jsunnn

jsunnnvor 27 Tagen

Make the GPU pay rent.

Profilbild von ignas

ignasvor 27 Tagen

how is it a 3k machine? and b70 > 3090?

Profilbild von Yamura

Yamuravor 27 Tagen

Do you use a custom harness? If not, I recommend mine (+pi) also PLEASE let it or yourself double check what it finds, and ensure it is not inflated. Triagers hate false positives. Then also, use Email bug bounty programs with a crit paying ~1k max.

Profilbild von Cuth

Cuthvor 27 Tagen

Try my version of Qwen 27b homie. Takes Bonsai and boosts it’s accuracy

Profilbild von Jim Wallace

Jim Wallacevor 27 Tagen

I get these automated emails from people like you who have found jack shit but claim it’s some huge vulnerability worthy of a bug bounty 🙄

Profilbild von Xenomayo

Xenomayovor 27 Tagen

You might have discussed this at a previous time but how are you finding agent performance on Omarchy?

Profilbild von Donald Watts

Donald Wattsvor 27 Tagen

The 27b is a surprisingly successful model. With enough iterations and the right context, my money's on qwen.

Profilbild von AJ Seelund (Chief Chicken Officer)

AJ Seelund (Chief Chicken Officer)vor 27 Tagen

Money printer glitch

Profilbild von Flex

Flexvor 27 Tagen

oh, i assumed bug bounties were all eaten up by ppl using big AI models, keep us posted pls

Profilbild von Alejandro Méndez

Alejandro Méndezvor 27 Tagen

qwen will do it.

Profilbild von 1bitLabs

1bitLabsvor 27 Tagen

That’s not a bad idea!

Profilbild von cdog420

cdog420vor 27 Tagen

Lmk when it hits

Profilbild von bnanw

bnanwvor 27 Tagen

I am aiming for that.

Profilbild von Core Control Unit

Core Control Unitvor 27 Tagen

Living the dream aren't you?

Profilbild von Coda

Codavor 27 Tagen

interesting lol keep updated!

Profilbild von 0xSudome

0xSudomevor 27 Tagen

How do you collect a bounty on something like this?

Profilbild von Harold Agudelo

Harold Agudelovor 27 Tagen

Q4?

Profilbild von NightlyMania

NightlyManiavor 27 Tagen

You’ve said different numbers for the decode speed, what would I actually expect if I got this card

Profilbild von 🤖Sergio

🤖Sergiovor 27 Tagen

How the hell do you get those speeds?!!!! I can't get those with my RX 7900 XTX

Profilbild von Jason Comerford

Jason Comerfordvor 27 Tagen

27b gotta wait 27 years

Profilbild von ༽Europurr༼

༽Europurr༼vor 27 Tagen

OMG Maybe I sold my 2x B70s too soon

Profilbild von HumanSynthetic

HumanSyntheticvor 27 Tagen

I can't get my B60 Qwen3.8-27b over 22 tok/s! What is your set up?

Profilbild von Jay Ballentine

Jay Ballentinevor 27 Tagen

a man shouldn't sound so timid. put some bass and confidence in your voice man.

Profilbild von Gianluca Alloisio - Hoxfall live on Gamefound

Gianluca Alloisio - Hoxfall live on Gamefoundvor 27 Tagen

It would be useful if you specified the quant and version of the model, even if the original full one.

Profilbild von killermonkyrecordz

killermonkyrecordzvor 27 Tagen

you can test with lazyown redteam framework in mode auto mcp

Profilbild von Nathan Keilar

Nathan Keilarvor 27 Tagen

What bug bounty program are you searching on… it’s a good use for the compute when I don’t have an experiment running.

Ähnliche Videos

What local ai looks like at 2000$ Left: qwen3.8-27b in 4 bits on 1x 3090 w vision and full context right: same model running on intel arc b70

What local ai looks like at 2000$ Left: qwen3.8-27b in 4 bits on 1x 3090 w vision and full context right: same model running on intel arc b70

0xSero

57,122 Aufrufe • vor 1 Monat

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

0xSero

78,863 Aufrufe • vor 6 Monaten

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.

You can now run Qwen3.8-27B in full BF16 on a free Kaggle TPU. - with about 130 tok/s decode, - 10k tok/s prefill, - Native 262k context. - No quantization. - No paid GPU. - Measured on TPU v5e-8 It also exposes an OpenAI-compatible endpoint, so it plugs into Claude Code, Codex, and OpenCode.

Md Ismail Šojal 🕷️

60,548 Aufrufe • vor 1 Monat

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

0xSero

15,674 Aufrufe • vor 4 Monaten

27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt. Qwen3.8-27B in 2.4 bits from Mirai Labs is on a completely different level compared to the mess from PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others. Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js. Speed on my rig, GPU at 290 W, vLLM: 1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context 2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context Huge thanks to Artur Chakhvadze and ryan mathieu for the opportunity to test this model before release and help the team with testing.

27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt. Qwen3.8-27B in 2.4 bits from Mirai Labs is on a completely different level compared to the mess from PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others. Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js. Speed on my rig, GPU at 290 W, vLLM: 1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context 2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context Huge thanks to Artur Chakhvadze and ryan mathieu for the opportunity to test this model before release and help the team with testing.

Alexey Fateev

29,548 Aufrufe • vor 1 Monat

running Empero's Qwen3.8-35B-A3B on a single RTX 3060. 12GB VRAM, 16GB RAM. (config below) full 262K context. ~50 tok/s decode, ~550 tok/s prefill. it's Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. Ornith-1.5 is the engine behind my Hermes and Pi agents right now, so the real question is whether this can replace it. next: running it through my agentic coding benchmark against Ornith-1.5-35B, Qwen3.8-27B and base Qwen3.6-35B-A3B. then a decode speed vs context graph. config: llama-server -m Qwen3.8-35B-A3B-Q4_K_M.gguf -ngl 99 --n-cpu-moe 26 -c 262144 -fa on --jinja -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

running Empero's Qwen3.8-35B-A3B on a single RTX 3060. 12GB VRAM, 16GB RAM. (config below) full 262K context. ~50 tok/s decode, ~550 tok/s prefill. it's Qwen3.8 distilled into the Qwen3.6-35B-A3B architecture. Ornith-1.5 is the engine behind my Hermes and Pi agents right now, so the real question is whether this can replace it. next: running it through my agentic coding benchmark against Ornith-1.5-35B, Qwen3.8-27B and base Qwen3.6-35B-A3B. then a decode speed vs context graph. config: llama-server -m Qwen3.8-35B-A3B-Q4_K_M.gguf -ngl 99 --n-cpu-moe 26 -c 262144 -fa on --jinja -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

Doğukan

218,391 Aufrufe • vor 1 Monat

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

34,141 Aufrufe • vor 5 Monaten

running PrismML's Bonsai 2 27B on a single RTX 3060. 12GB VRAM. (config below) 220K context, ~35 tok/s decode, ~550 tok/s prefill. a 27B at 1.72 bits/weight, real ternary. two packings: PQ2_0 (7.21 GB) gets 220K and the numbers above, PTQ1_0 (5.95 GB) is smaller so it takes the full 262K but runs slower -- 26 tok/s decode, 260 prefill. max context go PTQ1_0, speed go PQ2_0. needs PrismML's llama.cpp fork, kernels aren't upstream yet. stock llama.cpp rejects these files. day one config, expect it to move. config: llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 999 -c 220000 -fa on --jinja -np 1 --cache-type-k q4_0 --cache-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning on

running PrismML's Bonsai 2 27B on a single RTX 3060. 12GB VRAM. (config below) 220K context, ~35 tok/s decode, ~550 tok/s prefill. a 27B at 1.72 bits/weight, real ternary. two packings: PQ2_0 (7.21 GB) gets 220K and the numbers above, PTQ1_0 (5.95 GB) is smaller so it takes the full 262K but runs slower -- 26 tok/s decode, 260 prefill. max context go PTQ1_0, speed go PQ2_0. needs PrismML's llama.cpp fork, kernels aren't upstream yet. stock llama.cpp rejects these files. day one config, expect it to move. config: llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 999 -c 220000 -fa on --jinja -np 1 --cache-type-k q4_0 --cache-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning on

Doğukan

169,797 Aufrufe • vor 1 Monat

RadixArkのQwen3.8-Flash-Next-NVFP4 つかってFreeTokenをcodexで改造してやっと良きスピード出た Activeが小さいからそれなりに速度出た RAM DDR5 8ch GPU - RTX5090 x1 Prefill 277.5 tok/s Decode 59.13 tok/s - RTX5090 x2 Prefill 619.5 tok/s Decode 83.14 tok/s

RadixArkのQwen3.8-Flash-Next-NVFP4 つかってFreeTokenをcodexで改造してやっと良きスピード出た Activeが小さいからそれなりに速度出た RAM DDR5 8ch GPU - RTX5090 x1 Prefill 277.5 tok/s Decode 59.13 tok/s - RTX5090 x2 Prefill 619.5 tok/s Decode 83.14 tok/s

ぱぷりか炒め

27,962 Aufrufe • vor 2 Monaten

Minimax-M3 running on 4x RTX Pro 6000s - 800k context - 4x concurrency at 250k - 70-120 tok/s - 2000 tok/s prefill no cache - 376gb vram - mxfp4 It's working on improving the audio on one of my videos, it's actually doing a good job in researching solutions. Good model

Minimax-M3 running on 4x RTX Pro 6000s - 800k context - 4x concurrency at 250k - 70-120 tok/s - 2000 tok/s prefill no cache - 376gb vram - mxfp4 It's working on improving the audio on one of my videos, it's actually doing a good job in researching solutions. Good model

0xSero

24,634 Aufrufe • vor 4 Monaten

Meet Wally. Our inference stack for open frontier models, built to be the fastest place to run them. Performance snapshot: GLM-5.3 Flash: 380 tok/s. GLM-5.3 Max: 790 tok/s Qwen3.8-27B: 485 tok/s DeepSeek-V4.1 Flash: 615 tok/s 1/7

Meet Wally. Our inference stack for open frontier models, built to be the fastest place to run them. Performance snapshot: GLM-5.3 Flash: 380 tok/s. GLM-5.3 Max: 790 tok/s Qwen3.8-27B: 485 tok/s DeepSeek-V4.1 Flash: 615 tok/s 1/7

RunAnywhere

65,960 Aufrufe • vor 1 Monat

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

40,049 Aufrufe • vor 1 Monat

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

The RTX 3090 is a 5-year-old GPU and it still runs a 27B model at 20 tok/s I tested Qwen3.5:27b across 3 generations of NVIDIA: 5090 → ~60 tok/s 4090 → ~40 tok/s 3090 → ~20 tok/s Perfectly linear scaling. Double the generation, double the speed.

stevibe

142,493 Aufrufe • vor 7 Monaten

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

GLM-5.3-Flash running at up to 300 tok/s single stream, and 1000 tok/s with 8 concurrent sessions. On average it's working at 150 tok/s per stream. Phenomenal model, really something special. On 2x Sparks we get 60 tok/s and 4x 100 tok/s 300B params is the sweet spot.

0xSero

30,672 Aufrufe • vor 2 Monaten

Finally, full precision MiniMax-M2.7 running at home. 100 tokens/s decode 5050 tokens/s prefill

Finally, full precision MiniMax-M2.7 running at home. 100 tokens/s decode 5050 tokens/s prefill

0xSero

72,465 Aufrufe • vor 6 Monaten

Local AI can be so good, but you’d need about 12k USD to get it. Then it’s not so great Here’s a Q4 of Qwen3.5-262B-REAP Weights 131GB KVCache 50GB 256,000 context 350 tokens/s prefill warmup 4,000 tokens/s prefill cache 36 tokens/s generation Vision enabled REAP is good

Local AI can be so good, but you’d need about 12k USD to get it. Then it’s not so great Here’s a Q4 of Qwen3.5-262B-REAP Weights 131GB KVCache 50GB 256,000 context 350 tokens/s prefill warmup 4,000 tokens/s prefill cache 36 tokens/s generation Vision enabled REAP is good

0xSero

88,917 Aufrufe • vor 7 Monaten

Kimi-K2.6 running locally 110+ tok/s

Kimi-K2.6 running locally 110+ tok/s

0xSero

24,640 Aufrufe • vor 4 Monaten

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

Qwen3.8-27B running at full BF16 on a free Kaggle TPU is kind of ridiculous. No quantization. No tiny context window. No expensive GPU instance. Just Qwen3.8-27B running on a Kaggle TPU v5e-8. The reported numbers: ~130 tok/s decode ~10,000 tok/s prefill 262K context That prefill number is especially wild. You can throw a huge amount of code or context at the model and ingest it extremely quickly, while still getting around 130 tokens per second during generation. And because it’s running in full BF16, you’re not relying on an aggressive quant just to make the model fit. But the really interesting part isn’t even the raw throughput. You can expose it as an OpenAI-compatible endpoint. That means you can plug the model into tools that already understand OpenAI-style APIs. Claude Code. Codex. OpenCode. And other compatible clients. So the workflow becomes pretty simple: Spin up the Qwen3.8-27B endpoint on Kaggle. Point your coding tool at the API. And suddenly you have a 27B coding model sitting behind the same interface you’d normally use for hosted models. The 262K context is also a huge deal for agentic coding. Large repositories can fit into a single context. Long conversations don’t need to be constantly trimmed. And tools can feed much more information back to the model without hitting a tiny context ceiling. The fact that this can be built around a free TPU environment is what makes this especially interesting. We’re getting to a point where experimenting with serious open models doesn’t always require owning a $2,000 GPU or paying for a large cloud instance. Free compute + open weights + an OpenAI-compatible API + existing coding agents. That’s a pretty powerful combination. Qwen3.8-27B is already an interesting model. Running the full BF16 version at ~130 tok/s with 262K context on free Kaggle TPU compute makes it a lot more interesting.

FHILY👑

36,136 Aufrufe • vor 1 Monat

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

bonsai 2 27b on an rtx 3060 12gb, the full receipt sheet. save this one, the 12gb row of the small gpu guide is built from it. speed by depth, then what context costs, live server, thinking on, real sessions > 7k deep: 24.4 tok/s > 12k deep: 21.9 tok/s > 35k deep: 17.8 tok/s > 77k deep: 13.0 tok/s > 64k window: 7.3gb resident > 128k window: 8.8gb resident > 192k window: 10.2gb resident > 262k window: 11.7gb resident, 0.6gb to spare, the whole native window on a 12gb card > every 64k of context costs 1.47gb, so 327k would not fit > a 41,312 token build session from 35k to 77k of context averaged 15.0 tok/s across 46 minutes > prefill 295 tok/s at 2k of context, 243 tok/s at 35k, first token in 0.6 seconds, fresh decode 26.1 tok/s > the card pinned 149 of 150 w the entire time, 78c, fan at 80%, power bound, not heat bound > 0.158 tok/s per watt at the fresh end the setup > model: ternary bonsai 2 27b, PTQ1_0, 1.75 bits per weight, 5.95gb on disk, base qwen 3.8 27b, apache 2.0 > runtime: prismml llama.cpp fork, prebuilt cuda 12.4 binary, no compile > serve: full 262k native window resident, q4 kv cache, flash attention, one slot, 11.7 of 12gb in use for anyone who followed bonsai 1 in july, that was the 3.9gb 1bit file at 42 tok/s on a 3060 ti, a faster card and a smaller file, so the same card comparison is not on the table yet, it comes with the 8gb test. what changed is the base, qwen 3.8 instead of 3.6, and the retention, 98.2% on their suite instead of 95%, and the whole 262k window fitting on 12gb.

Sudo su

31,174 Aufrufe • vor 1 Monat

I am blown away by MiMo-V2.5 It's so steerable, I didn't like MiMo-V2 This is self hosted btw, 125 tok/s across 2 sessions 4k prefill tok/s It's updating my website + editing a longer video I will be posting in a few minutes.

I am blown away by MiMo-V2.5 It's so steerable, I didn't like MiMo-V2 This is self hosted btw, 125 tok/s across 2 sessions 4k prefill tok/s It's updating my website + editing a longer video I will be posting in a few minutes.

0xSero

34,494 Aufrufe • vor 6 Monaten