正在加载视频...

视频加载失败

返回首页

the timeline spent two days saying bonsai 2 cannot build. here is 1 hour 24 minutes of it building, one shot, from one paragraph, on an rtx 3060 12gb, sped to 8x so you can watch the whole thing. what you are watching is a 5.9gb ternary compression of...

sudoingX's profile picture

Sudo su

36,275 subscribers

41,330 次观看 • 2 天前 •via X (Twitter)

教育科学技术
LIVE

Anya Rossi• Live Now

Private livecam show

18 条评论

Sudo su 的头像
Sudo su2 天前

the exact serve, copy it as is. prism fork prism-b10685 or newer, prebuilt cuda binaries on the releases page, no compile. model: (the PTQ1_0 file, 5.95gb) fork: llama-server -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 262144 -np 1 -ctk q4_0 -ctv q4_0 --jinja --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8899 -c 262144 is the full window and needs 12gb, drop to 65536 on an 8gb card. -np 1 keeps one slot, four slots cost 450mb for nothing. --jinja is what makes tool calls work. temp 1.0, top-p 0.95, top-k 20 are their thinking defaults. the web ui is at localhost:8899 once it is up, that is what the video shows.

Sudo su 的头像
Sudo su2 天前

the exact prompt, one paragraph, so you can run the same test on your card and compare: "build me a single html file, gpu-monitor.html, a terminal styled gpu monitoring card for one node, the kind of thing nvidia-smi and nvtop show but as a clean ui card. one card per gpu, start with one: NVIDIA GeForce RTX 3060 12GB. the card shows the gpu name and driver version, temperature with a colored bar, fan percent, gpu utilization percent, vram used over total as a bar with the numbers, power draw over limit as a bar, sm clock and memory clock, pcie gen and lanes, and three small sparklines under the bars for utilization, vram and power over the last 60 seconds. below the card a small process table with pid, type, gpu mem and command, two rows, llama-server and hermes-agent. no real nvidia-smi calls, simulate the data client side, update every second with realistic fluctuation, and let the sparklines scroll. aesthetic: dark near-black background, monospace everything, terminal green for labels and borders, orange for the live numbers, red when a metric passes 85 percent of its max, and a thin header bar that reads NODE MONITOR · @sudoingX. vanilla html, css and js only, no libraries, everything inline in the one file, it must open straight from the file with no server and no console errors. make it look good enough to screenshot."

Sudo su 的头像
Sudo su2 天前

every number in the video comes from the 12gb receipt sheet, memory ladder, decode by depth, prefill, power, all measured on this serve:

matt 的头像
matt2 天前

1h24 for a gpu monitor html though

Sudo su 的头像
Sudo su2 天前

most of that is thinking, 76k tokens of it, at 26 tok/s falling to 13 tok/s as the context fills, on a 150 w card. a human does not design and ship that page in 84 minutes either. quality is the part that surprised me, speed is the part i am working on, the mtp head graft and a kernel fix are on the bench this week, that is where the 2x comes from.

Emilio^ 的头像
Emilio^2 天前

Mine gets perpetually stuck, even with thinking budgets and dry penalties

Sudo su 的头像
Sudo su2 天前

mine just keeps going, 76k tokens without a stall. two things that cause the stuck loop in my runs: the stock llama.cpp build (it loads the file and talks nonsense, you need the prismml fork) and fighting the sampler. drop the dry and rep penalties, run their thinking defaults, temp 1.0, top-p 0.95, top-k 20, presence 0, with --jinja on. the exact serve line is in my replies above.

B effect.iota 🦋🐝 的头像
B effect.iota 🦋🐝2 天前

All it takes is getting the setup right 🎯

Chris Geeman 的头像
Chris Geeman2 天前

Hell ya!

Badanzer 的头像
Badanzer2 天前

12GB is still where a lot of real desks live. Does ternary actually keep you from swapping when you push batch size?

Spok 的头像
Spok2 天前

Yeh, I my tests it also was a useful model, but not 98% of original one

La Voix du Chat Artiste 的头像
La Voix du Chat Artiste1 天前

Thanks i was waiting for this

David Walker 的头像
David Walker2 天前

People are saying that the Q2 model at full context precision is having quality issues. Tool calling works, but the actual output is noticeably worse than the full model. I can’t imagine what shenanigans Q1 and Q4 kv cache must do. I’d go with Q2 and Q8 kv cache minimum

EDDY VU 的头像
EDDY VU1 天前

Keeping that level of coherence across an hour and a half on a single 3060 is a massive win for ternary compression.

Fratrilogos 的头像
Fratrilogos2 天前

Some of us do not find a terminal gpu monitor particularly impressive.

basedcapital 的头像
basedcapital2 天前

the cache cannot be what halves the decode. 4 bit kv on 16 of 64 layers is about 16kb a token, so at 76k it is 1.2gb against 7.2gb of weights.

Ericreator 的头像
Ericreator2 天前

way slower than expected

Empyrean 的头像
Empyrean1 天前

Are you building anything real or just AI theatre? Genuinly curious! Thanks.

相关视频

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.
0:26

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.

Sudo su

23,453 次观看 • 1 个月前

Darkbloom is the first model provider to support Ternary Bonsai 2 27B -- concentrated intelligence that fits on your phone. Try it now: First 250 users get 100 million free tokens; PrismML's new flagship model: - a ternary compression of Qwen3.8 27B at 2bits per parameter. - 8.5 GB total, 5x smaller than original - keeps 98.2% of the Qwen's FP16 benchmark performance. - 75% cheaper than Qwen 27B. Qwen3.8 27B already operates comparably with Opus 4.6 and 5.6 Luna on certain tasks. This one does it in the memory of a phone. 262K context, image input, Apache 2.0. From our first run on the network, on a single M5 Max with no caching: - 35 tok/s decode at 1K context, - 31 tok/s at 10K, - 19 tok/s at 50K. But we expect more performance gain coming in a few weeks! That's a full 27B reasoning model running comfortably on any Mac. 1,000+ Macs are serving on Darkbloom right now. Go try it out!! Thank you to Babak Hassibi Sahin Lale Omead Pooladzandi Reza Sadri Tushar Bansal Evan Walters and the whole PrismML team. This is exactly the kind of model Darkbloom was built for. You can read the essay by Bonsai on Why Local AI Matters:
1:20

Darkbloom is the first model provider to support Ternary Bonsai 2 27B -- concentrated intelligence that fits on your phone. Try it now: First 250 users get 100 million free tokens; PrismML's new flagship model: - a ternary compression of Qwen3.8 27B at 2bits per parameter. - 8.5 GB total, 5x smaller than original - keeps 98.2% of the Qwen's FP16 benchmark performance. - 75% cheaper than Qwen 27B. Qwen3.8 27B already operates comparably with Opus 4.6 and 5.6 Luna on certain tasks. This one does it in the memory of a phone. 262K context, image input, Apache 2.0. From our first run on the network, on a single M5 Max with no caching: - 35 tok/s decode at 1K context, - 31 tok/s at 10K, - 19 tok/s at 50K. But we expect more performance gain coming in a few weeks! That's a full 27B reasoning model running comfortably on any Mac. 1,000+ Macs are serving on Darkbloom right now. Go try it out!! Thank you to Babak Hassibi Sahin Lale Omead Pooladzandi Reza Sadri Tushar Bansal Evan Walters and the whole PrismML team. This is exactly the kind of model Darkbloom was built for. You can read the essay by Bonsai on Why Local AI Matters:

Gajesh

45,318 次观看 • 2 天前

i measured qwen 3.8 27b dense, raw on a single RTX 3090 24gb GPU, before the hype finished loading. here is the honest read: 25.4 tok/s on the exact card and test where 3.6 ran 40tok/s. slower on day one, and i'm posting it anyway, because the rest of the data says watch this space. decode holds dead flat as context fills, 25 empty, 26 at 90k deep, no 27b has ever done that on my bench. one turn in the video generated 33,614 tokens and averaged 22.9 the whole way. the full 262k window sits resident in 22.2GB. the vision tower boots and the card genuinely sees. and the server logs show llama.cpp ignoring the model's MTP tensors completely, multi-token prediction trained into the weights and not wired up yet. this is the floor, measured before the optimizers woke up. tomorrow there are builders wiring MTP, tuning the hybrid attention, shipping kernels, and every commit is free speed for the same card. i benched day one so we can all watch it climb. build video, receipts, full chart in the gallery. repro below.
1:33

i measured qwen 3.8 27b dense, raw on a single RTX 3090 24gb GPU, before the hype finished loading. here is the honest read: 25.4 tok/s on the exact card and test where 3.6 ran 40tok/s. slower on day one, and i'm posting it anyway, because the rest of the data says watch this space. decode holds dead flat as context fills, 25 empty, 26 at 90k deep, no 27b has ever done that on my bench. one turn in the video generated 33,614 tokens and averaged 22.9 the whole way. the full 262k window sits resident in 22.2GB. the vision tower boots and the card genuinely sees. and the server logs show llama.cpp ignoring the model's MTP tensors completely, multi-token prediction trained into the weights and not wired up yet. this is the floor, measured before the optimizers woke up. tomorrow there are builders wiring MTP, tuning the hybrid attention, shipping kernels, and every commit is free speed for the same card. i benched day one so we can all watch it climb. build video, receipts, full chart in the gallery. repro below.

Sudo su

48,648 次观看 • 1 个月前

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.
0:21

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 次观看 • 2 个月前

watch bonsai 3.9gb model crush tool calls. this is running in hermes agent locally, and it doesn't just fire blind, it reasons first before every single action, thinks through what it actually needs, then makes the call. zero failures the whole run, not one dropped, not one malformed, across a full multi step agent loop. for anyone new here, bonsai is a 1 bit quant of qwen 3.6 27b dense, the exact model i spent all month calling the king of the 3090. PrismML took the king, crushed it down to a single sign bit per weight, 3.9gb, small enough to sit next to a gemma 12b, and it still holds an agent loop and reasons before it acts. sub 4 bit is where models are supposed to turn to mush. someone even clocked it at tool-calling 0 out of 100. not in my runs. this is the warm-up. the real test is the build gauntlet my q4 27b nailed, elements, particle swarm, mandelbrot, and openclaw runs the same one right after. we find out if 1bit actually keeps up.
0:30

watch bonsai 3.9gb model crush tool calls. this is running in hermes agent locally, and it doesn't just fire blind, it reasons first before every single action, thinks through what it actually needs, then makes the call. zero failures the whole run, not one dropped, not one malformed, across a full multi step agent loop. for anyone new here, bonsai is a 1 bit quant of qwen 3.6 27b dense, the exact model i spent all month calling the king of the 3090. PrismML took the king, crushed it down to a single sign bit per weight, 3.9gb, small enough to sit next to a gemma 12b, and it still holds an agent loop and reasons before it acts. sub 4 bit is where models are supposed to turn to mush. someone even clocked it at tool-calling 0 out of 100. not in my runs. this is the warm-up. the real test is the build gauntlet my q4 27b nailed, elements, particle swarm, mandelbrot, and openclaw runs the same one right after. we find out if 1bit actually keeps up.

Sudo su

55,317 次观看 • 2 个月前

running PrismML's Bonsai 2 27B on a single RTX 3060. 12GB VRAM. (config below) 220K context, ~35 tok/s decode, ~550 tok/s prefill. a 27B at 1.72 bits/weight, real ternary. two packings: PQ2_0 (7.21 GB) gets 220K and the numbers above, PTQ1_0 (5.95 GB) is smaller so it takes the full 262K but runs slower -- 26 tok/s decode, 260 prefill. max context go PTQ1_0, speed go PQ2_0. needs PrismML's llama.cpp fork, kernels aren't upstream yet. stock llama.cpp rejects these files. day one config, expect it to move. config: llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 999 -c 220000 -fa on --jinja -np 1 --cache-type-k q4_0 --cache-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning on
1:38

running PrismML's Bonsai 2 27B on a single RTX 3060. 12GB VRAM. (config below) 220K context, ~35 tok/s decode, ~550 tok/s prefill. a 27B at 1.72 bits/weight, real ternary. two packings: PQ2_0 (7.21 GB) gets 220K and the numbers above, PTQ1_0 (5.95 GB) is smaller so it takes the full 262K but runs slower -- 26 tok/s decode, 260 prefill. max context go PTQ1_0, speed go PQ2_0. needs PrismML's llama.cpp fork, kernels aren't upstream yet. stock llama.cpp rejects these files. day one config, expect it to move. config: llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 999 -c 220000 -fa on --jinja -np 1 --cache-type-k q4_0 --cache-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning on

Doğukan

152,249 次观看 • 3 天前

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.
0:20

125B on a single RTX 4090 at 25 tok/s is actually insane Qwen3.8-Flash-Next + MTP is hitting: 80K context 25.35 tok/s decode 471 tok/s prefill On one consumer GPU. Yes, it’s an MoE, so you’re not running all 125B parameters for every token. But still, getting a model this large to run this fast on a 4090 is wild. MTP is doing a lot of the heavy lifting here too. Instead of generating one token at a time, it can speculate on multiple tokens and verify them together. And 25 tok/s is the important number.

FHILY👑

39,811 次观看 • 18 天前

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.
0:27

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

RunAnywhere (YC W26)

17,587 次观看 • 2 个月前

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below
2:28

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

Antid

13,974 次观看 • 26 天前

update: qwen 3.6 27b dense q4 just one shotted octopus invaders game on a single 3090. hermes agent drove the whole thing, ~41 tok/s gen 21gb vram at full 262k context, thinking mode on. one prompt in and the canonical multi-file space shooter benchmark out, the same exact prompt i ran on qwen 3.5 27b dense back in march on the same card. 3.5 needed one external scope bug fix before the game would even load on first play. 3.6 needed nothing. 11 of 11 files written, 2411 lines of code, zero steering interventions, zero external fixes, playable on first load. 16 minutes 41 seconds wall clock from prompt to playable. consumer tier king on a single 3090 is locked tonight, and the silicon underneath my desk did not change between march and now. the open source ecosystem just moved the floor. watch it ship itself, the full 16 minutes 41 seconds sped to 3 minutes 45, no human touched the keyboard between the first prompt and the final frame.
3:45

update: qwen 3.6 27b dense q4 just one shotted octopus invaders game on a single 3090. hermes agent drove the whole thing, ~41 tok/s gen 21gb vram at full 262k context, thinking mode on. one prompt in and the canonical multi-file space shooter benchmark out, the same exact prompt i ran on qwen 3.5 27b dense back in march on the same card. 3.5 needed one external scope bug fix before the game would even load on first play. 3.6 needed nothing. 11 of 11 files written, 2411 lines of code, zero steering interventions, zero external fixes, playable on first load. 16 minutes 41 seconds wall clock from prompt to playable. consumer tier king on a single 3090 is locked tonight, and the silicon underneath my desk did not change between march and now. the open source ecosystem just moved the floor. watch it ship itself, the full 16 minutes 41 seconds sped to 3 minutes 45, no human touched the keyboard between the first prompt and the final frame.

Sudo su

126,670 次观看 • 4 个月前

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.
1:39

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.

Sudo su

29,584 次观看 • 4 个月前

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!
4:28

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!

Kyle Hessling

24,917 次观看 • 1 个月前

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.
4:15

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

Sudo su

39,764 次观看 • 3 个月前

ok this is wild. 10 year old gtx 1080 8gb pascal card running qwen3 8b locally at 18-20 tok/s via hermes agent and it's actually doing the thing. asked it to build a wireworld cellular automata simulator with 10 tests. autonomous run, no hand holding. expected it to fail on the tool calls. that's not what happened. write_file works. browser_navigate works. terminal commands work. file ops, package installs, version probes, environment setup. agent is firing tool calls cleanly and the model is reasoning about next steps at 18-20 tok/s. on hardware that pre dates "agentic" as a word. it even hit an npm install fail because node 12 is too old. didn't crash. didn't ask me. just started bootstrapping nvm on its own to fix the environment. 10 minutes in. 40% context used. 7.5gb of 8gb vram occupied. still going. i did not think this would work on this hardware. this is the most i've been wrong this month.
0:23

ok this is wild. 10 year old gtx 1080 8gb pascal card running qwen3 8b locally at 18-20 tok/s via hermes agent and it's actually doing the thing. asked it to build a wireworld cellular automata simulator with 10 tests. autonomous run, no hand holding. expected it to fail on the tool calls. that's not what happened. write_file works. browser_navigate works. terminal commands work. file ops, package installs, version probes, environment setup. agent is firing tool calls cleanly and the model is reasoning about next steps at 18-20 tok/s. on hardware that pre dates "agentic" as a word. it even hit an npm install fail because node 12 is too old. didn't crash. didn't ask me. just started bootstrapping nvm on its own to fix the environment. 10 minutes in. 40% context used. 7.5gb of 8gb vram occupied. still going. i did not think this would work on this hardware. this is the most i've been wrong this month.

Sudo su

92,585 次观看 • 3 个月前

ANTHROPIC LEAKED A SETUP WHERE A $1.2M A-YEAR TEAM FITS INTO ONE FILE FOR $4 A DAY you hire nobody - you write one file, and $106,500 of salaries in it turn into $4 of tokens. architect → 4 builders → tests → critic → docs → scout → chief → back into the file the architect runs twice a day and decides everything - the priciest model on the sheet is the smallest line on the bill at $1.05. four builders cost $1.20 a day - and a fifth builder is one more number in the file, not one more quarter of recruiting. the tester asks does it run, the critic asks should it exist at all - $1.22 together for two roles a solo dev never fills. nothing reaches you that hasn't been rejected at least once. the scout and the scribe cost 40 cents - the exact two roles that get cut from every real budget. nobody hires a documentation engineer, and everybody wishes they had one. the chief at 14 cents is the only one who writes back into the file - without it this is a list, with it the list edits itself. hiring the eleventh is a paragraph: no req, no interviews, no ramp - and firing is one command. save this and paste it into Claude Code - and you get 10 roles for the price of one coffee ↓
0:41

ANTHROPIC LEAKED A SETUP WHERE A $1.2M A-YEAR TEAM FITS INTO ONE FILE FOR $4 A DAY you hire nobody - you write one file, and $106,500 of salaries in it turn into $4 of tokens. architect → 4 builders → tests → critic → docs → scout → chief → back into the file the architect runs twice a day and decides everything - the priciest model on the sheet is the smallest line on the bill at $1.05. four builders cost $1.20 a day - and a fifth builder is one more number in the file, not one more quarter of recruiting. the tester asks does it run, the critic asks should it exist at all - $1.22 together for two roles a solo dev never fills. nothing reaches you that hasn't been rejected at least once. the scout and the scribe cost 40 cents - the exact two roles that get cut from every real budget. nobody hires a documentation engineer, and everybody wishes they had one. the chief at 14 cents is the only one who writes back into the file - without it this is a list, with it the list edits itself. hiring the eleventh is a paragraph: no req, no interviews, no ramp - and firing is one command. save this and paste it into Claude Code - and you get 10 roles for the price of one coffee ↓

Sprytix

47,679 次观看 • 1 个月前

Kimi K3 dropped 3 days ago and i already used it to build my own PUBG the model that just took the number one coding spot on earth and it runs on a free tier no team no engine i just fed it free open models and assets that are already public and told it what i wanted it built the entire game itself the shooting the recoil the hit detection all of it came straight out of the model i barely wrote a line what youre watching here is the deathmatch mode and its already live battle royale is next im deep in it and the graphics get reworked right after i know it looks raw right now i wont pretend otherwise but the endgame is a build that runs perfect on phones and the moment its on mobile you drop ads in and it earns on its own the part that still gets me is my whole cost for this was a single subscription
0:22

Kimi K3 dropped 3 days ago and i already used it to build my own PUBG the model that just took the number one coding spot on earth and it runs on a free tier no team no engine i just fed it free open models and assets that are already public and told it what i wanted it built the entire game itself the shooting the recoil the hit detection all of it came straight out of the model i barely wrote a line what youre watching here is the deathmatch mode and its already live battle royale is next im deep in it and the graphics get reworked right after i know it looks raw right now i wont pretend otherwise but the endgame is a build that runs perfect on phones and the moment its on mobile you drop ads in and it earns on its own the part that still gets me is my whole cost for this was a single subscription

savip.

68,984 次观看 • 2 个月前

98.2% of Qwen3.8 27B my ass. I got hyped and gave it a real agent job right away. Build me an FPS in three.js, 6 hours on a 3090. My most standard and default prompt that I always use. It spent the first 32K tokens on a plan without writing a single file, then shipped a black screen, 2 shaders that dont compile and a player who spawns dead. And wrote "verified" in the final report. Ok, too hard. I gave it the easiest thing I have, a voxel pagoda garden in one html file. Video attached. 3 hours for THIS. On the way it deleted its own file and spent an hour debugging a raycaster nobody asked for. Where the 98.2% is in all this I have no fucking idea. Same Qwen3.8 27B, same pagoda task, ISTA-DASLab GSQ-RCO IQ2_XS on a 12 GB 3080 Ti gave me day and night, real shadows and koi fish in the pond. 8.4 GB on disk, real 2.50 bpw, 131072 ctx with q4_0 KV, 47 tok/s at 128K. 2.5 bits beats 2.13 bits by a lot when the 2.13 is this shit. Post with the video and the exact command in the replies. In the replies, I'll attach what a proper Qwen3.8 27B created in my hands. Bottom line: if you have at least 12GB VRAM, use ISTA-DASLab GSQ-RCO of all THIS.
0:42

98.2% of Qwen3.8 27B my ass. I got hyped and gave it a real agent job right away. Build me an FPS in three.js, 6 hours on a 3090. My most standard and default prompt that I always use. It spent the first 32K tokens on a plan without writing a single file, then shipped a black screen, 2 shaders that dont compile and a player who spawns dead. And wrote "verified" in the final report. Ok, too hard. I gave it the easiest thing I have, a voxel pagoda garden in one html file. Video attached. 3 hours for THIS. On the way it deleted its own file and spent an hour debugging a raycaster nobody asked for. Where the 98.2% is in all this I have no fucking idea. Same Qwen3.8 27B, same pagoda task, ISTA-DASLab GSQ-RCO IQ2_XS on a 12 GB 3080 Ti gave me day and night, real shadows and koi fish in the pond. 8.4 GB on disk, real 2.50 bpw, 131072 ctx with q4_0 KV, 47 tok/s at 128K. 2.5 bits beats 2.13 bits by a lot when the 2.13 is this shit. Post with the video and the exact command in the replies. In the replies, I'll attach what a proper Qwen3.8 27B created in my hands. Bottom line: if you have at least 12GB VRAM, use ISTA-DASLab GSQ-RCO of all THIS.

Alexey Fateev

343,276 次观看 • 2 天前

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

343,270 次观看 • 29 天前

first impressions of qwen 3.5 27B dense on a single RTX 3090. 35 tok/s. from 4K all the way to 300K+ context. no speed drop. hermes 4.3 started at 35 and degraded to 15 as context filled. qwen dense holds. MoE held 112 flat. 3x faster but only 3B of 35B active per token. architecture tradeoff. Q4_K_M on 16.7GB. native context 262K. pushed past training limit to 376K before VRAM ceiling on 24GB. tried q8 KV cache at 262K, speed collapsed to 11 tok/s. q4_0 KV is the sweet spot. flash attention mandatory. built in reasoning mode. the model thinks step by step before it answers. full chain of thought surviving Q4 quant. 1,799+ token thinking chains with self correction loops. on a single consumer GPU. gave it one prompt: "build a realtime particle galaxy simulation in one HTML file." 3,340 tokens. 95 seconds. one shot. ran on first load. full reasoning and coding in the video below. optimal config if you want to skip the hours of testing: llama-server -ngl 99 -c 262144 -fa on --cache-type-k q4_0 --cache-type-v q4_0 this is just the warmup. octopus invaders is next: 10 files, 3,400+ lines, zero steering. the prompt hermes quit at 22%. already more impressed than expected. full results coming soon.
2:02

first impressions of qwen 3.5 27B dense on a single RTX 3090. 35 tok/s. from 4K all the way to 300K+ context. no speed drop. hermes 4.3 started at 35 and degraded to 15 as context filled. qwen dense holds. MoE held 112 flat. 3x faster but only 3B of 35B active per token. architecture tradeoff. Q4_K_M on 16.7GB. native context 262K. pushed past training limit to 376K before VRAM ceiling on 24GB. tried q8 KV cache at 262K, speed collapsed to 11 tok/s. q4_0 KV is the sweet spot. flash attention mandatory. built in reasoning mode. the model thinks step by step before it answers. full chain of thought surviving Q4 quant. 1,799+ token thinking chains with self correction loops. on a single consumer GPU. gave it one prompt: "build a realtime particle galaxy simulation in one HTML file." 3,340 tokens. 95 seconds. one shot. ran on first load. full reasoning and coding in the video below. optimal config if you want to skip the hours of testing: llama-server -ngl 99 -c 262144 -fa on --cache-type-k q4_0 --cache-type-v q4_0 this is just the warmup. octopus invaders is next: 10 files, 3,400+ lines, zero steering. the prompt hermes quit at 22%. already more impressed than expected. full results coming soon.

Sudo su

122,051 次观看 • 6 个月前

Someone bought a Mac Mini for local AI. 64GB of unified memory seemed like the obvious choice. More memory. Bigger models. No VRAM limits. Then they put it against an RTX 3060. And things got weird. The Mac Mini M4 Pro pushed Qwen 2.5 at 38 tok/s. The old RTX 3060 with just 12GB VRAM hit 52 tok/s. That’s the interesting part about local AI. Apple and NVIDIA are solving different problems. Apple gives you a huge pool of memory. NVIDIA gives you dedicated compute and CUDA. One helps you fit bigger models. The other can run smaller models faster. Memory gets the model in. Compute gets the tokens out.
1:11

Someone bought a Mac Mini for local AI. 64GB of unified memory seemed like the obvious choice. More memory. Bigger models. No VRAM limits. Then they put it against an RTX 3060. And things got weird. The Mac Mini M4 Pro pushed Qwen 2.5 at 38 tok/s. The old RTX 3060 with just 12GB VRAM hit 52 tok/s. That’s the interesting part about local AI. Apple and NVIDIA are solving different problems. Apple gives you a huge pool of memory. NVIDIA gives you dedicated compute and CUDA. One helps you fit bigger models. The other can run smaller models faster. Memory gets the model in. Compute gets the tokens out.

KyzoroX

26,269 次观看 • 1 个月前