Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those...

sudoingX's profile picture

Sudo su

35,364 subscribers

23,453 görüntüleme • 1 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

watch bonsai 3.9gb model crush tool calls. this is running in hermes agent locally, and it doesn't just fire blind, it reasons first before every single action, thinks through what it actually needs, then makes the call. zero failures the whole run, not one dropped, not one malformed, across a full multi step agent loop. for anyone new here, bonsai is a 1 bit quant of qwen 3.6 27b dense, the exact model i spent all month calling the king of the 3090. PrismML took the king, crushed it down to a single sign bit per weight, 3.9gb, small enough to sit next to a gemma 12b, and it still holds an agent loop and reasons before it acts. sub 4 bit is where models are supposed to turn to mush. someone even clocked it at tool-calling 0 out of 100. not in my runs. this is the warm-up. the real test is the build gauntlet my q4 27b nailed, elements, particle swarm, mandelbrot, and openclaw runs the same one right after. we find out if 1bit actually keeps up.
0:30

watch bonsai 3.9gb model crush tool calls. this is running in hermes agent locally, and it doesn't just fire blind, it reasons first before every single action, thinks through what it actually needs, then makes the call. zero failures the whole run, not one dropped, not one malformed, across a full multi step agent loop. for anyone new here, bonsai is a 1 bit quant of qwen 3.6 27b dense, the exact model i spent all month calling the king of the 3090. PrismML took the king, crushed it down to a single sign bit per weight, 3.9gb, small enough to sit next to a gemma 12b, and it still holds an agent loop and reasons before it acts. sub 4 bit is where models are supposed to turn to mush. someone even clocked it at tool-calling 0 out of 100. not in my runs. this is the warm-up. the real test is the build gauntlet my q4 27b nailed, elements, particle swarm, mandelbrot, and openclaw runs the same one right after. we find out if 1bit actually keeps up.

Sudo su

55,317 görüntüleme • 1 ay önce

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.
1:39

look anon, those of you who kept saying local AI is not there yet, who said open source can't compete, who said you need cloud APIs to get anything serious done, look at this gameplay for one minute. every pixel on this screen was written by one model, in one shot, on a single rtx 3090 with 24gb of vram. the model is qwen 3.6 27b dense q4. the harness is hermes agent. the hardware is a single consumer card you can buy used for 900 dollars. the prompt is open source on github. every claim verifiable, on your own desk. if your local AI take is from 2024, update it. the consumer tier is shipping work that was supposed to need 8 gpus and an api key. open source moved the floor while the rest of the field was busy explaining why it cannot. 24gb tier owners are eating ramen with half boiled egg and double chocolate.

Sudo su

29,584 görüntüleme • 4 ay önce

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.
0:21

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 görüntüleme • 2 ay önce

mark zuckerberg just did the pettiest thing in tech: he gave away a model one point below gpt-5.6 for free, purely to make sam altman's $200 subscription look insane. it worked. the free one scores 61 to openai's 62 on every benchmark that ships, and it runs on a card you already own. the $200 you hand altman every month is now the price of a single point you cannot feel. here is the whole switch, and it runs for about $6 instead of $400: -> pull the open model zuckerberg dropped and serve it on localhost in one line, so every app that used to call openai now calls your own machine -> keep cursor, cline and your own scripts exactly as they are, none of them notice the endpoint left openai for your gpu -> send the rare heavy job to a rented nvidia card by the hour and keep the daily work local for the cost of electricity -> your $200 chatgpt seat and your $200 claude seat both go quiet, and the bill drops the same week here is the part they will fight me on: openai and anthropic are not selling a smarter model anymore, they are selling one benchmark point and a monthly invoice. zuckerberg just proved the point is free, and once the point is free, $200 is not a price, it is a dare. the full switch is in the article below. save it before the next open model passes them outright and the dare turns into a joke.
0:21

mark zuckerberg just did the pettiest thing in tech: he gave away a model one point below gpt-5.6 for free, purely to make sam altman's $200 subscription look insane. it worked. the free one scores 61 to openai's 62 on every benchmark that ships, and it runs on a card you already own. the $200 you hand altman every month is now the price of a single point you cannot feel. here is the whole switch, and it runs for about $6 instead of $400: -> pull the open model zuckerberg dropped and serve it on localhost in one line, so every app that used to call openai now calls your own machine -> keep cursor, cline and your own scripts exactly as they are, none of them notice the endpoint left openai for your gpu -> send the rare heavy job to a rented nvidia card by the hour and keep the daily work local for the cost of electricity -> your $200 chatgpt seat and your $200 claude seat both go quiet, and the bill drops the same week here is the part they will fight me on: openai and anthropic are not selling a smarter model anymore, they are selling one benchmark point and a monthly invoice. zuckerberg just proved the point is free, and once the point is free, $200 is not a price, it is a dare. the full switch is in the article below. save it before the next open model passes them outright and the dare turns into a joke.

starmex

105,174 görüntüleme • 8 gün önce

The phone threshold is even harder than the storage number suggests. A phone exposes only part of its memory to an application, and the model must share that budget with its KV cache, activations, runtime, and the rest of the product. At 3.9 GB, 1-bit Bonsai 27B clears that threshold with room to work, making 27B-class local AI possible on a phone for the first time. (Demo Mode: cached & prefilled image context)
1:19

The phone threshold is even harder than the storage number suggests. A phone exposes only part of its memory to an application, and the model must share that budget with its KV cache, activations, runtime, and the rest of the product. At 3.9 GB, 1-bit Bonsai 27B clears that threshold with room to work, making 27B-class local AI possible on a phone for the first time. (Demo Mode: cached & prefilled image context)

PrismML

33,358 görüntüleme • 2 ay önce

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.
0:27

A 27B parameter model used to need a server room. Now it runs on: • iPhone • Android • Mac PrismML's Bonsai makes it possible. • 1-bit weights • 27B params in just 3.9GB • ~90% of full precision quality (PrismML evals) • Better than the 2-bit version at less than half the size The entire Bonsai family is now live in RunAnywhere. • 1.7B to 27B models • 1-bit + 2-bit ternary • Thinking mode On iOS and Mac, Bonsai runs through ggml's llama.cpp and Apple's MLX. On Android, Bonsai runs through ggml's llama.cpp and directly on the Qualcomm Hexagon NPU through QHexRT, our proprietary inference runtime. We built custom silicon kernels to make 1-bit inference on an NPU possible for the first time. Two years ago this needed a data center. Now it thinks in airplane mode. Now available in the RunAnywhere app on the App Store and Google Play.

RunAnywhere (YC W26)

17,391 görüntüleme • 1 ay önce

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!
4:28

Alright guys, wow. Qwen 3.8 27B is basically Qwen 4. What we have here is one of the best one-shot shark survival games I've made with any model. I ran this same prompt through fable, and the biggest thing all models struggle with in this test is the top-down view of the sharks. Fable even had all of the fins reversed and details jumbled on the first turn. Before now, it was really a 2 to 8 turn test to get something final regardless of model. Qwen 3.8 27B just nailed this in one output, no harness at all, just purely the model thoroughly thinking before outputting the final result. On every asset used, different shark variants, everything, it all looks incredibly tight. The boat movement, shark speed and engagement, everything is so much better than I expected, and I was expecting a lot. Imagine what 8 turns would do in a harness? This confirms to me that while Qwen 3.8 27B thinks a lot, for the first time in a local model on the base weights, IT MIGHT ACTUALLY BE WORTH IT This is a Frontier Lab model result, and it ran on my 5090 at 60tps in Q5_K_M. I am just in complete astoundment at what Qwen has accomplished here, and I am incredibly grateful that this model is open source. The speed will also increase drastically with optimisations. This is just an incredible day. What a blessing. More to come!

Kyle Hessling

24,917 görüntüleme • 1 ay önce

Kimi K3 dropped 3 days ago and i already used it to build my own PUBG the model that just took the number one coding spot on earth and it runs on a free tier no team no engine i just fed it free open models and assets that are already public and told it what i wanted it built the entire game itself the shooting the recoil the hit detection all of it came straight out of the model i barely wrote a line what youre watching here is the deathmatch mode and its already live battle royale is next im deep in it and the graphics get reworked right after i know it looks raw right now i wont pretend otherwise but the endgame is a build that runs perfect on phones and the moment its on mobile you drop ads in and it earns on its own the part that still gets me is my whole cost for this was a single subscription
0:22

Kimi K3 dropped 3 days ago and i already used it to build my own PUBG the model that just took the number one coding spot on earth and it runs on a free tier no team no engine i just fed it free open models and assets that are already public and told it what i wanted it built the entire game itself the shooting the recoil the hit detection all of it came straight out of the model i barely wrote a line what youre watching here is the deathmatch mode and its already live battle royale is next im deep in it and the graphics get reworked right after i know it looks raw right now i wont pretend otherwise but the endgame is a build that runs perfect on phones and the moment its on mobile you drop ads in and it earns on its own the part that still gets me is my whole cost for this was a single subscription

savip.

68,984 görüntüleme • 1 ay önce

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.
4:15

i watched gemma 4 12b build something genuinely impressive today, and then loop itself to death right in front of me. the full run is in the video, sped up but completely uncut, watch it to the end and you will catch the exact moment it stops building and starts looping right in the middle of the work. the task was clean, build a single file gravity simulator, n-body physics, orbits, collisions, running locally on one 3090 through an agent. and for ten minutes it was a joy to watch. it reached for a symplectic integrator on its own, the correct one, the kind that keeps orbits stable instead of spiralling out. real gravity with softening, proper orbital velocities, momentum conserved on collision. the physics was right. the thing actually worked. then on the very last step, writing a few tests to prove its own code, it fell into a loop. not a crash, a loop. it started repeating itself and would not stop. ten more minutes, thirty four thousand tokens into a single answer, the same fragments over and over, until i killed it myself. so it's not that gemma can't code. it did the hard part beautifully. it cannot finish. it cannot hold a long task together without unravelling, and finishing is the entire job in agentic work. here's the part that stings. i run this exact task, same harness, same card, on the chinese open models, qwen especially, and i never see this. they build it, they test it, they stop. every single time. google has the raw capability, you can see it sitting right there in the code, and then the model loops itself to death on a task a 27b from alibaba finishes clean. open weights, apache 2.0, so much to love on paper. i just need it to know when to stop talking.

Sudo su

39,719 görüntüleme • 3 ay önce

this is what 12 gigs of VRAM built in 2026. a 9 billion parameter model running on a 5 year old RTX 3060 wrote a full space shooter from a single prompt. blank screen on first try. i came back with a bug list and the same model on the same card fixed every issue across 11 files without touching a single line myself. enemies still looked wrong so i pushed another iteration and now the game has pixel art octopi, particle effects, screen shake, projectile physics and a combo system. all running locally on a card that was designed to play fortnite. three iterations. zero cloud. zero API calls. every token generated on hardware sitting under my desk. the model reads its own code, finds what's broken, patches it, validates syntax and restarts the server. i just describe what's wrong and it handles the rest. people are paying monthly subscriptions to type into a browser tab and wait for a server farm to respond. meanwhile a GPU you can find used on ebay is running a full autonomous hermes agent framework with 31 tools, 128K context window and thinking mode generating at 29 tokens per second nonstop. the game still needs work. level upgrades don't trigger and boss fights need tuning. but the fact that i'm iterating on gameplay balance instead of debugging whether the code runs at all tells you where this is headed. every iteration the game gets better on the same hardware. same 12 gigs. same 9 billion parameters. same RTX 3060 from 5 years ago your GPU is not a gaming card anymore. it's a local AI lab that never sends your data anywhere.
0:28

this is what 12 gigs of VRAM built in 2026. a 9 billion parameter model running on a 5 year old RTX 3060 wrote a full space shooter from a single prompt. blank screen on first try. i came back with a bug list and the same model on the same card fixed every issue across 11 files without touching a single line myself. enemies still looked wrong so i pushed another iteration and now the game has pixel art octopi, particle effects, screen shake, projectile physics and a combo system. all running locally on a card that was designed to play fortnite. three iterations. zero cloud. zero API calls. every token generated on hardware sitting under my desk. the model reads its own code, finds what's broken, patches it, validates syntax and restarts the server. i just describe what's wrong and it handles the rest. people are paying monthly subscriptions to type into a browser tab and wait for a server farm to respond. meanwhile a GPU you can find used on ebay is running a full autonomous hermes agent framework with 31 tools, 128K context window and thinking mode generating at 29 tokens per second nonstop. the game still needs work. level upgrades don't trigger and boss fights need tuning. but the fact that i'm iterating on gameplay balance instead of debugging whether the code runs at all tells you where this is headed. every iteration the game gets better on the same hardware. same 12 gigs. same 9 billion parameters. same RTX 3060 from 5 years ago your GPU is not a gaming card anymore. it's a local AI lab that never sends your data anywhere.

Sudo su

170,848 görüntüleme • 6 ay önce

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.
2:18

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.

Sudo su

32,163 görüntüleme • 3 ay önce

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.
2:15

HERMES AGENT VS OPENCLAW. a local ai onboarding flow test. a 3.9gb bonsai served on localhost, both agents upstream and latest, i point each one at the endpoint and watch which one even finds it. > hermes opens a provider menu, thirty plus options, local servers sitting right there next to the cloud ones, i hand it 127.0.0.1:8899, it verifies the endpoint, one model visible, auto-detects the model by name, bonsai-27b-q1_0, reads the context length straight off the server, saves it, and starts reasoning and firing real tool calls on my local model. no key. no friction. > openclaw has no menu. it goes hunting for a codex login, an openai key, finds none because there are none, prints no models available three times, defaults to openai/gpt-5.5, a cloud model it cannot reach, and dead ends on run auth login --provider openai. read that back. it asked me for an openai key. to run a model already running on my own machine. it never once looked at localhost. to be fair, openclaw can run local if you hand wire endpoint yourself. what it will not do is find the model already sitting on your box. hermes agent found it in one line. now the part i owe you. the auto-detect that just won, the model name read, the .gguf strip, the context length probe off the server, that is my code, it is in hermes agent main right now, authorship preserved, #2051 and #4218. the wizard fix that stops an agent from silently routing you to someone else's creds, the exact trap openclaw still falls into, mine too, #4210. i contribute to hermes agent, i told you that going in. one agent is built to talk to whatever you are running, the other is built to talk to a cloud api, so one found my model and ran it and the other asked me to log into openai. onboarding flow of both, mapped, below.

Sudo su

23,816 görüntüleme • 1 ay önce

no prompt engineering, no agentic harness, no tool calls. just me being lazy in llama.cpp's web ui and gemma 4 31b dense taking the task seriously. i typed "create gpu marketplace cards with hardware specs and prices per hour" and the model went and coded this ui, one shot, navy bg, glassmorphism cards, neon accent buttons, realistic pricing tiers per architecture. it even wrote a "why this looks premium" explanation under the code. for context this is a q4 quant of google's 31b dense thinking model, running on a rtx 5090 mobile 24gb in the rog scar 18 at around 15 tok/s sustained, same vram tier as a 3090 or 4090 desktop, so whatever you see here translates directly to your card at home. the whole interaction was me not trying and the model reasoning harder than the prompt deserved. that tells me more about where local ai is at in april 2026 than any leaderboard score. next test drops gemma 4 into hermes agent, autonomous tool calling, multi step reasoning, real agentic loop instead of a chat window. let's see what the same model does when it gets the right environment. more experiments coming anon. octopus invaders queued. same hardware, different tasks, all published here on x and all translatable to your 24gb card. for now the video below shows it coding live, gpu going brrr.
2:23

no prompt engineering, no agentic harness, no tool calls. just me being lazy in llama.cpp's web ui and gemma 4 31b dense taking the task seriously. i typed "create gpu marketplace cards with hardware specs and prices per hour" and the model went and coded this ui, one shot, navy bg, glassmorphism cards, neon accent buttons, realistic pricing tiers per architecture. it even wrote a "why this looks premium" explanation under the code. for context this is a q4 quant of google's 31b dense thinking model, running on a rtx 5090 mobile 24gb in the rog scar 18 at around 15 tok/s sustained, same vram tier as a 3090 or 4090 desktop, so whatever you see here translates directly to your card at home. the whole interaction was me not trying and the model reasoning harder than the prompt deserved. that tells me more about where local ai is at in april 2026 than any leaderboard score. next test drops gemma 4 into hermes agent, autonomous tool calling, multi step reasoning, real agentic loop instead of a chat window. let's see what the same model does when it gets the right environment. more experiments coming anon. octopus invaders queued. same hardware, different tasks, all published here on x and all translatable to your 24gb card. for now the video below shows it coding live, gpu going brrr.

Sudo su

28,939 görüntüleme • 5 ay önce

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

341,746 görüntüleme • 24 gün önce

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below
0:50

your agent reviewing its own work is not a check. it is a second opinion from the same source. this is the most common gap in agent systems and it hides in plain sight, because the step exists. there is a review. it just cannot do the thing you think it does. here is the mechanism. the model produced an output from a context. you then ask the same model, holding the same context, whether that output is correct. it answers fluently, because that is what it does. and the answer is drawn from the same distribution that produced the thing being judged. same weights, same window, same blind spots. if the reason the output is wrong is something the model does not know, the review does not know it either. if the reason is something the context does not contain, the review has the same context. the failure mode and the detector share a cause. > why it feels like it works because most of the time the output is fine, and the review says fine. agreement is not evidence of detection. a reviewer that says pass on everything agrees with reality most of the time too. what you actually want to measure is what happens on the cases that are wrong. that is the only place a check earns its name, and it is exactly the place where a self-review is weakest. there is research on this. Huang and colleagues at DeepMind showed at ICLR 2024 that intrinsic self-correction, revising without external grounding, does not reliably help and often makes things worse. > what to actually do move the check outside the model. a test that runs, a schema that validates, a file that exists or does not, an exit code from something you did not write. these are not smarter than the model. they are just not correlated with it, and that is the entire value. when the judgement genuinely needs a model, at minimum use a different family. same family means shared blind spots, and frontier judges measurably inflate scores for outputs that look like their own. and split the work by kind. anything objectively checkable goes to code. only the genuinely semantic calls go to a judge, and those get a rubric written as one line. a review inside the loop tells you the model is confident. a check outside it tells you whether the work is done. save this - then read the eval setup below

Hanako

14,325 görüntüleme • 1 ay önce

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own
0:20

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 görüntüleme • 2 ay önce

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.
1:46

small local model that falls apart in bloated agents like openclaw just runs like a wild horse in hermes agent. and that's not even my line, someone else called it that, i've just been quietly pointing people at this harness for months because it held up on everything i threw at it, 3b models all the way to one trillion params. watch this happen on my own machine. i pointed hermes agent at a local http endpoint, gemma 4 12b on my 3090 llama.cpp server, and it auto-detected the model and started working immediately. no config wrestling, no broken tool calls, no babysitting the output format, i typed in a url and it just went. the whole clip is exactly that, start to finish, no errors, no retries, butter smooth. and the tool calling, the one thing that quietly breaks most local setups, works here like it's nothing. it's not the model that's flaky, it's the harness around it. hermes agent is the first agent i've run that actually gets that right. one url, one local model on one card, and it runs like a wild horse.

Sudo su

27,339 görüntüleme • 3 ay önce

ok this is wild. 10 year old gtx 1080 8gb pascal card running qwen3 8b locally at 18-20 tok/s via hermes agent and it's actually doing the thing. asked it to build a wireworld cellular automata simulator with 10 tests. autonomous run, no hand holding. expected it to fail on the tool calls. that's not what happened. write_file works. browser_navigate works. terminal commands work. file ops, package installs, version probes, environment setup. agent is firing tool calls cleanly and the model is reasoning about next steps at 18-20 tok/s. on hardware that pre dates "agentic" as a word. it even hit an npm install fail because node 12 is too old. didn't crash. didn't ask me. just started bootstrapping nvm on its own to fix the environment. 10 minutes in. 40% context used. 7.5gb of 8gb vram occupied. still going. i did not think this would work on this hardware. this is the most i've been wrong this month.
0:23

ok this is wild. 10 year old gtx 1080 8gb pascal card running qwen3 8b locally at 18-20 tok/s via hermes agent and it's actually doing the thing. asked it to build a wireworld cellular automata simulator with 10 tests. autonomous run, no hand holding. expected it to fail on the tool calls. that's not what happened. write_file works. browser_navigate works. terminal commands work. file ops, package installs, version probes, environment setup. agent is firing tool calls cleanly and the model is reasoning about next steps at 18-20 tok/s. on hardware that pre dates "agentic" as a word. it even hit an npm install fail because node 12 is too old. didn't crash. didn't ask me. just started bootstrapping nvm on its own to fix the environment. 10 minutes in. 40% context used. 7.5gb of 8gb vram occupied. still going. i did not think this would work on this hardware. this is the most i've been wrong this month.

Sudo su

92,585 görüntüleme • 3 ay önce

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 görüntüleme • 3 ay önce

Every big MoE model I test on my 3090 is basically to see how good it is compared to Qwen3.6-27B. "How good is it compared to Qwen3.6-27B?" is the question I get asked the most, and that shows how much people love and, more importantly, trust the 27B 27B is battle tested. It has proven its worth and has nothing left to prove. Now it just needs to make way for the next 27B. Here’s an example video: a Three.js task, same prompt, not oneshot but two shots for all models. All ran on my 3090 + 64GB VRAM Qwen-3.6-27b fully on vram while others have experts offloaded to system ram. Pinned on the left: Qwen3.6-27B-Q5 On the right, i'm switching between Inkling-Small-Q4_K_M DSV4-Flash-0731-UD-IQ1 Ling-3.0-Flash-Q4_K_M Qwen3.6-27B nailed it. The effects and visuals are just spot on.The other models got some things right, but not everything. Second place would have to go to Inkling-Small while DSV4-Flash-0731 and Ling-3.0-Flash failed to capture the full requirements. This is also why we’re all excited for Qwen Qwen3.8-27B!
2:20

Every big MoE model I test on my 3090 is basically to see how good it is compared to Qwen3.6-27B. "How good is it compared to Qwen3.6-27B?" is the question I get asked the most, and that shows how much people love and, more importantly, trust the 27B 27B is battle tested. It has proven its worth and has nothing left to prove. Now it just needs to make way for the next 27B. Here’s an example video: a Three.js task, same prompt, not oneshot but two shots for all models. All ran on my 3090 + 64GB VRAM Qwen-3.6-27b fully on vram while others have experts offloaded to system ram. Pinned on the left: Qwen3.6-27B-Q5 On the right, i'm switching between Inkling-Small-Q4_K_M DSV4-Flash-0731-UD-IQ1 Ling-3.0-Flash-Q4_K_M Qwen3.6-27B nailed it. The effects and visuals are just spot on.The other models got some things right, but not everything. Second place would have to go to Inkling-Small while DSV4-Flash-0731 and Ling-3.0-Flash failed to capture the full requirements. This is also why we’re all excited for Qwen Qwen3.8-27B!

AJ

11,670 görüntüleme • 1 ay önce