Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Extremely fast multimodal inference! Damage Scout uses Gemma 4 on running at an impressive 2,300+ toks/s! It analyzes rental car walkaround videos and generates annotated damage reports with box coordinates in <6 seconds.

Google Gemma

95,790 subscribers

79,654 views • 27 days ago •via X (Twitter)

Science & Technology Gaming News & Politics

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Gemma 4 31B is now available in Public Preview on Cerebras. Our first multimodal model runs at over 1,800 tokens/s for ultra-fast image and text workflows. Give it a try:

Gemma 4 31B is now available in Public Preview on Cerebras. Our first multimodal model runs at over 1,800 tokens/s for ultra-fast image and text workflows. Give it a try:

Cerebras

281,269 views • 1 month ago

MLXs CUDA backend is getting better. It's especially nice if you appreciate fast startup times. But it's also quite fast in general. Here's Qwen3 4B in fp8 running on my DGX Spark. - Processed 18.5k tokens in < 4 seconds - Generates at 32.5 tok/sec with 18.5k context

MLXs CUDA backend is getting better. It's especially nice if you appreciate fast startup times. But it's also quite fast in general. Here's Qwen3 4B in fp8 running on my DGX Spark. - Processed 18.5k tokens in < 4 seconds - Generates at 32.5 tok/sec with 18.5k context

Awni Hannun

17,327 views • 6 months ago

Fast tokens are the most valuable tokens. Because inference speed is productivity. And Cerebras makes the fastest tokens in the world. Google DeepMind's Gemma 4 runs at 1,500+ tokens per second on Cerebras — more than an order of magnitude faster than on GPUs and 15x faster than Claude Haiku 4.5. Private preview today. GA later this month. Build blazing fast multimodal apps.

Fast tokens are the most valuable tokens. Because inference speed is productivity. And Cerebras makes the fastest tokens in the world. Google DeepMind's Gemma 4 runs at 1,500+ tokens per second on Cerebras — more than an order of magnitude faster than on GPUs and 15x faster than Claude Haiku 4.5. Private preview today. GA later this month. Build blazing fast multimodal apps.

Andrew Feldman

12,008 views • 2 months ago

Our cofounder tim built an AI storybook app using Black Forest Labs' FLUX2 and Google Gemma 4 on Modular Cloud. Pick a character, make choices, and the story branches endlessly, with every page written and illustrated in real time. Tim has spent his career obsessing over inference latency, first at Google, now at Modular. Building something his kids use settled it: in a real-time generative app, the inference platform determines the experience as much as the model. The numbers back that up. From 24 hours of production traffic: first prose in 420ms, a full illustration in under 6 seconds, 85% of page turns in 48ms. Create your own story with Inkwell and share it. We're sending swag to our favorites:

Our cofounder tim built an AI storybook app using Black Forest Labs' FLUX2 and Google Gemma 4 on Modular Cloud. Pick a character, make choices, and the story branches endlessly, with every page written and illustrated in real time. Tim has spent his career obsessing over inference latency, first at Google, now at Modular. Building something his kids use settled it: in a real-time generative app, the inference platform determines the experience as much as the model. The numbers back that up. From 24 hours of production traffic: first prose in 420ms, a full illustration in under 6 seconds, 85% of page turns in 48ms. Create your own story with Inkwell and share it. We're sending swag to our favorites:

Modular

14,324 views • 3 months ago

"It's the same as if they were throwing paint on the Lincoln statue." Interior Secretary Doug Burgum says vandalizing the Reflecting Pool is being treated as an attack on a national monument, with authorities already making seven arrests, issuing multiple citations, and filing 18 police reports tied to damage at the Lincoln Memorial. Burgum says people have used box cutters to vandalize the Reflecting Pool, damage surrounding grass, and even leave threatening messages aimed at President Trump. "President Trump came to D.C. to drain the swamp... Some of these folks are trying to bring it back. It's crazy." FOX & Friends

"It's the same as if they were throwing paint on the Lincoln statue." Interior Secretary Doug Burgum says vandalizing the Reflecting Pool is being treated as an attack on a national monument, with authorities already making seven arrests, issuing multiple citations, and filing 18 police reports tied to damage at the Lincoln Memorial. Burgum says people have used box cutters to vandalize the Reflecting Pool, damage surrounding grass, and even leave threatening messages aimed at President Trump. "President Trump came to D.C. to drain the swamp... Some of these folks are trying to bring it back. It's crazy." FOX & Friends

Fox News

565,097 views • 1 month ago

A single, viral post on X skyrocketed Groq’s business. Groq Founder Jonathan Ross was looking for a way to show people that fast inference would be extremely useful. He remembered an example from Eric Schmidt showing off an LLM at a conference three weeks before the ChatGPT moment. Nobody in the audience cared. Then he looked at what made ChatGPT’s launch successful and realized that it worked because they allowed people to ask their own questions. “They got an answer to their question that was specific to them. That was magical.” “Seeing text show up for someone else's answer wasn't magical.” So he put Groq’s fast inference hardware on the internet for anyone to try. Someone posted a video on X of an LLM running on Groq running super fast. “It was just such eye candy when people saw it that everyone started creating their own and we just went viral.”

A single, viral post on X skyrocketed Groq’s business. Groq Founder Jonathan Ross was looking for a way to show people that fast inference would be extremely useful. He remembered an example from Eric Schmidt showing off an LLM at a conference three weeks before the ChatGPT moment. Nobody in the audience cared. Then he looked at what made ChatGPT’s launch successful and realized that it worked because they allowed people to ask their own questions. “They got an answer to their question that was specific to them. That was magical.” “Seeing text show up for someone else's answer wasn't magical.” So he put Groq’s fast inference hardware on the internet for anyone to try. Someone posted a video on X of an LLM running on Groq running super fast. “It was just such eye candy when people saw it that everyone started creating their own and we just went viral.”

David Senra

156,503 views • 1 month ago

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

timelapse #85 (27.5 hrs): - currently cant rely on any other coding models except grok code fast 1 + grok 4 fast (for complex reasoning grok 4 fast is 20 cents for 1M tokens) - wrote qwen3-next trainer entirely from scratch to make it more managable - each piece completely done by grok-code-fast-1 in cursor as it seems to handle this task pretty well without the grok 4 fast reasoning - take on smaller problems and complete them quickly (makes it easier with 400 toks/sec over the api) - got distributed fp8 qwen3-next trainer running at 0.8 seconds per step on 8xH100s (still need to finish checkpoint loading logic) - perfect timing as the fp8 version of qwen3-next drops as im writing this - ill be in LA in 2 days (will visit SF mid way through as well) - 12.5% margarita - steak dinner with family - gained intuition on FlashAttention in very long context settings - caught up w/ Kearm h/eng and Arnie Ramesh

Elliot Arledge

283,820 views • 10 months ago

🚨 do you understand what this AMD engineer just proved on camera watch the clip, it’s all there in one frame: llama.cpp slot logs on the left, nvitop showing 79% GPU util holding steady, localhost chat answering a 26k token prompt at 711 tok/s prompt eval, the gemma-4-12B Q4_K_XL gguf running clean. no cloud call, no api key, no censorship layer. for 2 years the rule was: frontier models need the cloud, local is too slow for real work, pay $200/month and wait for tokens. today a $1,800 lunch box runs qwen 3 235B at 11 tok/s with zero data leaving the room, and the same box runs gemma 4 12B with multi token prediction at 14+ tok/s on 30k context with no throttling. $440/month subscription stack: dead. $200/month claude tier: dead. local is too slow excuse: dead. one lunch box. $9/month in electricity. two moats buried in one week. the you need the cloud line wasn’t protecting users it was protecting the subscription.

🚨 do you understand what this AMD engineer just proved on camera watch the clip, it’s all there in one frame: llama.cpp slot logs on the left, nvitop showing 79% GPU util holding steady, localhost chat answering a 26k token prompt at 711 tok/s prompt eval, the gemma-4-12B Q4_K_XL gguf running clean. no cloud call, no api key, no censorship layer. for 2 years the rule was: frontier models need the cloud, local is too slow for real work, pay $200/month and wait for tokens. today a $1,800 lunch box runs qwen 3 235B at 11 tok/s with zero data leaving the room, and the same box runs gemma 4 12B with multi token prediction at 14+ tok/s on 30k context with no throttling. $440/month subscription stack: dead. $200/month claude tier: dead. local is too slow excuse: dead. one lunch box. $9/month in electricity. two moats buried in one week. the you need the cloud line wasn’t protecting users it was protecting the subscription.

AdiiX

22,223 views • 2 months ago

How Tyler Cowen Uses AI 1) Grok: Quick and effective for fact-checking tweets. 2) o1 Pro: Best for queries. His primary daily tool. 3) Deep Research: Ideal for long-form (10-page) reports; extremely impressive but less practical for personal routine; occasionally useful for teaching; significant for replacing human labor. 4) Claude: Darn good writer — thoughtful, philosophical, and versatile. 5) DeepSeek: Sends data to China, though Tyler is personally comfortable using it. 6) Gemini: Good for multimodal tasks and handling large or thick documents; best for legal work and integrating with video and YouTube. 7) Meta (Llama models): Integrates with WhatsApp; strong in marketing and open source; important globally, though not part of Tyler's regular routine.

How Tyler Cowen Uses AI 1) Grok: Quick and effective for fact-checking tweets. 2) o1 Pro: Best for queries. His primary daily tool. 3) Deep Research: Ideal for long-form (10-page) reports; extremely impressive but less practical for personal routine; occasionally useful for teaching; significant for replacing human labor. 4) Claude: Darn good writer — thoughtful, philosophical, and versatile. 5) DeepSeek: Sends data to China, though Tyler is personally comfortable using it. 6) Gemini: Good for multimodal tasks and handling large or thick documents; best for legal work and integrating with video and YouTube. 7) Meta (Llama models): Integrates with WhatsApp; strong in marketing and open source; important globally, though not part of Tyler's regular routine.

David Perell

225,716 views • 1 year ago

Roger Ebert on Louis Malle's "Damage" (1992): "Watching 'Damage' (1992) is like watching an emotional traffic accident as it unfolds. The film is based on the best seller by Josephine Hart, which had a certain undeniable power, but the right place for this material is the screen, I think, because it can show exactly how the two look at one another. This is a movie about sight; from the first moment the two meet, it is filled with what is seen and what is not seen, as Stephen [Jeremy Irons] suffers though a dinner party with his wife, his son, Anna [Juliette Binoche] and her mother – and some observe, and some do not, what has happened. (...) 'Damage' is one of those rare movies that is about s*xuality, not s*x; about the tension between people, not “relationships”; about how physical love is meaningless without a psychic engine behind it. Stephen and Anna are wrong to do what they do in 'Damage' but they cannot help themselves. We know they are careening toward disaster. We cannot look away." ("Damage", Roger Ebert, 1993)

Roger Ebert on Louis Malle's "Damage" (1992): "Watching 'Damage' (1992) is like watching an emotional traffic accident as it unfolds. The film is based on the best seller by Josephine Hart, which had a certain undeniable power, but the right place for this material is the screen, I think, because it can show exactly how the two look at one another. This is a movie about sight; from the first moment the two meet, it is filled with what is seen and what is not seen, as Stephen [Jeremy Irons] suffers though a dinner party with his wife, his son, Anna [Juliette Binoche] and her mother – and some observe, and some do not, what has happened. (...) 'Damage' is one of those rare movies that is about sxuality, not sx; about the tension between people, not “relationships”; about how physical love is meaningless without a psychic engine behind it. Stephen and Anna are wrong to do what they do in 'Damage' but they cannot help themselves. We know they are careening toward disaster. We cannot look away." ("Damage", Roger Ebert, 1993)

DepressedBergman

40,321 views • 2 months ago

🚨#BREAKING: A confirmed large damaging tornado on the ground with Reports of multiple buildings being damage 📌#Johnson | #indiana Moments ago, a confirmed large and extremely violent tornado was reported on the ground. Multiple videos from eyewitnesses show the tornado causing significant damage as it passed through the Mount Vernon, Indiana area and other nearby towns. The National Weather Service in Indiana has now issued a PDS (Particularly Dangerous Situation) tornado warning for Gibson County, IN; Posey County, IN; and Wabash County, IL. This tornado is moving quickly and poses a severe threat to life and property. Take immediate shelter this supercell has a history of producing and spawning tornadoes

🚨#BREAKING: A confirmed large damaging tornado on the ground with Reports of multiple buildings being damage 📌#Johnson | #indiana Moments ago, a confirmed large and extremely violent tornado was reported on the ground. Multiple videos from eyewitnesses show the tornado causing significant damage as it passed through the Mount Vernon, Indiana area and other nearby towns. The National Weather Service in Indiana has now issued a PDS (Particularly Dangerous Situation) tornado warning for Gibson County, IN; Posey County, IN; and Wabash County, IL. This tornado is moving quickly and poses a severe threat to life and property. Take immediate shelter this supercell has a history of producing and spawning tornadoes

R A W S A L E R T S

593,811 views • 2 years ago

18-year-old American girl made $10,000 in a month on YouTube videos made entirely by ChatGPT 5.5 $20 subscription, an old laptop and a formula most people don't see. Finds a kids channel with millions of views - copies the structure, not the content. ChatGPT rewrites the script, PixArt Flow with Sora 2 generates the video, ElevenLabs voices it. 20 minutes - a finished video with the viral formula built in. YouTube pays $4,000-15,000 per million views on kids content - the highest rate on the entire platform. TikTok Shop on top: AI analyzes a product in 30 seconds, writes 10 hooks, generates content for 5 platforms simultaneously. Month 1 - $3000 Month 6 - $7,000 Year - $17,000 every month. The most expensive camera in the world is the one you never opened.

18-year-old American girl made $10,000 in a month on YouTube videos made entirely by ChatGPT 5.5 $20 subscription, an old laptop and a formula most people don't see. Finds a kids channel with millions of views - copies the structure, not the content. ChatGPT rewrites the script, PixArt Flow with Sora 2 generates the video, ElevenLabs voices it. 20 minutes - a finished video with the viral formula built in. YouTube pays $4,000-15,000 per million views on kids content - the highest rate on the entire platform. TikTok Shop on top: AI analyzes a product in 30 seconds, writes 10 hooks, generates content for 5 platforms simultaneously. Month 1 - $3000 Month 6 - $7,000 Year - $17,000 every month. The most expensive camera in the world is the one you never opened.

Noisy

158,701 views • 3 months ago

Google Veo 3 now available in Gemini API! Generate 8 second videos with audio and vocals from simple text prompts. Starting at $0.75 per second with audio! 📹 🔉 Here is an python script on how to make viral yeti vlogs from TikTok or Youtube shorts. It uses a 3 step workflow: 1. Based on an idea, it generates a series of scene prompts using Gemini 2.5. 2. For each scene prompt Veo 3 generates a video clip. 3. Combine the individual video clips into a single final video using MoviePy Script and some examples below🧵

Google Veo 3 now available in Gemini API! Generate 8 second videos with audio and vocals from simple text prompts. Starting at $0.75 per second with audio! 📹 🔉 Here is an python script on how to make viral yeti vlogs from TikTok or Youtube shorts. It uses a 3 step workflow: 1. Based on an idea, it generates a series of scene prompts using Gemini 2.5. 2. For each scene prompt Veo 3 generates a video clip. 3. Combine the individual video clips into a single final video using MoviePy Script and some examples below🧵

Philipp Schmid

13,983 views • 1 year ago

Moms Across America Blowing The Whistle On Dunkin' Donuts, It’s Chemical Poison Only 1 part per billion of glyphosate can cause sex hormone changes, damage sperm and cause kidney and organ damage 🚨🚨 Dunkin' Donuts contains 133 parts per billion of glyphosate when tested “I'm passing by at Dunkin' Donuts. Need to remind you that when we tested Dunkin' Donuts and 21 of the different top fast food restaurants, we found over a 133 parts per billion of glyphosate in Dunkin' Donuts samples. Of course, the of donuts and breakfast sandwiches. And that's not good, folks. Point 1 to 1 part per billion can cause sex hormone changes and kidney and organ damage. There glyphosate's also an endocrine disruptor, a carcinogenic carcinogenic, uh, causes kidney and liver damage, causes nervous system damage, neurotoxicity, it damages sperm, causes, uh, it increases the chances of miscarriages and prenatal births. It is no bueno. So it's not a good idea to start your day with foods that have glyphosate in them. Please buy organic. You are worth it, and it's worth it to buy whole foods, whole organic local foods. Thank you”

Moms Across America Blowing The Whistle On Dunkin' Donuts, It’s Chemical Poison Only 1 part per billion of glyphosate can cause sex hormone changes, damage sperm and cause kidney and organ damage 🚨🚨 Dunkin' Donuts contains 133 parts per billion of glyphosate when tested “I'm passing by at Dunkin' Donuts. Need to remind you that when we tested Dunkin' Donuts and 21 of the different top fast food restaurants, we found over a 133 parts per billion of glyphosate in Dunkin' Donuts samples. Of course, the of donuts and breakfast sandwiches. And that's not good, folks. Point 1 to 1 part per billion can cause sex hormone changes and kidney and organ damage. There glyphosate's also an endocrine disruptor, a carcinogenic carcinogenic, uh, causes kidney and liver damage, causes nervous system damage, neurotoxicity, it damages sperm, causes, uh, it increases the chances of miscarriages and prenatal births. It is no bueno. So it's not a good idea to start your day with foods that have glyphosate in them. Please buy organic. You are worth it, and it's worth it to buy whole foods, whole organic local foods. Thank you”

Wall Street Apes

1,072,374 views • 2 years ago

.GREATER CHENNAI POLICE -GCP Got hit by a lorry (TN 18 BH 9331) from the back on the Vandalur-Walajabad Highway at around 6:03 am. The lorry driver didn’t care one bit and turned out to be even more menacing afterwards. With a car full of senior citizens, I didn’t want to stop the car in the middle of the road and create a ruckus, but I hope you will do something about it. Sharing dashcam video to help you get a sense of the impact (you can’t see it since we got hit on the right rear), the vehicle number, and the damage to my car. Please let me know if you need any other info in this regard. Glad I was in a Volkswagen. Had this been a Maruti Suzuki, the car would have turned for sure and the damage to the passengers would have been worse.

.GREATER CHENNAI POLICE -GCP Got hit by a lorry (TN 18 BH 9331) from the back on the Vandalur-Walajabad Highway at around 6:03 am. The lorry driver didn’t care one bit and turned out to be even more menacing afterwards. With a car full of senior citizens, I didn’t want to stop the car in the middle of the road and create a ruckus, but I hope you will do something about it. Sharing dashcam video to help you get a sense of the impact (you can’t see it since we got hit on the right rear), the vehicle number, and the damage to my car. Please let me know if you need any other info in this regard. Glad I was in a Volkswagen. Had this been a Maruti Suzuki, the car would have turned for sure and the damage to the passengers would have been worse.

Narayanan Hariharan

63,170 views • 4 months ago

🚨#BREAKING: Watch as twin tornadoes emerge from an extremely large, life-threatening tornado currently on the ground 📌#Braman | #oklahoma At this time, the National Weather Service has issued a Particularly Dangerous Situation (PDS) Tornado Warning for northern Oklahoma. A confirmed, life-threatening tornado is on the ground, with reports of twin tornadoes actively causing damage near Braman, Oklahoma. This is an extremely rare and serious designation, indicating a high risk of catastrophic destruction. Authorities warn that this is a large and extremely dangerous tornado capable of devastating impacts. If you are in the affected area, TAKE SHELTER IMMEDIATELY. Move to a basement or an interior room on the lowest floor of a sturdy building, away from windows. This is a rapidly evolving and highly dangerous situation.

🚨#BREAKING: Watch as twin tornadoes emerge from an extremely large, life-threatening tornado currently on the ground 📌#Braman | #oklahoma At this time, the National Weather Service has issued a Particularly Dangerous Situation (PDS) Tornado Warning for northern Oklahoma. A confirmed, life-threatening tornado is on the ground, with reports of twin tornadoes actively causing damage near Braman, Oklahoma. This is an extremely rare and serious designation, indicating a high risk of catastrophic destruction. Authorities warn that this is a large and extremely dangerous tornado capable of devastating impacts. If you are in the affected area, TAKE SHELTER IMMEDIATELY. Move to a basement or an interior room on the lowest floor of a sturdy building, away from windows. This is a rapidly evolving and highly dangerous situation.

R A W S A L E R T S

1,731,239 views • 3 months ago

So far, all I have seen are images of an ally on fire next to maybe a small clinic or office that Hamas claims is a hospital where 1,000 people died. There is no way 1,000 people were in that 500 square meter single story building they show on fire. There are also images of bodies in sheets laying on a lawn outside and people laying on a floor in a building where there are lights and no damage. Am I missing something? Where is the 6 story hospital that got flattened? What are the geolocation coordinates? Where is the video of rescues? Why is the western press running with this? #HamasAttack #Israel #Hamas #Hezbollah #Palestine #OSINT

So far, all I have seen are images of an ally on fire next to maybe a small clinic or office that Hamas claims is a hospital where 1,000 people died. There is no way 1,000 people were in that 500 square meter single story building they show on fire. There are also images of bodies in sheets laying on a lawn outside and people laying on a floor in a building where there are lights and no damage. Am I missing something? Where is the 6 story hospital that got flattened? What are the geolocation coordinates? Where is the video of rescues? Why is the western press running with this? #HamasAttack #Israel #Hamas #Hezbollah #Palestine #OSINT

OSINT Intuit™

38,255 views • 2 years ago

🚨 ALIBABA JUST OPEN-SOURCED THE MOST POWERFUL MODEL IN THE WORLD - Qwen3.5 plus And it changes the AI race forever. This is not another “bigger model” release. This is a complete architectural reset. 397B parameters → only 17B active at inference, Qwen3.5 plus delivers performance on par with GPT-5.2 and Gemini-3-Pro. A new Hybrid Sparse MoE + Linear Attention architecture + Multi-Token Prediction = near-instant long-context responses and massive speed gains. Built as a true native multimodal model, it doesn’t just chat it builds. With a single prompt: “Use the React framework to create a visually impressive Mapbox demo featuring locations in Beijing and Shanghai” it generates a live interactive 3D map UI, structured components, real layouts, and production-ready frontend logic. This is real agentic, multimodal coding not a static code block. MMLU-Pro: 87.8 GPQA: 88.4 IFBench: 76.5 ⚡ up to 8× faster inference And it’s fully open-sourced under Apache-2.0. No locked APIs. No restricted tiers. 201 languages supported bringing frontier AI to developers, startups, and researchers worldwide. The race is no longer about the biggest model. It’s about the most usable intelligence. Open source just entered the top tier.

🚨 ALIBABA JUST OPEN-SOURCED THE MOST POWERFUL MODEL IN THE WORLD - Qwen3.5 plus And it changes the AI race forever. This is not another “bigger model” release. This is a complete architectural reset. 397B parameters → only 17B active at inference, Qwen3.5 plus delivers performance on par with GPT-5.2 and Gemini-3-Pro. A new Hybrid Sparse MoE + Linear Attention architecture + Multi-Token Prediction = near-instant long-context responses and massive speed gains. Built as a true native multimodal model, it doesn’t just chat it builds. With a single prompt: “Use the React framework to create a visually impressive Mapbox demo featuring locations in Beijing and Shanghai” it generates a live interactive 3D map UI, structured components, real layouts, and production-ready frontend logic. This is real agentic, multimodal coding not a static code block. MMLU-Pro: 87.8 GPQA: 88.4 IFBench: 76.5 ⚡ up to 8× faster inference And it’s fully open-sourced under Apache-2.0. No locked APIs. No restricted tiers. 201 languages supported bringing frontier AI to developers, startups, and researchers worldwide. The race is no longer about the biggest model. It’s about the most usable intelligence. Open source just entered the top tier.

RAVI KUMAR SAHU

380,123 views • 6 months ago

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

six months ago this wasn't happening on 8gb vram. running unsloth's Q4_K_XL quant of gemma 4 26b-a4b-it-qat, a sparse MoE model with only 4b active params on a single rtx 4060 laptop gpu, 8gb vram, 20+ tok/s decode. no cloud, no api, no offload hacks. just a gaming laptop on battery. what makes it fit: google's QAT (quantization aware training), plus MTP (multi token prediction) support in the latest llama.cpp builds. that combo is the single biggest unlock for local inference on low vram. rtx 3060, rtx 3070, gtx 1070, gtx 1080, rtx 4050, rtx 4060, rtx 5050, rtx 5060 — any 6-8gb consumer gpu, old or new — this model runs on it. world cup season, so i told it to build a soccer themed flappy bird clone. one shot, zero iteration, fully playable. six months ago an 8gb model could barely clone vanilla flappy bird. now it's shipping a themed game from a sparse MoE model running locally on a laptop battery. inference benchmarks: - decode throughput: 30 tok/s - context: 64k. this is the real unlock. 64k ctx is what makes a hermes agent loop viable locally on this model, not just single-turn chat. llama.cpp flags: -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -c 64000 -cmoe --port 8080 game's deployed on my own site, built and shipped end to end with open source llm, zero closed source api dependency in the pipeline. link in the description. gguf weights on huggingface, link in the comments. pull it down, run it on whatever 8gb card is sitting in your rig. try the game and tell me your score and what you want in v2. local llms on consumer gpus stopped being a meme.

Alok

60,866 views • 1 month ago