Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and... show more

dunik

4,500 subscribers

111,308 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

37 Kommentare

Profilbild von Raichubuilds

Raichubuildsvor 3 Monaten

37373737 seconds for 1 token?

Profilbild von leo

leovor 3 Monaten

not a new concept, but just practically useless for doing any real llm driven tasks due to how slow it is. Cool tho

Profilbild von 0std | Pentester | AppSec

0std | Pentester | AppSecvor 3 Monaten

hahahha another 0.12 tokens/s repo, good luck doing anything with such speed

Profilbild von O(1)

O(1)vor 3 Monaten

BS, every single token requires re-reading essentially the entire model off disk. A 671B model at 4-bit is 350GB.. per token. Real-world reports put it at 5–30x slower than proper full-VRAM inference and 50–200x slower than APIs seconds-per-token or worse, with CPU pinned at 100%

Profilbild von Herminia

Herminiavor 3 Monaten

808 seguidores, 22.700 estrellas. La asimetría entre quién construye y quién tiene atención es casi más interesante que el logro técnico.

Profilbild von Le bureau Virtuel

Le bureau Virtuelvor 3 Monaten

Merci à lui

Profilbild von Atenov int.

Atenov int.vor 3 Monaten

i trust what was actually built into airllm

Profilbild von dunik

dunikvor 3 Monaten

yes, this project it's amazing

Profilbild von Abhishek Shinde

Abhishek Shindevor 3 Monaten

I guess github is the new place for AI research

Profilbild von AIAIO

AIAIOvor 3 Monaten

One would think 800 followers isn't that much for a project that seems to solve some of the biggest pain points of local AI. How fast does this thing run? More of a proof of concept or actually usable? I recall the port of a Linux kernel in a small 8 bit pic using emulation. Great idea. Just not usable.

Profilbild von Alex

Alexvor 3 Monaten

Never seen this GitHub before. I'll dig into it

Profilbild von dunik

dunikvor 3 Monaten

yep, start doing it right now!

Profilbild von DEV 0x***(Full Stack)💾

DEV 0x***(Full Stack)💾vor 3 Monaten

for me when i hear "bay area" i run and never look back lol

Profilbild von Dany Boy

Dany Boyvor 3 Monaten

it's not a "crazy" new thing. It's been doable from day 1. The problem with this is speed, concurrency, KV heads, quants etc. It all makes the model unusable for any real work.

Profilbild von 0xSlyth

0xSlythvor 3 Monaten

absolute madman to share this openly

Profilbild von SubThought Corp

SubThought Corpvor 3 Monaten

He didn't specialize on one content area for a target market. That was his mistake. Start with Finance or Real Estate next try.

Profilbild von porto4mil

porto4milvor 3 Monaten

@eltuganomics

Profilbild von 🤯

🤯vor 3 Monaten

1 tok/week

Profilbild von Dimuth Kurukula

Dimuth Kurukulavor 3 Monaten

@grok how is the approach here different from calibri for glm5.2

Profilbild von Chuck Petras

Chuck Petrasvor 3 Monaten

@BrianRoemmele

Profilbild von nanobro🎈

nanobro🎈vor 3 Monaten

Wen glm5.2 in dgx spark

Profilbild von 🅰🅽🆂🅷🆄🅻 🆂🅸🅽🅶🅷

🅰🅽🆂🅷🆄🅻 🆂🅸🅽🅶🅷vor 3 Monaten

1token per day?

Profilbild von Shesaidmewakeup

Shesaidmewakeupvor 3 Monaten

AirLLM is exactly the kind of tool that lets you get the most out of low-end hardware. That guy is really impressive. 🔥

Profilbild von AI Mastery Guide

AI Mastery Guidevor 3 Monaten

Fitting a 671B model into 12GB is a genuinely wild feat

Profilbild von Andrius Mostauskas

Andrius Mostauskasvor 3 Monaten

Will this work with LM Studio?

Profilbild von knopka_web3

knopka_web3vor 3 Monaten

Вы не проверяли что это за Airllm этот инструмент не работает.

Profilbild von Jörg Kachelofen

Jörg Kachelofenvor 3 Monaten

Have fun…

Profilbild von Qedge

Qedgevor 3 Monaten

airllm streams one layer at a time from cpu to gpu. That layer-wise swap gives us a clean hook for quantum-inspired prefetch on next token path.

Profilbild von Anotos

Anotosvor 3 Monaten

No. This thing is utter bullshit and is WAY too slow to be useful AT ALL. It generates a token PER MINUTE.

Profilbild von David

Davidvor 3 Monaten

@midudev

Profilbild von Young Sneed

Young Sneedvor 3 Monaten

holy based! where GGUF? chapeau to the legend open sourcing this!!

Profilbild von Nahid

Nahidvor 3 Monaten

airllms layer streaming is a wild approach

Profilbild von Marshall

Marshallvor 3 Monaten

How many seconds per token 😂

Profilbild von Ella Smith

Ella Smithvor 3 Monaten

This is amazing, we really need efficient LLMs now. Benchmarks should start talking about that now.

Profilbild von Shara

Sharavor 3 Monaten

Open sourcing instead of monetizing that breakthrough is a generous move.

Profilbild von SubThought Corp

SubThought Corpvor 3 Monaten

Why?

Profilbild von Saman Ahmed

Saman Ahmedvor 3 Monaten

I love these cases where one clever engineering choice changes who gets access to a technology.

Ähnliche Videos

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 Aufrufe • vor 3 Monaten

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

Brooks Whale X 🐋

40,529 Aufrufe • vor 19 Tagen

it's open source time, with a real leap for world models 🎉 NVIDIA's SANA-WM: a camera-conditioned world model that fits on one GPU. 60s of 720p in 34s on a single 5090 - 2.6B params and Apache 2.0!

it's open source time, with a real leap for world models 🎉 NVIDIA's SANA-WM: a camera-conditioned world model that fits on one GPU. 60s of 720p in 34s on a single 5090 - 2.6B params and Apache 2.0!

Victor M

34,386 Aufrufe • vor 5 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

345,385 Aufrufe • vor 2 Monaten

Daniel Han wrote Unsloth, the reason half of open-source can fine-tune a model on one GPU instead of a cluster. He didn't optimize the math. He rewrote the kernels by hand, found bugs in everyone else's code, and made training 2 to 3 times faster with zero accuracy loss. Millions of fine-tunes run through his code every month. Most people training a model locally are standing on it without knowing. Everyone talks about who has the most GPUs. He made yours enough.

Daniel Han wrote Unsloth, the reason half of open-source can fine-tune a model on one GPU instead of a cluster. He didn't optimize the math. He rewrote the kernels by hand, found bugs in everyone else's code, and made training 2 to 3 times faster with zero accuracy loss. Millions of fine-tunes run through his code every month. Most people training a model locally are standing on it without knowing. Everyone talks about who has the most GPUs. He made yours enough.

h100envy

30,856 Aufrufe • vor 4 Monaten

this is absolutely unhinged jack dorsey (twitter co-founder) just open-sourced a full ai-agent stack for operating a business and the repo is already sitting at 26.2k stars how to get it running: 1. clone the repo 2. host it yourself so channels, search, git, automation all run in one place 3. invite the agent into a channel like it’s another teammate, lock down its permissions, and steer it in real time with the crew save this and keep it bookmarked, seriously

this is absolutely unhinged jack dorsey (twitter co-founder) just open-sourced a full ai-agent stack for operating a business and the repo is already sitting at 26.2k stars how to get it running: 1. clone the repo 2. host it yourself so channels, search, git, automation all run in one place 3. invite the agent into a channel like it’s another teammate, lock down its permissions, and steer it in real time with the crew save this and keep it bookmarked, seriously

CyrilXBT

18,142 Aufrufe • vor 2 Monaten

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 Aufrufe • vor 4 Monaten

Gavin Baker: "which AI company actually survives? Google - already multi-trillion Microsoft, OpenAI, Meta chose open source - the only ones left is xAI and Anthropic - xAI + X will be worth over a trillion" this is him explaining what he actually invests in - and why Taiwan sells sand to Nvidia for $700 that Nvidia sells for $50,000 His words: what to invest in: "we're investing in things that increase the GPU utilization rate - because if you take a GPU from 30% utilized to 60%, you've doubled the output of that AI factory" what not to Invest in: "the models will basically go to zero - the data center is a commodity - energy might be short for a while but is also a commodity - the only thing in the whole stack with real value is the data moat and reinforcement learning" "the GPU has gotten 50 times faster in the last four years - that's a big part of why we're living through this AI revolution" he doesn't rent intelligence - he owns the thing it runs on bookmark & watch - see how below with local agent ↓

Gavin Baker: "which AI company actually survives? Google - already multi-trillion Microsoft, OpenAI, Meta chose open source - the only ones left is xAI and Anthropic - xAI + X will be worth over a trillion" this is him explaining what he actually invests in - and why Taiwan sells sand to Nvidia for $700 that Nvidia sells for $50,000 His words: what to invest in: "we're investing in things that increase the GPU utilization rate - because if you take a GPU from 30% utilized to 60%, you've doubled the output of that AI factory" what not to Invest in: "the models will basically go to zero - the data center is a commodity - energy might be short for a while but is also a commodity - the only thing in the whole stack with real value is the data moat and reinforcement learning" "the GPU has gotten 50 times faster in the last four years - that's a big part of why we're living through this AI revolution" he doesn't rent intelligence - he owns the thing it runs on bookmark & watch - see how below with local agent ↓

bodila

312,539 Aufrufe • vor 3 Monaten

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:

Slash AI costs by 99%, run 1000+ models on a single GPU! OpenLoRA lets you run thousands of models on one GPU instead of one per model, drastically cutting costs and boosting flexibility. Excited to work with Aethir, Hyperbolic for infrastructure and for verifiability. Watch the demo:

OpenLedger

155,363 Aufrufe • vor 1 Jahr

A hedge fund manager made $15 billion from a single trade - The greatest trade in Wall Street history in this 24-minute interview he explains how he shorted $25 billion in mortgages, lost money for months, and turned it into the greatest trade ever made Bookmark & Watch It. Then read the post below how the CEO of Citadel started with $265K and turned it into $90 billion ↓

A hedge fund manager made $15 billion from a single trade - The greatest trade in Wall Street history in this 24-minute interview he explains how he shorted $25 billion in mortgages, lost money for months, and turned it into the greatest trade ever made Bookmark & Watch It. Then read the post below how the CEO of Citadel started with $265K and turned it into $90 billion ↓

Сarm1ne

150,967 Aufrufe • vor 6 Monaten

This has been possible for months and barely anyone is doing it Months ago, Andrej Karpathy, a co-founder of OpenAI, open-sourced a project that gets you the best possible answers instead of trusting one model on its own. It's called LLM Council. Repo: /karpathy/llm-council The idea in two lines: your question goes to several models at once, Claude, GPT, Gemini, whatever you pick. They answer independently, then anonymously critique and rank each other (anonymous so they can't play favorites), and a chairman model merges the best of all of it into one final answer. One model just agrees with you; a council catches each other's mistakes. How to set it up: > Clone the repo > It uses uv for setup, install dependencies with uv sync > Get an OpenRouter API key and add it, this is what lets one app reach Claude, GPT, and Gemini together > Pick which models sit on your council in the config, then run it. It opens as a local web app, like ChatGPT but with a whole panel answering Bookmark this

This has been possible for months and barely anyone is doing it Months ago, Andrej Karpathy, a co-founder of OpenAI, open-sourced a project that gets you the best possible answers instead of trusting one model on its own. It's called LLM Council. Repo: /karpathy/llm-council The idea in two lines: your question goes to several models at once, Claude, GPT, Gemini, whatever you pick. They answer independently, then anonymously critique and rank each other (anonymous so they can't play favorites), and a chairman model merges the best of all of it into one final answer. One model just agrees with you; a council catches each other's mistakes. How to set it up: > Clone the repo > It uses uv for setup, install dependencies with uv sync > Get an OpenRouter API key and add it, this is what lets one app reach Claude, GPT, and Gemini together > Pick which models sit on your council in the config, then run it. It opens as a local web app, like ChatGPT but with a whole panel answering Bookmark this

Yarchi

21,488 Aufrufe • vor 4 Monaten

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.

RetroChainer

16,995 Aufrufe • vor 2 Monaten

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

LeRobot

23,450 Aufrufe • vor 3 Monaten

Woow Nvidia has just released a 2.6B open-source world model 🔥 You can turn a single image, text prompt and trajectory into controllable worlds... And on a single GPU! - Code available on GitHub - Paper as well on arxiv You can use it for many things like embodied AI and robotics research, simulations, etc. Because it can run on a single GPU (like an RTX 5090 or H100) it makes world models accessible to basically everyone!

Woow Nvidia has just released a 2.6B open-source world model 🔥 You can turn a single image, text prompt and trajectory into controllable worlds... And on a single GPU! - Code available on GitHub - Paper as well on arxiv You can use it for many things like embodied AI and robotics research, simulations, etc. Because it can run on a single GPU (like an RTX 5090 or H100) it makes world models accessible to basically everyone!

Paul Couvert

174,537 Aufrufe • vor 5 Monaten

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 Aufrufe • vor 4 Monaten

Tomorrow I'm sitting down with Riley Meik (Riley Meik), founder and CEO of The American Housing Corporation. He came out of aerospace, with a stint at SpaceX, before turning his attention to the housing crisis. Most people assume a company like his is trying to reinvent the entire industry. It isn't. "In hard tech, you only want to have to perform one miracle at a time. And if your business relies on multiple miracles, your odds of success are significantly lower." So they picked one. The way the homes are actually built. "We are going to do one thing different, and that's the way we build our homes. When you're inside of it, you shouldn't really know that it was built in a factory. It just looks like a nice house." Everything else, the financing, the land, the banks, he keeps about as conventional as it gets. In tomorrow's clip, Riley breaks down the focus behind doing one thing differently and nothing else.

Tomorrow I'm sitting down with Riley Meik (Riley Meik), founder and CEO of The American Housing Corporation. He came out of aerospace, with a stint at SpaceX, before turning his attention to the housing crisis. Most people assume a company like his is trying to reinvent the entire industry. It isn't. "In hard tech, you only want to have to perform one miracle at a time. And if your business relies on multiple miracles, your odds of success are significantly lower." So they picked one. The way the homes are actually built. "We are going to do one thing different, and that's the way we build our homes. When you're inside of it, you shouldn't really know that it was built in a factory. It just looks like a nice house." Everything else, the financing, the land, the banks, he keeps about as conventional as it gets. In tomorrow's clip, Riley breaks down the focus behind doing one thing differently and nothing else.

Chris Powers

13,829 Aufrufe • vor 4 Monaten

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.

watch anon! 27 billion parameter model dancing at 42 tokens a second on a used rtx 3060ti, that costs 200 dollars. consuming 159 watts. temp 62 degrees. pushing almost 100% on the gpu. 6.8 of 8 gigs used with a 128k context window loaded. and every one of those spikes is the model reasoning and firing tool calls. the model is bonsai 27b at 1 bit. PrismML took qwen 3.6 27b, and crushed it to 3.9 gigabytes, about a single sign bit per weight. it drops onto an 8gb rtx 3060 ti, an ampere card from 2020, and it doesn't just load, it serves, it holds the full 128k window, it runs a hermes agent for half an hour straight, it hits its own failures and recovers, it never once drops the loop. the absurd part is the power meter. that same 27b, in its normal quant 4, wants a 24gb card and nearly 17 gigs of weights. this is it at 3.9, doing agentic work at 160 watts, on the tier people call budget. the numbers are sitting right there on nvtop, not in a slide, not a claim, the meter reading while it works. so i keep landing on one question. the model is open, apache 2.0. the card is old and cheap. the whole thing runs offline. why is almost nobody actually running this. is it the hardware, the setup, or did nobody tell you a 27b fits on the card already in your closet.

Sudo su

23,691 Aufrufe • vor 3 Monaten

an engineer openai laid off in january open-sourced the exact thing that made gpt-6 astra worth $200 a month, three weeks after they walked him out. call it revenge, call it the most expensive severance package in history. sam altman spent two years telling everyone the magic was locked in the cloud. one bitter ex-employee proved it fits in a github repo, and now the $200 seat is guarding a moat that is already public. here is what he leaked, and it runs local for about $6 instead of $400: -> pull his open repo, load it onto one nvidia card, and you have the part of gpt-6 astra people were actually paying for -> point cursor, cline and every app at localhost, and none of them notice they left openai for a model that never phones home -> route the rare heavy job to a rented gpu by the hour and keep the daily work local for the cost of power -> your $200 chatgpt seat and your $200 claude seat both go dark the same week the repo trended here is the part they will fight me on: openai and anthropic were never selling the model, they were selling the fact that it lived somewhere you could not reach. one guy they underpaid moved it somewhere you can. the $200 was the distance, and he deleted the distance. drop your $400 stack to a $6 local model and run the thing they fired him for sharing. the full build is in the article below, save it before they get the repo taken down.

an engineer openai laid off in january open-sourced the exact thing that made gpt-6 astra worth $200 a month, three weeks after they walked him out. call it revenge, call it the most expensive severance package in history. sam altman spent two years telling everyone the magic was locked in the cloud. one bitter ex-employee proved it fits in a github repo, and now the $200 seat is guarding a moat that is already public. here is what he leaked, and it runs local for about $6 instead of $400: -> pull his open repo, load it onto one nvidia card, and you have the part of gpt-6 astra people were actually paying for -> point cursor, cline and every app at localhost, and none of them notice they left openai for a model that never phones home -> route the rare heavy job to a rented gpu by the hour and keep the daily work local for the cost of power -> your $200 chatgpt seat and your $200 claude seat both go dark the same week the repo trended here is the part they will fight me on: openai and anthropic were never selling the model, they were selling the fact that it lived somewhere you could not reach. one guy they underpaid moved it somewhere you can. the $200 was the distance, and he deleted the distance. drop your $400 stack to a $6 local model and run the thing they fired him for sharing. the full build is in the article below, save it before they get the repo taken down.

starmex

12,339 Aufrufe • vor 1 Monat