Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

7-YEAR-OLD RASPBERRY PI WITH 2GB OF RAM IS RUNNING DEEPSEEK RIGHT NOW WHILE SAM ALTMAN BILLS $200/MO AND DARIO AMODEI BILLS $200/MO FOR A MODEL YOU NEVER GET TO KEEP 00:34 he points at the smallest board on the bench and says "out of all these I'm genuinely shocked...

antisadh's profile picture

Antid

2,483 subscribers

10,442 Aufrufe • vor 11 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

7 Kommentare

Profilbild von twinedon
twinedonvor 11 Tagen

2GB and it still runs DeepSeek, wild

Profilbild von 724
724vor 11 Tagen

A 1.5b on a Pi is a fun stunt but you are right that it is a structure model, not an answers model. The real jump is 8b class on a used GPU, that is where local actually replaces a subscription.

Profilbild von Kata Seiko
Kata Seikovor 10 Tagen

Where can you get a system that will run on a Raspberry Pi, that will ask two systems (maybe Ollama with Qwen on one system and something else on the Mac) and then have that third system compare the answers, resolve inconsistencies and return the final answer to me?

Profilbild von rewind
rewindvor 11 Tagen

2gb is wild

Profilbild von Marin 😘
Marin 😘vor 11 Tagen

has he considered actually getting a job? this is unemployed content

Profilbild von Chajkovska
Chajkovskavor 10 Tagen

This is the part of the AI race nobody talks about enough. When powerful models run locally on 7-year-old hardware, AI stops being a subscription and starts becoming infrastructure anyone can own.

Profilbild von Pixel Eye Pictures (P.E.P) 🇨🇦
Pixel Eye Pictures (P.E.P) 🇨🇦vor 11 Tagen

Running “AI” lol super legit

Ähnliche Videos

THIS PORTABLE HACKER KIT PACKS A RASPBERRY PI 5 WITH 16GB OF RAM AND A 512GB NVME THAT ALSO RUNS LOCAL LLMs, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT MOBILE 00:08 he sets down the pi and starts the tour, "this is my raspberry pi 5, 16 gigs of ram, 512 gigabyte nvme ssd storage, this thing means business" the kit stacks the pi next to a bluetooth analyzer, a 5g wifi router with a battery, and a pen-testing dongle for packet injection, the whole rig runs off usb-c and fits in a shoulder bag he uses it for evil-twin wifi tests and bluetooth device audits, and the same pi 5 also handles ollama with a 7b or 14b model locally at usable speed with every packet capture staying inside the case claude code points at localhost through a single environment variable, the cli behaves identically to the cloud version except every token generates on the pi in his hand and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit the same card that pays his internet, the whole subscription stack died the day the pi got mobile and ai bills went to zero because the model runs on the same box that runs his packet capture bookmark this and read the article below
1:03

THIS PORTABLE HACKER KIT PACKS A RASPBERRY PI 5 WITH 16GB OF RAM AND A 512GB NVME THAT ALSO RUNS LOCAL LLMs, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT MOBILE 00:08 he sets down the pi and starts the tour, "this is my raspberry pi 5, 16 gigs of ram, 512 gigabyte nvme ssd storage, this thing means business" the kit stacks the pi next to a bluetooth analyzer, a 5g wifi router with a battery, and a pen-testing dongle for packet injection, the whole rig runs off usb-c and fits in a shoulder bag he uses it for evil-twin wifi tests and bluetooth device audits, and the same pi 5 also handles ollama with a 7b or 14b model locally at usable speed with every packet capture staying inside the case claude code points at localhost through a single environment variable, the cli behaves identically to the cloud version except every token generates on the pi in his hand and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit the same card that pays his internet, the whole subscription stack died the day the pi got mobile and ai bills went to zero because the model runs on the same box that runs his packet capture bookmark this and read the article below

starmex

25,094 Aufrufe • vor 2 Monaten

TWITTER ORDERED THESE EPYC CHIPS AND NEVER TOOK DELIVERY. HE BOUGHT THE ORPHANED BATCH, STACKED 512GB OF RAM AND NOW RUNS DEEPSEEK 671B UNQUANTIZED WHILE ANTHROPIC BILLS $200/MO FOR THE SAME CLASS OF MODEL 00:17 he holds the cpu up and says "these were custom made I think for twitter and then all the twitter stuff happened, so they never got put to use" amd epyc 7v13, the cancelled twitter batch -> supermicro h12ssl-nt, 8 memory channels -> 8x 64gb ddr4-3200, 512gb total -> noctua sp3 cooler -> gutted an old atx case -> the whole model loads into ram, no swap, no quantization his gaming rig has a 9950x3d and 128gb of ddr5 and only two memory channels, this board has eight, so older ddr4 on the epyc moves more data per second than the newer ram on the gaming machine, that's the whole reason cpu inference works at all here the honest part: he expects 1 to 4 tokens per second, which is slower than reading speed, you run this because it holds a 671b model at full precision, not because it's fast amd epyc 7v13 + supermicro h12ssl-nt + 512gb ddr4-3200 across 8 channels + noctua sp3 + deepseek 671b + llama 405b watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
2:51

TWITTER ORDERED THESE EPYC CHIPS AND NEVER TOOK DELIVERY. HE BOUGHT THE ORPHANED BATCH, STACKED 512GB OF RAM AND NOW RUNS DEEPSEEK 671B UNQUANTIZED WHILE ANTHROPIC BILLS $200/MO FOR THE SAME CLASS OF MODEL 00:17 he holds the cpu up and says "these were custom made I think for twitter and then all the twitter stuff happened, so they never got put to use" amd epyc 7v13, the cancelled twitter batch -> supermicro h12ssl-nt, 8 memory channels -> 8x 64gb ddr4-3200, 512gb total -> noctua sp3 cooler -> gutted an old atx case -> the whole model loads into ram, no swap, no quantization his gaming rig has a 9950x3d and 128gb of ddr5 and only two memory channels, this board has eight, so older ddr4 on the epyc moves more data per second than the newer ram on the gaming machine, that's the whole reason cpu inference works at all here the honest part: he expects 1 to 4 tokens per second, which is slower than reading speed, you run this because it holds a 671b model at full precision, not because it's fast amd epyc 7v13 + supermicro h12ssl-nt + 512gb ddr4-3200 across 8 channels + noctua sp3 + deepseek 671b + llama 405b watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

10,074 Aufrufe • vor 12 Tagen

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below
0:59

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

starmex

981,107 Aufrufe • vor 3 Monaten

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below
2:28

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

Antid

13,974 Aufrufe • vor 19 Tagen

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below
1:06

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below

starmex

93,580 Aufrufe • vor 3 Monaten

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes
0:37

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes

Antid

64,291 Aufrufe • vor 2 Monaten

A $40 BC250 BOARD WITH 16GB GDDR6 GETS A GITHUB FIRMWARE FLASH, THE DEFAULT 8/8 CPU/GPU SPLIT REWRITES TO 0.5/15.5 AND OPENS 15.5GB OF UNIFIED MEMORY TO OLLAMA, DEAD CRYPTO HARDWARE JUST DOUBLED ITS AI CEILING 01:08 the operator points at the chart, "you only give 512 megs to the GPU part and that's reserved, that means all the rest of the RAM is available to either the CPU or the GPU" a BC250 ships from the factory with 16GB GDDR6 split 8/8 between Oberon CPU cores and the RDNA 2 iGPU, the iGPU only ever uses 4-5GB during inference, the other 3-4GB sits stranded SEC Bolt's firmware mod on the moth-enjoyer GitHub flashes a 0.5/15.5 split, the GPU reserves 512MB and 15.5GB falls into a unified pool that ollama treats as VRAM, qwen 3.6 14B at 4 bit fits with 2GB context headroom 15.5GB on a $40 board with a $40 CH347 flasher beats the $249 Jetson Orin Nano's 8GB by nearly 2x at 1/6th the price, the firmware is open source, the dump backup process takes 4 minutes per board your map's tier zero floor was the OptiPlex at $35-50 with iGPU only, the modded BC250 sits at the same $40 mark with 15.5GB of GDDR6 and an actual RDNA 2 GPU, the buyer who flashes once unlocks a 13B class local AI host for the price of a dinner the window is open, follow and bookmark before it closes
2:38

A $40 BC250 BOARD WITH 16GB GDDR6 GETS A GITHUB FIRMWARE FLASH, THE DEFAULT 8/8 CPU/GPU SPLIT REWRITES TO 0.5/15.5 AND OPENS 15.5GB OF UNIFIED MEMORY TO OLLAMA, DEAD CRYPTO HARDWARE JUST DOUBLED ITS AI CEILING 01:08 the operator points at the chart, "you only give 512 megs to the GPU part and that's reserved, that means all the rest of the RAM is available to either the CPU or the GPU" a BC250 ships from the factory with 16GB GDDR6 split 8/8 between Oberon CPU cores and the RDNA 2 iGPU, the iGPU only ever uses 4-5GB during inference, the other 3-4GB sits stranded SEC Bolt's firmware mod on the moth-enjoyer GitHub flashes a 0.5/15.5 split, the GPU reserves 512MB and 15.5GB falls into a unified pool that ollama treats as VRAM, qwen 3.6 14B at 4 bit fits with 2GB context headroom 15.5GB on a $40 board with a $40 CH347 flasher beats the $249 Jetson Orin Nano's 8GB by nearly 2x at 1/6th the price, the firmware is open source, the dump backup process takes 4 minutes per board your map's tier zero floor was the OptiPlex at $35-50 with iGPU only, the modded BC250 sits at the same $40 mark with 15.5GB of GDDR6 and an actual RDNA 2 GPU, the buyer who flashes once unlocks a 13B class local AI host for the price of a dinner the window is open, follow and bookmark before it closes

Antid

25,619 Aufrufe • vor 2 Monaten

$35 QUAD-CORE ARM BOX STAPLED TO A HARBOR FREIGHT SOLAR PANEL IN HIS GARAGE RUNS A LOCAL LLM ON $60 OF CHARGE CONTROLLER GEAR AND A T-MOBILE 5G BACKUP LINK, KILLED $200 CHATGPT PRO AND $200 CLAUDE CODE MAX SUBSCRIPTIONS THE WEEK HE WENT OFF-GRID 00:03 marco tilts the phone toward the rig on his garage bench and says, "i am in my garage running off t-mobile's 5g network, got the ai up and running on the pi 4 with the main rain system in the background" the setup is a raspberry pi 4 wired to an inverter, a $60 charge controller and a small harbor freight solar panel, marco runs a quantized qwen 2.5 7b on ollama and the whole rig draws less than 15 watts off a 20ah lifepo4 battery when the sun is up the panel keeps the battery topped off, when it's cloudy the pi coasts on the pack for 8 hours, when he needs the internet the pi tethers to t-mobile 5g as backup and never asks the cloud for a single token claude code on his laptop points at the pi over tailscale, cli behaves identically to the cloud version except every token generates in his garage and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit his card monthly, both cancelled the day the panel started charging because his own hardware handles what those apis used to charge for bookmark this and read the article below
1:10

$35 QUAD-CORE ARM BOX STAPLED TO A HARBOR FREIGHT SOLAR PANEL IN HIS GARAGE RUNS A LOCAL LLM ON $60 OF CHARGE CONTROLLER GEAR AND A T-MOBILE 5G BACKUP LINK, KILLED $200 CHATGPT PRO AND $200 CLAUDE CODE MAX SUBSCRIPTIONS THE WEEK HE WENT OFF-GRID 00:03 marco tilts the phone toward the rig on his garage bench and says, "i am in my garage running off t-mobile's 5g network, got the ai up and running on the pi 4 with the main rain system in the background" the setup is a raspberry pi 4 wired to an inverter, a $60 charge controller and a small harbor freight solar panel, marco runs a quantized qwen 2.5 7b on ollama and the whole rig draws less than 15 watts off a 20ah lifepo4 battery when the sun is up the panel keeps the battery topped off, when it's cloudy the pi coasts on the pack for 8 hours, when he needs the internet the pi tethers to t-mobile 5g as backup and never asks the cloud for a single token claude code on his laptop points at the pi over tailscale, cli behaves identically to the cloud version except every token generates in his garage and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit his card monthly, both cancelled the day the panel started charging because his own hardware handles what those apis used to charge for bookmark this and read the article below

starmex

21,856 Aufrufe • vor 2 Monaten

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

341,499 Aufrufe • vor 22 Tagen

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
2:45

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

38,824 Aufrufe • vor 5 Tagen

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.
0:21

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 Aufrufe • vor 2 Monaten

What Lisa Su actually held on stage: A mini PC the size of a lunchbox running Qwen3-235B locally, with no cloud and no discrete GPU Inside: the Ryzen AI Max+ 395, 128GB unified memory, 110GB usable as VRAM on Linux The first x86 chip that handles 200+ billion parameters on a single die AMD claims it beats the RTX 5080 by several times on memory-bound models — because the 5080 simply cannot fit them $1,400 to $2,500 once. cloud bills run $200 to $400 a month It pays for itself in a few months, then costs nothing per request This is not a faster GPU. it is the first real argument that your AI does not belong in someone else's data center
0:31

What Lisa Su actually held on stage: A mini PC the size of a lunchbox running Qwen3-235B locally, with no cloud and no discrete GPU Inside: the Ryzen AI Max+ 395, 128GB unified memory, 110GB usable as VRAM on Linux The first x86 chip that handles 200+ billion parameters on a single die AMD claims it beats the RTX 5080 by several times on memory-bound models — because the 5080 simply cannot fit them $1,400 to $2,500 once. cloud bills run $200 to $400 a month It pays for itself in a few months, then costs nothing per request This is not a faster GPU. it is the first real argument that your AI does not belong in someone else's data center

plutos

1,237,742 Aufrufe • vor 3 Monaten

AMD on local AI: "this is the smallest AI development system in the world, capable of running models up to 200 billion parameters locally, not connected to anything" 128GB of unified memory shared by CPU, GPU, and NPU fits in your hand. Runs a model larger than GPT-3 with no internet $3,999 once. roughly $16 a month in power Cloud equivalent: up to $750 a month The question was never whether local AI could compete on power It was whether it could compete on price. Now it does
1:00

AMD on local AI: "this is the smallest AI development system in the world, capable of running models up to 200 billion parameters locally, not connected to anything" 128GB of unified memory shared by CPU, GPU, and NPU fits in your hand. Runs a model larger than GPT-3 with no internet $3,999 once. roughly $16 a month in power Cloud equivalent: up to $750 a month The question was never whether local AI could compete on power It was whether it could compete on price. Now it does

plutos

152,392 Aufrufe • vor 3 Monaten

NVIDIA SOLD THESE TESLA K80s FOR $5,000 EACH AND KILLED DRIVER SUPPORT FOR THEM IN 2018. HE PAID $578 FOR THE WHOLE 24-CORE WORKSTATION AND 48GB OF VRAM CAME WITH IT 00:12 he wipes three years of dust off the case and says "back in april of 2023 I picked up this workstation, it's a 24 core dual processor, no OS, no GPU, for 578" $578 dual xeon e5, 24 cores, 128gb of ram -> two tesla k80s at 24gb each, 48gb pooled -> six ball bearing fans already inside -> the k80s ship with no fans at all because datacenters ducted the air for them -> 3d print a shroud and hope it clears the cables -> drop a 3080 ti in the top x16 while the prints run each k80 draws 300 watts and is slower than a 3060 that costs a fraction of that, and every card presents as two separate gpus, so two cards land in your system as four devices that most inference stacks handle badly the honest part: he says outright he may pull them back out, dirt cheap vram is not free vram when it costs 600 watts and a weekend of 3d printing to make it breathe dual xeon e5 + 128gb ddr4 + 2x tesla k80 at 48gb pooled + 3080 ti 12gb + 3d printed fan shrouds + linux watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
8:29

NVIDIA SOLD THESE TESLA K80s FOR $5,000 EACH AND KILLED DRIVER SUPPORT FOR THEM IN 2018. HE PAID $578 FOR THE WHOLE 24-CORE WORKSTATION AND 48GB OF VRAM CAME WITH IT 00:12 he wipes three years of dust off the case and says "back in april of 2023 I picked up this workstation, it's a 24 core dual processor, no OS, no GPU, for 578" $578 dual xeon e5, 24 cores, 128gb of ram -> two tesla k80s at 24gb each, 48gb pooled -> six ball bearing fans already inside -> the k80s ship with no fans at all because datacenters ducted the air for them -> 3d print a shroud and hope it clears the cables -> drop a 3080 ti in the top x16 while the prints run each k80 draws 300 watts and is slower than a 3060 that costs a fraction of that, and every card presents as two separate gpus, so two cards land in your system as four devices that most inference stacks handle badly the honest part: he says outright he may pull them back out, dirt cheap vram is not free vram when it costs 600 watts and a weekend of 3d printing to make it breathe dual xeon e5 + 128gb ddr4 + 2x tesla k80 at 48gb pooled + 3080 ti 12gb + 3d printed fan shrouds + linux watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

49,589 Aufrufe • vor 10 Tagen

THIS 1-WATT RASPBERRY PI ON A MOUNTAIN TOP PIPES AI QUERIES OVER A PRIVATE MESH TO A LOCAL LLM AT HOME, KILLED $200 CHATGPT PRO AND $200 CLAUDE CODE MAX SUBSCRIPTIONS THE DAY IT WENT LIVE 01:25 he picks up the handheld and says, "this node is tough as nails and the battery life is around 10 days" the mountain pi sits a few kilometers from his apartment with a 915 mhz antenna tuned for long range, meshtastic firmware handles the packet routing between nodes on the mesh wireguard stitches the mountain pi to his home rig running qwen 3.6 27b on a used rtx 3090, ai prompts travel over the private mesh and answers come back the same way without ever touching the public internet the ollama stack handles his coding, writing and research from any point on the network, no cloud api reads his prompts and no rate limit hits on a friday launch because nothing runs on someone else's servers $200 chatgpt pro and $200 claude code max used to cover the queries he now sends over his own antennas, the whole subscription stack died the day the pi went live because the local rig handles what those apis used to charge for bookmark this and read the article below
1:44

THIS 1-WATT RASPBERRY PI ON A MOUNTAIN TOP PIPES AI QUERIES OVER A PRIVATE MESH TO A LOCAL LLM AT HOME, KILLED $200 CHATGPT PRO AND $200 CLAUDE CODE MAX SUBSCRIPTIONS THE DAY IT WENT LIVE 01:25 he picks up the handheld and says, "this node is tough as nails and the battery life is around 10 days" the mountain pi sits a few kilometers from his apartment with a 915 mhz antenna tuned for long range, meshtastic firmware handles the packet routing between nodes on the mesh wireguard stitches the mountain pi to his home rig running qwen 3.6 27b on a used rtx 3090, ai prompts travel over the private mesh and answers come back the same way without ever touching the public internet the ollama stack handles his coding, writing and research from any point on the network, no cloud api reads his prompts and no rate limit hits on a friday launch because nothing runs on someone else's servers $200 chatgpt pro and $200 claude code max used to cover the queries he now sends over his own antennas, the whole subscription stack died the day the pi went live because the local rig handles what those apis used to charge for bookmark this and read the article below

starmex

38,742 Aufrufe • vor 2 Monaten

Andrej Karpathy (co-founder of OpenAI, Eureka Labs): "The future of AI is not all in the cloud. The most important models will run locally on your hardware under your control." June 2026 is the first time that the future is actually accessible to everyone! Here is what the local LLM landscape looks like right now: → raspberry pi 5: coherent chatbot, no GPU required → macbook air: matches GPT-3.5 quality on most tasks natively → used RTX 3090 at $700: something close to GPT-4 running locally → ollama: one command to an openAI-compatible endpoint, every major toolchain integrates with it → LM studio: best GUI, native MLX acceleration on apple silicon, MCP support in 0.4.0 → llama.cpp: the engine everything else runs on, 90MB footprint, runs on raspberry pi to android → the honest case: 80% of daily work is genuinely good enough locally right now Your prompts never leave your machine. no per-token billing. no rate limits. no refusals. The playbook for every hardware tier, from Raspberry Pi to RTX 5090, is here. Bookmark it so you do not lose it! Follow NeilXbt for more local AI and open source intelligence that tracks the hardware and tooling most people discover six months late.
3:47

Andrej Karpathy (co-founder of OpenAI, Eureka Labs): "The future of AI is not all in the cloud. The most important models will run locally on your hardware under your control." June 2026 is the first time that the future is actually accessible to everyone! Here is what the local LLM landscape looks like right now: → raspberry pi 5: coherent chatbot, no GPU required → macbook air: matches GPT-3.5 quality on most tasks natively → used RTX 3090 at $700: something close to GPT-4 running locally → ollama: one command to an openAI-compatible endpoint, every major toolchain integrates with it → LM studio: best GUI, native MLX acceleration on apple silicon, MCP support in 0.4.0 → llama.cpp: the engine everything else runs on, 90MB footprint, runs on raspberry pi to android → the honest case: 80% of daily work is genuinely good enough locally right now Your prompts never leave your machine. no per-token billing. no rate limits. no refusals. The playbook for every hardware tier, from Raspberry Pi to RTX 5090, is here. Bookmark it so you do not lose it! Follow NeilXbt for more local AI and open source intelligence that tracks the hardware and tooling most people discover six months late.

NeilXbt

19,405 Aufrufe • vor 3 Monaten

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.
0:41

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.

RetroChainer

16,995 Aufrufe • vor 1 Monat

THIS SOLAR-POWERED MESH NODE PACKS A LOCAL LLM AND A 6-WATT PANEL, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT LIVE AND RUNS AT $0/MO EVEN UNDER CLOUD COVER 00:22 he taps the case with the red led blinking, "even though we have complete cloud cover today, the sun's not even out and we're still getting a charge, that's absolutely phenomenal" the node runs a rak 4630 lora board with a 6-watt solar panel, a small battery bank inside the ip67 case, and a raspberry pi zero 2w wedged next to it running a quantized 3b model on ollama from the rooftop it services 71 other meshtastic nodes across the neighborhood offline, and the same box also handles ai queries piped in over the mesh from his phone without touching the internet claude code points at the pi over tailscale, the cli behaves identically to the cloud version except every token generates on his own rooftop under the panel that keeps charging through cloud cover $200 chatgpt pro and $200 claude code max used to hit his card monthly, and the whole stack died the day the rooftop went live because ai queries now travel over lora mesh and generate on his own hardware for free bookmark this and read the article below
1:56

THIS SOLAR-POWERED MESH NODE PACKS A LOCAL LLM AND A 6-WATT PANEL, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT LIVE AND RUNS AT $0/MO EVEN UNDER CLOUD COVER 00:22 he taps the case with the red led blinking, "even though we have complete cloud cover today, the sun's not even out and we're still getting a charge, that's absolutely phenomenal" the node runs a rak 4630 lora board with a 6-watt solar panel, a small battery bank inside the ip67 case, and a raspberry pi zero 2w wedged next to it running a quantized 3b model on ollama from the rooftop it services 71 other meshtastic nodes across the neighborhood offline, and the same box also handles ai queries piped in over the mesh from his phone without touching the internet claude code points at the pi over tailscale, the cli behaves identically to the cloud version except every token generates on his own rooftop under the panel that keeps charging through cloud cover $200 chatgpt pro and $200 claude code max used to hit his card monthly, and the whole stack died the day the rooftop went live because ai queries now travel over lora mesh and generate on his own hardware for free bookmark this and read the article below

starmex

50,557 Aufrufe • vor 2 Monaten

a Tsinghua University lab just put a project on GitHub that replaces a $400,000 H100 rack with a single 24GB graphics card. it's called ktransformers, and the trick is almost stupidly simple: the experts you actually use stay on the gpu, the ones you don't sit on the cpu until they're called / deepseek-v3 and r1 with 139K context in 24gb of vram / up to 28x speedup over the standard setup / fine-tune deepseek-v3 across four rtx 4090s instead of a datacenter / built by tsinghua university's madsys lab, not a startup with a landing page apache 2.0, and already past 17,000 stars. -> bookmark it.
0:47

a Tsinghua University lab just put a project on GitHub that replaces a $400,000 H100 rack with a single 24GB graphics card. it's called ktransformers, and the trick is almost stupidly simple: the experts you actually use stay on the gpu, the ones you don't sit on the cpu until they're called / deepseek-v3 and r1 with 139K context in 24gb of vram / up to 28x speedup over the standard setup / fine-tune deepseek-v3 across four rtx 4090s instead of a datacenter / built by tsinghua university's madsys lab, not a startup with a landing page apache 2.0, and already past 17,000 stars. -> bookmark it.

dunik

313,656 Aufrufe • vor 1 Monat