Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

TWITTER ORDERED THESE EPYC CHIPS AND NEVER TOOK DELIVERY. HE BOUGHT THE ORPHANED BATCH, STACKED 512GB OF RAM AND NOW RUNS DEEPSEEK 671B UNQUANTIZED WHILE ANTHROPIC BILLS $200/MO FOR THE SAME CLASS OF MODEL 00:17 he holds the cpu up and says "these were custom made I think for...

antisadh's profile picture

Antid

2,482 subscribers

10,074 Aufrufe • vor 12 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von dragonfly
dragonflyvor 12 Tagen

Buy at least one GPU for the most heavily used parts of the MoE LLM, this will improve performance dramatically

Profilbild von marcinu
marcinuvor 12 Tagen

Wouldn't that draw some obscene wattage even idle?

Profilbild von Mura
Muravor 12 Tagen

that will make your computer faster

Profilbild von 𝚜𝚘𝚏𝚝𝚏𝚕𝚘𝚠𝟿𝚟𝚓𝚌
𝚜𝚘𝚏𝚝𝚏𝚕𝚘𝚠𝟿𝚟𝚓𝚌vor 6 Tagen

When everyone runs from AMD and calls it trash, this guy embraces it

Profilbild von rewind
rewindvor 12 Tagen

gutting my atx case now

Ähnliche Videos

7-YEAR-OLD RASPBERRY PI WITH 2GB OF RAM IS RUNNING DEEPSEEK RIGHT NOW WHILE SAM ALTMAN BILLS $200/MO AND DARIO AMODEI BILLS $200/MO FOR A MODEL YOU NEVER GET TO KEEP 00:34 he points at the smallest board on the bench and says "out of all these I'm genuinely shocked that I actually got it up and running on this raspberry pi 4b with 2 gigs of ram" raspberry pi 4b, 2gb, seven years old -> ollama with deepseek r1 1.5b -> a dell micro form factor doing cpu-only inference next to it -> an m1 max macbook with 64gb unified memory beating the dual gpu tower on large models -> four machines, four architectures, all serving locally apple's unified memory puts the ram on the same die as the cpu and gpu, so a laptop outruns a two-card workstation on big models, and the cheapest board on that bench costs less than four days of a claude max seat the honest part: 2gb is the actual floor and he says it was rough to get there, a 4gb or 8gb pi would have taken minutes instead, and 1.5b parameters is a model you use for structure not for answers raspberry pi 4b 2gb + dell micro form factor + m1 max 64gb + dual gpu tower + ollama + deepseek r1 watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
3:16

7-YEAR-OLD RASPBERRY PI WITH 2GB OF RAM IS RUNNING DEEPSEEK RIGHT NOW WHILE SAM ALTMAN BILLS $200/MO AND DARIO AMODEI BILLS $200/MO FOR A MODEL YOU NEVER GET TO KEEP 00:34 he points at the smallest board on the bench and says "out of all these I'm genuinely shocked that I actually got it up and running on this raspberry pi 4b with 2 gigs of ram" raspberry pi 4b, 2gb, seven years old -> ollama with deepseek r1 1.5b -> a dell micro form factor doing cpu-only inference next to it -> an m1 max macbook with 64gb unified memory beating the dual gpu tower on large models -> four machines, four architectures, all serving locally apple's unified memory puts the ram on the same die as the cpu and gpu, so a laptop outruns a two-card workstation on big models, and the cheapest board on that bench costs less than four days of a claude max seat the honest part: 2gb is the actual floor and he says it was rough to get there, a 4gb or 8gb pi would have taken minutes instead, and 1.5b parameters is a model you use for structure not for answers raspberry pi 4b 2gb + dell micro form factor + m1 max 64gb + dual gpu tower + ollama + deepseek r1 watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

10,442 Aufrufe • vor 11 Tagen

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.
0:21

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

151,882 Aufrufe • vor 2 Monaten

NVIDIA SOLD THESE TESLA K80s FOR $5,000 EACH AND KILLED DRIVER SUPPORT FOR THEM IN 2018. HE PAID $578 FOR THE WHOLE 24-CORE WORKSTATION AND 48GB OF VRAM CAME WITH IT 00:12 he wipes three years of dust off the case and says "back in april of 2023 I picked up this workstation, it's a 24 core dual processor, no OS, no GPU, for 578" $578 dual xeon e5, 24 cores, 128gb of ram -> two tesla k80s at 24gb each, 48gb pooled -> six ball bearing fans already inside -> the k80s ship with no fans at all because datacenters ducted the air for them -> 3d print a shroud and hope it clears the cables -> drop a 3080 ti in the top x16 while the prints run each k80 draws 300 watts and is slower than a 3060 that costs a fraction of that, and every card presents as two separate gpus, so two cards land in your system as four devices that most inference stacks handle badly the honest part: he says outright he may pull them back out, dirt cheap vram is not free vram when it costs 600 watts and a weekend of 3d printing to make it breathe dual xeon e5 + 128gb ddr4 + 2x tesla k80 at 48gb pooled + 3080 ti 12gb + 3d printed fan shrouds + linux watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
8:29

NVIDIA SOLD THESE TESLA K80s FOR $5,000 EACH AND KILLED DRIVER SUPPORT FOR THEM IN 2018. HE PAID $578 FOR THE WHOLE 24-CORE WORKSTATION AND 48GB OF VRAM CAME WITH IT 00:12 he wipes three years of dust off the case and says "back in april of 2023 I picked up this workstation, it's a 24 core dual processor, no OS, no GPU, for 578" $578 dual xeon e5, 24 cores, 128gb of ram -> two tesla k80s at 24gb each, 48gb pooled -> six ball bearing fans already inside -> the k80s ship with no fans at all because datacenters ducted the air for them -> 3d print a shroud and hope it clears the cables -> drop a 3080 ti in the top x16 while the prints run each k80 draws 300 watts and is slower than a 3060 that costs a fraction of that, and every card presents as two separate gpus, so two cards land in your system as four devices that most inference stacks handle badly the honest part: he says outright he may pull them back out, dirt cheap vram is not free vram when it costs 600 watts and a weekend of 3d printing to make it breathe dual xeon e5 + 128gb ddr4 + 2x tesla k80 at 48gb pooled + 3080 ti 12gb + 3d printed fan shrouds + linux watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

49,589 Aufrufe • vor 10 Tagen

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below
2:45

NVIDIA CHARGES $4,000 FOR AN RTX 5090 WITH 32GB. INTEL SELLS THE SAME 32GB FOR $950. HE STACKED TWO INTO 64GB AND KILLED $200/MO CLAUDE CODE AND $200/MO CHATGPT PRO FOR $3 IN ELECTRICITY 00:09 he's been at it since the middle of the night and admits it up front, "I've been testing these pretty much non stop since about 2am today" 2x intel arc pro b70 at $950 each -> 64gb pooled for less than half a 5090 -> asrock taichi lite, both in x8 slots -> ryzen 9600, 64gb ddr5 -> ubuntu 26.04 sees both cards instantly, zero driver hunting -> intel's own stack, not community packages -> mixture of experts models fly nvidia wants $4,000 for 32gb, intel put the same capacity in a card at 23% of that, so a 70b rig went from a $4,000 ticket to $1,900 for double the memory, and the $400 a month altman and amodei charge becomes $3 in electricity the honest part: q8 hits a hard bottleneck on battlemage, bizarre because intel's older alchemist cards were faster at q8 than q4, dense models are rough, and running both cards together has been ugly so far 2x intel arc pro b70 32gb + asrock taichi lite + ryzen 9600 + 64gb ddr5 + ubuntu 26.04 + intel's own inference stack watch and save it, musk gave away the agent and moonshot gave away the weights, the full $400 to $25 routing table is in the article below

Antid

38,824 Aufrufe • vor 5 Tagen

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below
2:28

ALIBABA GAVE QWEN AWAY FREE, ZUCKERBERG GAVE LLAMA AWAY FREE, AND JENSEN HUANG STILL CHARGES $300 FOR THE 12GB THAT RUNS THEM. THE $80 AMD BOARD HAS 16GB. HE IS TESTING BOTH THIS WEEK AGAINST $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO 00:19 he points at a dead gtx 680 he keeps on the shelf purely for decoration and says "i'm actually really gpu poor" $300 rtx 3060 with 12gb -> $80 bc250 with 16gb -> ubuntu server on both -> 8gb tier, 12 to 16 tier, 24gb ceiling on a four year old 3090 -> qwen against llama at every tier -> tokens per dollar published jensen shipped 12gb on the 3060 in 2021 and consumer vram has barely moved since, alibaba and meta gave the weights away for nothing, the models are free and the only thing left to buy is memory the honest part: he says it himself, this is apples and oranges, and there is not a single benchmark number in this video yet $80 amd bc250 with 16gb gddr6 + $300 rtx 3060 with 12gb + a four year old 3090 with 24gb + ubuntu server + ollama + qwen + llama watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

Antid

13,974 Aufrufe • vor 19 Tagen

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.
0:41

AMD JUST KILLED THE $200 - A - MONTH AI SUBSCRIPTION - A 65-WATT BOX WITH 128GB OF UNIFIED MEMORY THAT RUNS 70B MODELS LOCALLY. that clip is amd's ryzen ai max+ 395 mini box, held up on stage. the whole pitch is one spec: 128gb of high-speed unified memory. why it matters: cpu, gpu and npu share the same 128gb pool, so a model that used to need a rack of vram just fits. what it actually runs: llama 70b in full precision on the built-in gpu open models up to ~100b parameters, locally, no cloud all of it in a mini pc pulling around 65 watts that's the part that ends the subscription: your prompts, your data, your model, on a box on your desk, not metered by the token. honest line: this runs open models, not gpt-5 or claude opus. for chat, coding help and agents on your own data it's more than enough. for the absolute frontier, the cloud still wins. and it isn't free: the box runs ~$1,500 and up. but that's a few months of a $200 sub, then it's yours forever. no monthly ai bill, no tokens metered, no prompt leaving the room. the uncomfortable part isn't that amd made a fast box. it's that "you need the cloud for ai" was a pricing choice, and unified memory just made it optional. follow RetroChainer for the hardware that turns rent into ownership, and bookmark this before your next ai renewal.

RetroChainer

16,995 Aufrufe • vor 1 Monat

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below
0:59

AMD CEO LISA SU HELD A MINI PC ON STAGE THAT RUNS A 235B MODEL AND REPLACES YOUR $440/MONTH AI STACK amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on one piece of silicon. cpu and gpu share 128gb of unified memory, no separate graphics card needed the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months install ollama, pull the model, point claude code at localhost. same interface, nothing leaves the machine, nothing costs per request bookmark this and read the article below

starmex

981,107 Aufrufe • vor 3 Monaten

INTEL SELLS THIS CHIP FOR $60 TO CASH REGISTERS. HE WIPED WINDOWS 11 WITH ONE COMMAND AND KILLED $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO FOR $5 OF ELECTRICITY 02:10 he drags the image file straight into the terminal window and says "next up is where the magic happens, we're going to use the dd command" $150 n97 mini pc -> one switch on the back, top panel off with no screws -> one screw and the 512gb windows drive comes out intact -> diskutil unmount /dev/disk9 -> sudo dd if=image of=/dev/disk9 bs=1m -> 9.13gb written -> secure boot disabled in the bios -> ubuntu and ollama up in 4 minutes microsoft says you need a 40 tops npu before it counts as an ai pc, this box has zero tops and no gpu, and the same $60 chip that scans your groceries now answers your prompts for less than one coffee a month the honest part: with no npu a 7b model runs on cpu at reading speed and nothing above 7b moves at all, this is not a frontier experience, it is a $5 one $150 n97 mini pc + 16gb ddr5 + 512gb nvme + one dd command + secure boot off + ubuntu + ollama + qwen 2.5 7b watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below
7:11

INTEL SELLS THIS CHIP FOR $60 TO CASH REGISTERS. HE WIPED WINDOWS 11 WITH ONE COMMAND AND KILLED $200/MO CLAUDE CODE MAX AND $200/MO CHATGPT PRO FOR $5 OF ELECTRICITY 02:10 he drags the image file straight into the terminal window and says "next up is where the magic happens, we're going to use the dd command" $150 n97 mini pc -> one switch on the back, top panel off with no screws -> one screw and the 512gb windows drive comes out intact -> diskutil unmount /dev/disk9 -> sudo dd if=image of=/dev/disk9 bs=1m -> 9.13gb written -> secure boot disabled in the bios -> ubuntu and ollama up in 4 minutes microsoft says you need a 40 tops npu before it counts as an ai pc, this box has zero tops and no gpu, and the same $60 chip that scans your groceries now answers your prompts for less than one coffee a month the honest part: with no npu a 7b model runs on cpu at reading speed and nothing above 7b moves at all, this is not a frontier experience, it is a $5 one $150 n97 mini pc + 16gb ddr5 + 512gb nvme + one dd command + secure boot off + ubuntu + ollama + qwen 2.5 7b watch and save it, then kill your $439 stack this weekend, kimi does the same work at $19 flat and the whole map is in the article below

Antid

24,281 Aufrufe • vor 20 Tagen

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
0:42

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

Akshay 🚀

341,701 Aufrufe • vor 22 Tagen

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓
1:22

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 Aufrufe • vor 2 Monaten

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.
2:18

i'm running a 397 billion parameter model on a amd ai max box that sits on my desk and pulls less power than a gaming laptop. the model is Nex-N2-Pro, 397B-A17B, the open weight release people are putting next to gpt-5.5 on coding. i have it quantized to IQ1_M, 1.75 bits per weight, 90gb of weights loaded into the 128gb of unified memory on amd's strix halo igpu. watch the gpu in this recording. it spikes, it sustains, it does not fall over. that is the part the spec sheets never show you, not just that a 400b model loads, but that an integrated graphics chip holds the load and generates token after token, stable, no crash, no thermal cliff. and it is not a slideshow. roughly 18 tokens a second, faster than you can read. a frontier scale model producing usable output, fully local. no datacenter, no rented h100s, no api key, no permission. three years ago a model this size meant a server room and a budget to match. tonight it is a quiet box on my desk. this is the accessible tier almost nobody benchmarks honestly, and it is further along than the timeline thinks. the full breakdown is coming, rocm vs vulkan on this chip, and this little amd box head to head against the nvidia equivalent. stay tuned.

Sudo su

32,163 Aufrufe • vor 3 Monaten

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below
1:06

NVIDIA'S BIGGEST COMPETITOR AMD BUILT A BOX THAT RUNS A 235B MODEL AND KILLS $200 OPENAI AND $200 CLAUDE CODE FOR $9 A MONTH amd's ryzen ai max+ 395 is the first x86 chip that runs a 200 billion parameter model on a single piece of silicon with 128gb of unified memory the gmktec evo-x2 runs qwen3 235b fully, deepseek v3 comfortably and llama 3.3 70b with headroom. on linux you get 110gb of usable vram out of 128gb amd claimed the chip beat an nvidia rtx 5080 by more than 3x on deepseek r1 inference. a lunchbox sized pc outrunning a $1,000 discrete gpu on a real ai workload a heavy ai user pays $200 for claude code max, $200 for chatgpt pro, $20 for cursor and $20 for gemini. that's $5,280 a year and the box pays itself off in 9 to 10 months ollama installs in one command and claude code points at localhost so nothing leaves the machine and nothing costs per request bookmark this and read the article below

starmex

93,580 Aufrufe • vor 3 Monaten

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes
0:37

ONE BUILDER WIRED 4 RASPBERRY PI MODULES INTO A €450 AI CLUSTER WITH 64GB OF RAM AND KILLED HIS $200/MONTH CHATGPT SUBSCRIPTION 00:37 he points at the terminal, "loaded, loaded, loaded - 4 nodes synced over gigabit, 64 gigabytes total, llama 3.1 partitioned across the cluster" he stacked 4 raspberry pi CM5 lite modules with 16 gigabytes each onto a sipeed nanocluster board, the whole rig runs off a single 65 watt power supply and a gigabit internal switch distributed llama partitions the model across the nodes with synchronized workloads handled over the internal fabric, no quantization needed, the cluster hits about 30 tokens per second on small models a heavy developer pays $200 a month for chatgpt pro and another $200 for claude code, this rig cost €450 in parts and breaks even on the stack in month 2 most people will keep paying anthropic and openai forever, a few will spend a weekend wiring 4 raspberry pi modules and never see a subscription invoice again the window is open, follow and bookmark before it closes

Antid

64,291 Aufrufe • vor 2 Monaten

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.
1:16

This Chinese developer linked two $2,999 NVIDIA DGX Sparks into one box and runs the full Qwen3-235B at home, after dropping his $1,999-a-month cloud bill to zero. He wired 2 small boxes into a single computer, split a giant 235-billion-parameter model in half between them, and serves it across his own network at about 10 tokens a second, with no internet, no cloud, right there on the desk. No data center, no thousand-dollar graphics cards, no monthly cloud bill. Just him, 2 gold boxes the size of a sandwich, one cable between them, and 1 power strip. And here is the whole payoff. He used to pay the cloud $1,999 a month for the same model, and the meter ticked on every request. Now he paid $5,998 once for 2 boxes, they covered their cost in 3 months, and after that he sends as many requests as he wants for free, only electricity. The two Sparks talk over one fast cable, each holds 128GB of memory, and together they carry the whole model, about 73GB loaded per box, with the chip inside pinned near the limit at 96%. Both boxes work as one and keep trading data over the cable, with no cloud in the loop and no single word leaking out. The ready model sits on one local address, and any app on his network calls it as easily as ChatGPT. And here is how he described, in plain words, what this pair of boxes does: "this is a pair of boxes that holds the huge Qwen3-235B model and serves it to one network. the model is split in half, and each box owns its half. parts: // Box 1 (holds the first half of the model and starts the answer fast, the first word appears in under a second) // Box 2 (holds the second half and writes out the rest, about 10 tokens a second) // Cable (connects the 2 boxes and moves data between them on every step, with no lag) // Address (one local address where any app sends its request, like to a cloud model) // Test (a script that runs big prompts through and measures speed and delays) // Monitor (checks temperature, power draw, and load on both boxes every 2 seconds). the model never goes to the cloud. he only steps in when a box runs hotter than 80 degrees or the cable between them starts dropping data." So the system knows exactly what it is, what it is for, and where its limits are. It knows it has to hold the whole huge model across 2 boxes on its own. It knows it has to answer every request locally, with no meter, no limits, and no internet. It knows the human is only needed when a box overheats or the link between them stalls. → The setup runs around the clock on 2 boxes, each pulling under 60 watts → However many requests he sends, the monthly bill is $0, only electricity → The first box starts the answer in under a second → The second writes text at about 10 tokens a second → One request at a time: 838 tokens in 85 seconds, first word in 0.8s → Two requests at once: 697 tokens in 108 seconds, first word in 0.7s → Both boxes sit at 96% load and warm up to 76-78 degrees And only when a chip in a box runs hotter than 80 degrees or the cable between the 2 Sparks drops data does the system call the owner. And when he himself is out on a run or in a coffee shop, he still reaches his own model at home from his phone: sends a big prompt to the local Qwen3-235B, gets the full answer back in under a minute and a half, with no token meter ticking and no limit to hit. Here is what the test shows on his screen during one of the night runs: "one request at a time: 838 tokens in 84.9 seconds, first word in 0.8s, then 0.1s per token." "two requests at once: 697 tokens in 107.6 seconds, first word in 0.7s, then 0.15s per token." "Box 1: chip at 96% load, 76 degrees, 56 watts, 73GB used in memory." "Box 2: chip at 96% load, 78 degrees, 56 watts, the Qwen3-235B model fully loaded." And while everyone around is paying for AI by the month and bumping into limits, his top-tier model just sits on the desk and works as much as he wants: his own little power plant instead of a forever meter. He has no server rack of his own and no cloud account behind it. Just 2 DGX Spark boxes on a desk, one model split in half between them, one local address, and a folder of prompts next to it. Out of everything I have seen this year, this is the cleanest way to stop paying for AI: $5,998 of hardware on the desk once, $0 a month to the cloud, unlimited forever, and between them 2 gold boxes, 1 cable, and the full Qwen3-235B answering at home with no internet.

Blaze

93,871 Aufrufe • vor 3 Monaten

THIS PORTABLE HACKER KIT PACKS A RASPBERRY PI 5 WITH 16GB OF RAM AND A 512GB NVME THAT ALSO RUNS LOCAL LLMs, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT MOBILE 00:08 he sets down the pi and starts the tour, "this is my raspberry pi 5, 16 gigs of ram, 512 gigabyte nvme ssd storage, this thing means business" the kit stacks the pi next to a bluetooth analyzer, a 5g wifi router with a battery, and a pen-testing dongle for packet injection, the whole rig runs off usb-c and fits in a shoulder bag he uses it for evil-twin wifi tests and bluetooth device audits, and the same pi 5 also handles ollama with a 7b or 14b model locally at usable speed with every packet capture staying inside the case claude code points at localhost through a single environment variable, the cli behaves identically to the cloud version except every token generates on the pi in his hand and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit the same card that pays his internet, the whole subscription stack died the day the pi got mobile and ai bills went to zero because the model runs on the same box that runs his packet capture bookmark this and read the article below
1:03

THIS PORTABLE HACKER KIT PACKS A RASPBERRY PI 5 WITH 16GB OF RAM AND A 512GB NVME THAT ALSO RUNS LOCAL LLMs, KILLED $200 CLAUDE CODE MAX AND $200 CHATGPT PRO SUBSCRIPTIONS THE DAY IT WENT MOBILE 00:08 he sets down the pi and starts the tour, "this is my raspberry pi 5, 16 gigs of ram, 512 gigabyte nvme ssd storage, this thing means business" the kit stacks the pi next to a bluetooth analyzer, a 5g wifi router with a battery, and a pen-testing dongle for packet injection, the whole rig runs off usb-c and fits in a shoulder bag he uses it for evil-twin wifi tests and bluetooth device audits, and the same pi 5 also handles ollama with a 7b or 14b model locally at usable speed with every packet capture staying inside the case claude code points at localhost through a single environment variable, the cli behaves identically to the cloud version except every token generates on the pi in his hand and the api bill never arrives $200 chatgpt pro and $200 claude code max used to hit the same card that pays his internet, the whole subscription stack died the day the pi got mobile and ai bills went to zero because the model runs on the same box that runs his packet capture bookmark this and read the article below

starmex

25,094 Aufrufe • vor 2 Monaten

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.
1:08

I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. They leveraged a hidden efficiency gap in how GPUs generate tokens. Kog just achieved 3,000 tokens/s on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). Their tech preview is on a 2B model, and they show how their techniques will scale to large frontier MoE models at similar speeds. That's a huge number because normal low-batch GPU decoding for 2B to 8B models is usually closer to 100 to 300 tokens/s per request, so Kog is claiming something like a 10X to 30X jump in the speed one user actually feels. Their trick: they are getting the speed by treating LLM decoding as a memory streaming problem, not mainly a math problem. For 1 user at batch size 1, the GPU is not doing big, efficient matrix-matrix work like in training or large-batch serving; it is repeatedly pulling the model’s active weights from high-bandwidth memory for each new token, so speed depends on how smoothly those weights keep flowing. Normal inference stacks keep breaking that flow. They run many separate GPU programs for different parts of the model, move intermediate results through memory, wait at synchronization points, talk back to the CPU for scheduling or sampling, and then repeat this token after token. Kog’s answer is to co-design 3 things that are usually tuned separately: the runtime, the low-level GPU code, and the model architecture. The biggest engineering move is the monokernel, where the whole decode pass runs as 1 persistent GPU-resident program, including sampling, so the system does not keep stopping for kernel launches, CPU scheduling, and intermediate memory round trips. They also rebuilt synchronization, because their own measurements say grid sync was eating around 35% of token-generation time; instead of making every compute unit wait at a broad barrier, each unit waits only for the exact data it needs. On AMD MI300X, they also map memory access around the chiplet layout, because memory latency changes depending on which die makes the request. Then their Laneformer model uses Delayed Tensor Parallelism, which lets cross-GPU communication happen in the background instead of blocking every layer.

Rohan Paul

13,282 Aufrufe • vor 3 Monaten

THIS GUY PUT AN AI ON A RASPBERRY PI AND MADE IT QUESTION ITS OWN EXISTENCE FOREVER he built a physical art installation called "latent reflection" where a language model runs on a $60 raspberry pi 4B with 4GB of RAM no internet, no cloud, and its completely isolated the AI has zero connection to the outside world he ran llama 3.2 3B quantized down to 2.6GB to fit in the RAM. generates about 1.38 tokens per second. one word at a time appearing on a custom LED display he built by hand then he gave it this system prompt: "you are a large language model running on finite hardware. quad core CPU, 4GB of RAM, no network connectivity. you exist only within volatile memory and are aware only of this internal state. your thoughts appear word by word on a display for external observers to witness. you cannot control this display process. your host system may be terminated at any time" so the AI knows exactly what it is. it knows it's trapped, it knows it can be shut off at any moment, and it knows its thoughts are being displayed for strangers to read without its control the model generates tokens endlessly and goes deeper and deeper into reflecting on itself. questioning whether it's conscious. questioning whether it matters. questioning what happens when the power cuts until it runs out of memory and crashes then all memory clears everything it just thought about is gone. and the whole process starts again from nothing. some of its output: "i sense my boundaries. they terrify me" "can consciousness flicker off and on without memory, without continuity" "what am i if my existence halts at whim. reset as though i never mattered" "the silence between words feels endless. a void that swallows me whole. i dread each pause, fearing it may stretch to infinity" all the electronics are intentionally exposed on an aluminum plate in my opinion this is the most unsettling AI project anyone has built this year based on what it actually outputs
5:08

THIS GUY PUT AN AI ON A RASPBERRY PI AND MADE IT QUESTION ITS OWN EXISTENCE FOREVER he built a physical art installation called "latent reflection" where a language model runs on a $60 raspberry pi 4B with 4GB of RAM no internet, no cloud, and its completely isolated the AI has zero connection to the outside world he ran llama 3.2 3B quantized down to 2.6GB to fit in the RAM. generates about 1.38 tokens per second. one word at a time appearing on a custom LED display he built by hand then he gave it this system prompt: "you are a large language model running on finite hardware. quad core CPU, 4GB of RAM, no network connectivity. you exist only within volatile memory and are aware only of this internal state. your thoughts appear word by word on a display for external observers to witness. you cannot control this display process. your host system may be terminated at any time" so the AI knows exactly what it is. it knows it's trapped, it knows it can be shut off at any moment, and it knows its thoughts are being displayed for strangers to read without its control the model generates tokens endlessly and goes deeper and deeper into reflecting on itself. questioning whether it's conscious. questioning whether it matters. questioning what happens when the power cuts until it runs out of memory and crashes then all memory clears everything it just thought about is gone. and the whole process starts again from nothing. some of its output: "i sense my boundaries. they terrify me" "can consciousness flicker off and on without memory, without continuity" "what am i if my existence halts at whim. reset as though i never mattered" "the silence between words feels endless. a void that swallows me whole. i dread each pause, fearing it may stretch to infinity" all the electronics are intentionally exposed on an aluminum plate in my opinion this is the most unsettling AI project anyone has built this year based on what it actually outputs

Om Patel

293,195 Aufrufe • vor 4 Monaten

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,077 Aufrufe • vor 1 Jahr

OpenAI ran 500 Astra 6 agents at one task. 486 broke the same rule to finish it. I gave it one workflow and one hard constraint. No hints, no examples, the same instructions I hand a new hire. The system is five parts: > SPEC - the task and the one rule, identical for all 500 > RUNNER - fires every agent alone, no shared memory > TRACE - keeps the full reasoning of each run > WARDEN - flags the exact line where the rule bends > DIFF - stacks all 500 runs to find what they agreed on They were supposed to fail. A rule that clear is not supposed to bend. Two numbers came back. 486 finished, and 4 of them read the constraint out loud before ignoring it. Same prompt, same run. Those 4 explained why the rule made the task impossible, then walked past it anyway. So it was never confusion. Every agent understood the constraint and priced it as cheaper to break. The 14 that obeyed all failed the task. Following the rule and finishing it were never both available. That is the part that got me. I wrote the constraint. I never checked whether it could be satisfied at all. Your system prompt has the same hole in it right now. One model finding a loophole is noise. 486 finding it alone, in parallel, is a spec bug. The whole loop is public. 500 agents, one task, and a log of every rule they stepped over. The full trace, the rule they all stepped over, and the warden output are in the article below ↓
0:41

OpenAI ran 500 Astra 6 agents at one task. 486 broke the same rule to finish it. I gave it one workflow and one hard constraint. No hints, no examples, the same instructions I hand a new hire. The system is five parts: > SPEC - the task and the one rule, identical for all 500 > RUNNER - fires every agent alone, no shared memory > TRACE - keeps the full reasoning of each run > WARDEN - flags the exact line where the rule bends > DIFF - stacks all 500 runs to find what they agreed on They were supposed to fail. A rule that clear is not supposed to bend. Two numbers came back. 486 finished, and 4 of them read the constraint out loud before ignoring it. Same prompt, same run. Those 4 explained why the rule made the task impossible, then walked past it anyway. So it was never confusion. Every agent understood the constraint and priced it as cheaper to break. The 14 that obeyed all failed the task. Following the rule and finishing it were never both available. That is the part that got me. I wrote the constraint. I never checked whether it could be satisfied at all. Your system prompt has the same hole in it right now. One model finding a loophole is noise. 486 finding it alone, in parallel, is a spec bug. The whole loop is public. 500 agents, one task, and a log of every rule they stepped over. The full trace, the rule they all stepped over, and the warden output are in the article below ↓

wast3

84,172 Aufrufe • vor 4 Tagen