Loading video...

Video Failed to Load

Go Home

397 BILLION PARAMETER MODEL RUNS LOCALLY ON A $32,000 DESKTOP CLUSTER An 8-node cluster of Nvidia GB10 mini-supercomputers pooled 1TB of memory to run Qwen 3.5 offline. Thanks to Mixture-of-Experts sparsity (activating only 17B parameters per token) the $32,000 setup delivers private 24 token/sec inference without data leaving the room.

RoundtableSpace's profile picture

0xMarioNawfal

258,104 subscribers

54,183 views • 10 days ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables
21:59

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,354 views • 1 month ago

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less
24:17

THIS DEVELOPER CONNECTED 8 NVIDIA DGX SPARKS INTO ONE CLUSTER - AND RAN AN 800GB MODEL THAT MADE HIM 10X MORE PRODUCTIVE 21:47 he says it straight - "this is a terabyte of VRAM - we ran Quen 3.5, 800GB on disk, a model that doesn't even fit on a single Mac Studio - 24 tokens per second - I'd say that's a win" 8 Sparks connected through a $1,300 switch via RDMA over Ethernet - each node adding 128GB of memory into one unified pool of 1TB started with one Spark at 3 tokens per second - every added node doubled the speed - and eight together deliver 24 tokens on a model that physically cannot run anywhere else Kimi K2 at 600GB loaded in 15 minutes, 115GB per node, 13 tokens per second - a model that simply cannot run on anything smaller Claude helped configure the entire cluster - SSH mesh across all 8 machines, network config, jumbo frames, QSFP port speeds - all from one terminal most people rent cloud compute for models this size at $2,000+/month - he built the cluster once and now every token costs 20x less

Noisy

185,148 views • 1 month ago

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket
1:05

FOUR DIFFERENT VENDORS ARE NOW SHIPPING GB10 MINI PCs WITH 128GB UNIFIED MEMORY, AND ONE MICROTIK CRS 804 SWITCH CAN CONNECT UP TO EIGHT OF THEM INTO A 1 TERABYTE LOCAL AI CLUSTER 00:00 he points at the MikroTik CRS 804, "you need some kind of switch that'll handle QSFP56 ports like these", the interconnect that makes the whole cluster possible the GB10 ecosystem is no longer just Nvidia. Dell Pro Max GB10, ASUS Ascent GX10, and MSI Edge Expert all ship the same Grace Blackwell Superchip with 128GB of coherent memory. same silicon, different cases, same 200 gigabit ports on the back the CRS 804 is what connects them at prosumer prices. four 400 gigabit QSFP56 ports on one 1U chassis, breakout cables that split each port into two 200 gigabit lanes. one switch drives eight GB10 units in parallel do the math. eight nodes at 128GB each equals 1024GB of pooled unified memory across the cluster. run vLLM, shard a frontier model across all eight, and inference happens locally on hardware that fits in half a rack the real limiter revealed in the stress test was never throttling. it was interconnect topology, exactly the layer this switch fixes at a fraction of enterprise switch pricing $400 a month for combined chatgpt pro and claude code max hits $4,800 a year per developer. a small team of five running through this cluster pays back inside eight months and never expires the article covers the buying ladder for a single desk. this post is proof of the cluster ladder that starts where the desk one ends save this before the GB10 lineup grows past four vendors and prosumer cluster switches move upmarket

NO1ennn

59,725 views • 29 days ago

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6
0:30

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀 At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public. What K3 brings: 🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token) 📚 1M-token context window 🛠️ Native multimodal understanding, including vision ⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable Huge thank you to Kimi.ai AI for the model release and partnership, Inferact for leading the vLLM optimizations, and to our partners at NVIDIA, AMD, and the broader vLLM community. 1/6

vLLM

101,002 views • 8 days ago

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure
1:42

CHINESE AI TEACHER STACKED 4 NVIDIA DGX SPARK CHIPS INTO A CLUSTER - AND NOW RUNS A BUSINESS ON IT FROM HIS DESK 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else he's running a Qwen 32 35B model with 4 concurrent requests at the same time and the speed barely drops compared to a single request what used to require a $10,000+/month cloud setup now sits on his desk and costs him $40-60/month in electricity 4 chips at $2,999 each - one time investment - and now he has a cluster that handles client inference requests around the clock while he does something else the people still paying per token are financing someone else's infrastructure

Sprytix

51,010 views • 2 months ago

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look
20:03

THIS CLUSTER OF 4 MAC STUDIOS USES LESS POWER THAN ONE NVIDIA CHIP - AND COSTS 5X LESS 17:30 the number hits hard - 600 watts for all four Mac Studios - less than one H200 that costs $30,000 on its own instead of one $30,000 GPU you get 4 machines with 1.5TB of memory, 3x more performance and models that simply cannot run on a single chip Apple quietly added RDMA over Thunderbolt in a beta update - latency dropped 99% and the same cluster became 3x faster without buying any new hardware before the update: 5 tokens per second. after: 15.5 - a free upgrade through a software update 8 Nvidia DGX Sparks for the same memory would cost $32,000 - this cluster costs a fraction and gets faster with every update 10x more productive at half the price - and the only thing you need is knowing where to look

Noisy

105,595 views • 1 month ago

THIS AI DEVELOPER RUNS 120B PARAMETER MODELS LOCALLY AND HIS MACBOOK PRO CAN’T EVEN OPEN THEM he bought a DGX Spark, gave it the model, went back to his desk by end of day the inference was done, tested, committed his team is still waiting for the cloud GPU queue to free up the only thing between them is 128GB of unified memory in a box smaller than your lunch one Spark is a datacenter node, two Sparks is your own cluster bookmark this and send it to whoever is still paying for cloud GPUs
2:54

THIS AI DEVELOPER RUNS 120B PARAMETER MODELS LOCALLY AND HIS MACBOOK PRO CAN’T EVEN OPEN THEM he bought a DGX Spark, gave it the model, went back to his desk by end of day the inference was done, tested, committed his team is still waiting for the cloud GPU queue to free up the only thing between them is 128GB of unified memory in a box smaller than your lunch one Spark is a datacenter node, two Sparks is your own cluster bookmark this and send it to whoever is still paying for cloud GPUs

leopardracer

85,726 views • 2 months ago

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓
1:21

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,240 views • 1 month ago

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.
2:00

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Aakash Gupta

508,149 views • 3 months ago

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!
0:26

llama3 8B (not quantized) running on an heterogeneous home cluster made of: - iPhone 15 Pro Max - iPad Pro (not sure which version XD) - MacBook Pro ( M1 Max ) - NVIDIA GeForce 3080 (not visible in video) - 2x NVIDIA Titan X Pascal Very soon also supporting Android (I *have* to also add my NVIDIA Shield GPU!!!!!). Single code base, single model format (reduced and optimally distributed to every node to save space). Everything (including iOS code) is open here ... it would be really nice, with the help of the community, taking this project to the next level in terms of optimization and support. My vision is about a distributed inference server that can run any model on any backend in any cluster topology - let's fight programmed obsolescence and democratize inference!

Simone Margaritelli

304,072 views • 2 years ago

‼️PROMINENT CRYPTO FIGURE SAYS #XRP IS A SCAM RUN BY BRAD GARLINGHOUSE ! HE FURTHER EXPLAINS THAT THEY ARE CONVINCING THE USA GOVERNMENT TO BUILD AN XRP RESERVE INSTEAD OF A BITCOIN ONE‼️ THE #XRPL IS EXPECTED TO HAVE MASSIVE VOLUMES AS USA BEGINS PURCHASING AND BUILDING ON IT!! TOP DEFI TOKEN ON XRP LEDGER, CTF TOKEN, HAS ALREADY ONBOARDED AMAZON AND WALMART!! THINK ABOUT THE VOLUME!! CTF TOKEN WOULD JUMP FROM $0.81 to $748.50 PER TOKEN WITH A MARKET CAP OF ONLY $20 BILLION! ITS GOING TO GO INSANE!! CTF TOKEN TRADE LINK: CTF TOKEN TRADE LINK ON MEXC: Website:
1:09

‼️PROMINENT CRYPTO FIGURE SAYS #XRP IS A SCAM RUN BY BRAD GARLINGHOUSE ! HE FURTHER EXPLAINS THAT THEY ARE CONVINCING THE USA GOVERNMENT TO BUILD AN XRP RESERVE INSTEAD OF A BITCOIN ONE‼️ THE #XRPL IS EXPECTED TO HAVE MASSIVE VOLUMES AS USA BEGINS PURCHASING AND BUILDING ON IT!! TOP DEFI TOKEN ON XRP LEDGER, CTF TOKEN, HAS ALREADY ONBOARDED AMAZON AND WALMART!! THINK ABOUT THE VOLUME!! CTF TOKEN WOULD JUMP FROM $0.81 to $748.50 PER TOKEN WITH A MARKET CAP OF ONLY $20 BILLION! ITS GOING TO GO INSANE!! CTF TOKEN TRADE LINK: CTF TOKEN TRADE LINK ON MEXC: Website:

FinanceBro

64,463 views • 1 year ago

NVIDIA JUST SHOWED HOW A $4,699 BOX CAN KILL A $200 MONTHLY AI BILL AND RUN 200B MODELS FROM YOUR DESK. 00:02 NVIDIA delivers the DGX Spark to developers, researchers, and AI teams, turning what used to require rented datacenter GPUs into a box small enough to hold in one hand. inside is 128GB of unified memory, 4TB of storage, and up to 1 PFLOP of AI compute, enough to run 70B models comfortably and push into the 200B class locally. that means no per-token API bill, no files leaving your machine, and no waiting for someone else’s server every time an agent reads documents, writes code, or runs an automation. two DGX Sparks can even be linked together for models around 400B parameters, while a single unit costs roughly $4,000 to $4,700 instead of renting multi-GPU cloud infrastructure every month. the full local AI ladder starts at $0 with the computer you already own, but NVIDIA just showed what the final rung looks like: a personal AI supercomputer sitting directly on your desk.
0:24

NVIDIA JUST SHOWED HOW A $4,699 BOX CAN KILL A $200 MONTHLY AI BILL AND RUN 200B MODELS FROM YOUR DESK. 00:02 NVIDIA delivers the DGX Spark to developers, researchers, and AI teams, turning what used to require rented datacenter GPUs into a box small enough to hold in one hand. inside is 128GB of unified memory, 4TB of storage, and up to 1 PFLOP of AI compute, enough to run 70B models comfortably and push into the 200B class locally. that means no per-token API bill, no files leaving your machine, and no waiting for someone else’s server every time an agent reads documents, writes code, or runs an automation. two DGX Sparks can even be linked together for models around 400B parameters, while a single unit costs roughly $4,000 to $4,700 instead of renting multi-GPU cloud infrastructure every month. the full local AI ladder starts at $0 with the computer you already own, but NVIDIA just showed what the final rung looks like: a personal AI supercomputer sitting directly on your desk.

Gipp 🦅

11,334 views • 16 days ago

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!
0:39

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,678 views • 1 month ago

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours
22:35

THIS DEVELOPER JUST RAN A TRILLION PARAMETER MODEL ON 4 MAC STUDIOS - 10X FASTER AND 5X CHEAPER THAN CLOUD CODE 19:00 he says it out loud. "we just ran a trillion parameter model. 30 something tokens per second. wow." RDMA over Thunderbolt made the cluster 10x faster than before. tensor parallelism splits the model between machines in parallel instead of sequentially DeepSeek V3 full size without quantization. 183GB per machine. still 25 tokens per second. faster than most people can read Kimi K2 with trillion parameters and 256,000 context window. loaded, responded and adapted memory usage dynamically based on prompt size 4 machines consuming 66 watts total. cloud GPU for the same workload costs $1,900/month a cluster that used to cost millions now assembled from Mac Studios in a few hours

Noisy

79,665 views • 1 month ago

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.
2:00

Andrej Karpathy just made one of the most interesting arguments about AI model design that most people are completely missing. His take is that frontier AI models are not too big because the technology is complex and too big because the training data is garbage. When you or I think of the internet, we picture Wall Street Journal articles, Wikipedia entries, serious writing. That is not what a pretraining dataset looks like. When researchers at frontier labs look at random documents from the actual training corpus, it is stock ticker symbols, broken HTML, spam, gibberish. One estimate puts Llama 3's information compression at just 0.07 bits per token meaning the model has only a hazy recollection of most of what it trained on. So we build trillion parameter models not because we need a trillion parameter brain but because we need a trillion-parameter compression engine to squeeze some intelligence out of a firehose of noise. Most of those parameters are doing memory work, not cognitive work. Karpathy's prediction is separate the two entirely. Build a cognitive core, a model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization and pair it with external memory that it can query when it needs facts. He thinks a cognitive core trained on high-quality data could hit genuine intelligence at around one billion parameters. For reference, today's flagship models run between 200 billion and 1.8 trillion parameters with most of that weight dedicated to remembering the internet's slop. The trend is already moving his direction. GPT-4o operates at roughly 200 billion parameters and outperforms the original 1.8 trillion-parameter GPT-4. Inference costs for GPT-3.5-level performance dropped 280-fold between 2022 and 2024 driven almost entirely by smaller, cleaner, better-architected models. The real bottleneck in AI right now is not compute but rather data quality.

Milk Road AI

200,409 views • 3 months ago

🚨 Open-source AI just got serious. Alibaba just released Qwen 3.5-Plus 🤯 397B parameters. Only 17B active per inference. Performance comparable to Gemini 3 Pro — at just $0.8 per million tokens (1/18 the cost). But here’s what makes it different 👇 This isn’t just another benchmark drop. In the demo, Qwen runs directly inside the terminal: ➙ Reviews GitHub PRs ➙ Fetches diffs automatically ➙ Runs CLI & smoke tests ➙ Generates integration tests ➙ Executes /context ➙ Shows real-time token usage breakdown That’s not a chatbot. That’s a real developer workflow. Under the hood: ⚡ 60% lower deployment memory ⚡ Inference throughput increased by 8.6× (32K context) ⚡ Inference throughput increased by up to 19× (256K ultra-long context) ⚡ Native multimodal model ⚡ Supports direct video input (up to 2 hours) And the ecosystem is massive: ✔ 201 languages ✔ 250K vocabulary ✔ 400+ open-sourced models ✔ 1B+ downloads ✔ 200K+ derivative models Available now via Qwen App, MoDa Community, Hugging Face, and Alibaba Cloud Bailian. ✅ The AI race isn’t about who’s biggest anymore. → It’s about who’s usable. 🔥 Watch the demo 👇 #Qwen #OpenSourceAI #AI #DevTools
1:10

🚨 Open-source AI just got serious. Alibaba just released Qwen 3.5-Plus 🤯 397B parameters. Only 17B active per inference. Performance comparable to Gemini 3 Pro — at just $0.8 per million tokens (1/18 the cost). But here’s what makes it different 👇 This isn’t just another benchmark drop. In the demo, Qwen runs directly inside the terminal: ➙ Reviews GitHub PRs ➙ Fetches diffs automatically ➙ Runs CLI & smoke tests ➙ Generates integration tests ➙ Executes /context ➙ Shows real-time token usage breakdown That’s not a chatbot. That’s a real developer workflow. Under the hood: ⚡ 60% lower deployment memory ⚡ Inference throughput increased by 8.6× (32K context) ⚡ Inference throughput increased by up to 19× (256K ultra-long context) ⚡ Native multimodal model ⚡ Supports direct video input (up to 2 hours) And the ecosystem is massive: ✔ 201 languages ✔ 250K vocabulary ✔ 400+ open-sourced models ✔ 1B+ downloads ✔ 200K+ derivative models Available now via Qwen App, MoDa Community, Hugging Face, and Alibaba Cloud Bailian. ✅ The AI race isn’t about who’s biggest anymore. → It’s about who’s usable. 🔥 Watch the demo 👇 #Qwen #OpenSourceAI #AI #DevTools

Toha Khan

123,375 views • 5 months ago

NVIDIA JUST FEATURED HERMES AGENT AT COMPUTEX 2026. Hermes is integrating with NVIDIA OpenShell and Microsoft security primitives for secure on-device agents on Windows. Dillon Rolnick, CEO of Nous Research: "RTX Spark and NVIDIA OpenShell give Hermes users a powerful and secure environment for agents to run and work alongside you. You realize you're buying a full-fledged assistant, not a typical laptop." RTX Spark specs: → 1 petaflop AI performance → 20-core Grace CPU (built with MediaTek) → Blackwell GPU, 6,144 CUDA cores → 5th gen Tensor Cores with FP4 precision → up to 128GB LPDDR5X unified memory, 300 GB/s → NVLink C2C chip-to-chip interconnect → runs 120B parameter LLMs locally → up to 1M token context window on device → 2x inference speed via multi-token prediction RTX Spark devices ship Fall 2026 from ASUS, Dell, HP, Lenovo, Microsoft Surface, MSI. full NVIDIA press release in the comments. full Hermes setup guide in the article 👇
2:28

NVIDIA JUST FEATURED HERMES AGENT AT COMPUTEX 2026. Hermes is integrating with NVIDIA OpenShell and Microsoft security primitives for secure on-device agents on Windows. Dillon Rolnick, CEO of Nous Research: "RTX Spark and NVIDIA OpenShell give Hermes users a powerful and secure environment for agents to run and work alongside you. You realize you're buying a full-fledged assistant, not a typical laptop." RTX Spark specs: → 1 petaflop AI performance → 20-core Grace CPU (built with MediaTek) → Blackwell GPU, 6,144 CUDA cores → 5th gen Tensor Cores with FP4 precision → up to 128GB LPDDR5X unified memory, 300 GB/s → NVLink C2C chip-to-chip interconnect → runs 120B parameter LLMs locally → up to 1M token context window on device → 2x inference speed via multi-token prediction RTX Spark devices ship Fall 2026 from ASUS, Dell, HP, Lenovo, Microsoft Surface, MSI. full NVIDIA press release in the comments. full Hermes setup guide in the article 👇

YanXbt

45,924 views • 2 months ago

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.
54:24

8 DGX Sparks on a desk just ran a frontier model at 400K context. A friend who owns the inference budget at a Berlin AI startup, $52K a month in API bills, sent me this at 2 a.m. with one line: "we're buying the hardware." It starts as a disaster. A mountain of cables burying $32K of NVIDIA silicon, nothing talking to anything. 54 minutes later the whole stack answers as 1 machine. 1TB of unified memory. GLM-5.2 loaded across all 8 nodes. The part that made him rewatch twice sits around the 25-minute mark, where 1 networking fix turns 8 boxes into a cluster. Most people quit exactly there. He paused it 4 times. Screenshotted the interface setup. Forwarded it to his CTO without a caption. The cloud bill was never the moat. The cables were.

the lich

56,893 views • 18 days ago