Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and... show more

dunik

4,351 subscribers

108,286 Aufrufe • vor 7 Tagen •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

running a 405-billion-parameter model used to mean a $30,000 GPU rack. one developer made it run on a single 8GB card the kind that's been sitting in gaming laptops for years. it's called AirLLM, and the trick is almost stupidly simple: instead of loading the whole model, it keeps one layer on the GPU at a time and streams the rest through. / Llama 3.1 405B on 8GB / DeepSeek-V3, all 671B of it, on about 12GB / no quantization, no distillation, no pruning the full-precision model / works with Llama, Qwen, DeepSeek, Mistral, Gemma, and more it's Apache 2.0 and already past 22,600 stars. → bookmark it.

dunik

148,663 Aufrufe • vor 8 Tagen

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 Aufrufe • vor 1 Monat

A hedge fund manager made $15 billion from a single trade - The greatest trade in Wall Street history in this 24-minute interview he explains how he shorted $25 billion in mortgages, lost money for months, and turned it into the greatest trade ever made Bookmark & Watch It. Then read the post below how the CEO of Citadel started with $265K and turned it into $90 billion ↓

A hedge fund manager made $15 billion from a single trade - The greatest trade in Wall Street history in this 24-minute interview he explains how he shorted $25 billion in mortgages, lost money for months, and turned it into the greatest trade ever made Bookmark & Watch It. Then read the post below how the CEO of Citadel started with $265K and turned it into $90 billion ↓

Сarm1ne

150,282 Aufrufe • vor 2 Monaten

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

MolmoAct2 runs zero-shot on an SO-ARM101, no training required 🦾 Ai2 open Action Reasoning Model is now in LeRobot with the full lifecycle covered: fine-tuning (full or LoRA), evaluation, and real-robot deployment. It pairs a Molmo2-ER vision-language backbone with a flow-matching action expert to turn images, language, and proprioceptive state into action chunks. The best part: ready-made checkpoints ship with calibration correction baked in, so you can point it at your SO-100/101 and just watch it work. Inference fits in ~12GB at bf16, and LoRA fine-tuning runs on a single 24GB GPU. Big thanks to the Ai2 team for building this in the open.

LeRobot

19,891 Aufrufe • vor 13 Tagen

🇺🇸 ZUCK: META AI IS THE BIGGEST, BEST, IN THE WORLD!! Has anyone ever used it?... “Meta AI now has nearly 600 million monthly actives and as promised is on track to be the most used AI assistant in the world by the end of the year. Llama 3.3 is a new 70 billion parameter text model that performs about as well as our 405 billion parameter model but now it is easier and more efficient to run. So that is the last Lama 3.0 release. The next stop is Lama 4.0.” Source: Instagram

🇺🇸 ZUCK: META AI IS THE BIGGEST, BEST, IN THE WORLD!! Has anyone ever used it?... “Meta AI now has nearly 600 million monthly actives and as promised is on track to be the most used AI assistant in the world by the end of the year. Llama 3.3 is a new 70 billion parameter text model that performs about as well as our 405 billion parameter model but now it is easier and more efficient to run. So that is the last Lama 3.0 release. The next stop is Lama 4.0.” Source: Instagram

Mario Nawfal

482,099 Aufrufe • vor 1 Jahr

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,133 Aufrufe • vor 1 Monat

Tomorrow I'm sitting down with Riley Meik (Riley Meik), founder and CEO of The American Housing Corporation. He came out of aerospace, with a stint at SpaceX, before turning his attention to the housing crisis. Most people assume a company like his is trying to reinvent the entire industry. It isn't. "In hard tech, you only want to have to perform one miracle at a time. And if your business relies on multiple miracles, your odds of success are significantly lower." So they picked one. The way the homes are actually built. "We are going to do one thing different, and that's the way we build our homes. When you're inside of it, you shouldn't really know that it was built in a factory. It just looks like a nice house." Everything else, the financing, the land, the banks, he keeps about as conventional as it gets. In tomorrow's clip, Riley breaks down the focus behind doing one thing differently and nothing else.

Tomorrow I'm sitting down with Riley Meik (Riley Meik), founder and CEO of The American Housing Corporation. He came out of aerospace, with a stint at SpaceX, before turning his attention to the housing crisis. Most people assume a company like his is trying to reinvent the entire industry. It isn't. "In hard tech, you only want to have to perform one miracle at a time. And if your business relies on multiple miracles, your odds of success are significantly lower." So they picked one. The way the homes are actually built. "We are going to do one thing different, and that's the way we build our homes. When you're inside of it, you shouldn't really know that it was built in a factory. It just looks like a nice house." Everything else, the financing, the land, the banks, he keeps about as conventional as it gets. In tomorrow's clip, Riley breaks down the focus behind doing one thing differently and nothing else.

Chris Powers

13,632 Aufrufe • vor 23 Tagen

Did Samira mention his name? Or it is a case of “who the cap fits let him wear it”, so he has worn the cap and using it as a basis to attack someone’s wife via his usual wild loose talk. Honestly, he has landed himself in very low waters & he may drown in it by close of Saturday.

Did Samira mention his name? Or it is a case of “who the cap fits let him wear it”, so he has worn the cap and using it as a basis to attack someone’s wife via his usual wild loose talk. Honestly, he has landed himself in very low waters & he may drown in it by close of Saturday.

Samuel Koku Anyidoho🇬🇭

27,568 Aufrufe • vor 2 Jahren

🚨 JUST IN: In a STUNNING blow to the "experts," Scott Bessent confirms President Trump has made America $30-$40 BILLION DOLLARS through his deal with Intel Another win! "I was in the Oval when he told the CEO of Intel that he would like a 10% stake. The CEO turned it over, and the US government's made between $30 and $40 billion on that, and no one wants to give the President credit for that, and he is working for the American people every day!" Rapid Response 47 They said it was impossible 🇺🇸

🚨 JUST IN: In a STUNNING blow to the "experts," Scott Bessent confirms President Trump has made America $30-$40 BILLION DOLLARS through his deal with Intel Another win! "I was in the Oval when he told the CEO of Intel that he would like a 10% stake. The CEO turned it over, and the US government's made between $30 and $40 billion on that, and no one wants to give the President credit for that, and he is working for the American people every day!" Rapid Response 47 They said it was impossible 🇺🇸

Eric Daugherty

196,343 Aufrufe • vor 2 Monaten

A guy got tired of renting intelligence from the cloud, so he put a 120-billion-parameter model in his closet for $0 a month. One mini PC. The Minisforum MS-S1 MAX. 128GB of unified memory, and a trick that hands 96GB of it straight to the GPU. Here's how it breaks: That 96GB is the whole game. Enough to run GPT-OSS-120B fully local, and still keep the agent and dashboard on the same box. 120 billion parameters. 56.6 tokens a second. A 5,000-word essay in one pass at 110W and 68°C. Quiet. Cool. Local. Then Hermes Agent goes on top, pointed at localhost. No API key, because there's nothing to phone home to. It drafts, researches, runs tool calls, manages sub-agents. None of it leaves the network. The cloud version cost $10 to $20 a day, keys and customer data routed through machines he didn't own. Now it's a box in a closet. $0 a month. The chip is real. The build is brilliant. And somewhere in the first few minutes, the AI stopped being a bill he paid every morning and became a machine he owns, humming in a closet, working for free.

A guy got tired of renting intelligence from the cloud, so he put a 120-billion-parameter model in his closet for $0 a month. One mini PC. The Minisforum MS-S1 MAX. 128GB of unified memory, and a trick that hands 96GB of it straight to the GPU. Here's how it breaks: That 96GB is the whole game. Enough to run GPT-OSS-120B fully local, and still keep the agent and dashboard on the same box. 120 billion parameters. 56.6 tokens a second. A 5,000-word essay in one pass at 110W and 68°C. Quiet. Cool. Local. Then Hermes Agent goes on top, pointed at localhost. No API key, because there's nothing to phone home to. It drafts, researches, runs tool calls, manages sub-agents. None of it leaves the network. The cloud version cost $10 to $20 a day, keys and customer data routed through machines he didn't own. Now it's a box in a closet. $0 a month. The chip is real. The build is brilliant. And somewhere in the first few minutes, the AI stopped being a bill he paid every morning and became a machine he owns, humming in a closet, working for free.

shmidt

245,607 Aufrufe • vor 1 Monat

Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs. She also built FlexGen, which made a 175-billion model run on a single consumer GPU, and helped build Chatbot Arena. Three artifacts the whole field uses, one researcher. SGLang hit a 5x cost cut over DeepSeek's own API, and a dozen teams reproduced it. Everyone argues about models. She builds the engines that actually serve them cheaply enough to survive.

Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs. She also built FlexGen, which made a 175-billion model run on a single consumer GPU, and helped build Chatbot Arena. Three artifacts the whole field uses, one researcher. SGLang hit a 5x cost cut over DeepSeek's own API, and a dozen teams reproduced it. Everyone argues about models. She builds the engines that actually serve them cheaply enough to survive.

h100envy

40,959 Aufrufe • vor 1 Monat

He broke the Bank of England, made $1 billion in a single day, ran a $12 billion fund for 30 years, and never had one losing year this is him sitting down with Bloomberg to explain why he just put most of his portfolio into one AI stock and why a guy who shorted entire currencies just put most of his portfolio into one AI stock bookmark & watch it today ↓

He broke the Bank of England, made $1 billion in a single day, ran a $12 billion fund for 30 years, and never had one losing year this is him sitting down with Bloomberg to explain why he just put most of his portfolio into one AI stock and why a guy who shorted entire currencies just put most of his portfolio into one AI stock bookmark & watch it today ↓

slash1s

225,010 Aufrufe • vor 1 Monat

Andrej Karpathy spent 2h showing how he actually uses AI day to day he's a co-founder of OpenAI and led AI at Tesla, so when he shows how he works, it’s worth watching and the whole session is just him telling the machine what he wants in simple terms, like he's briefing a coworker watch what's actually happening the entire time: > he describes the task in normal words > it goes off and does the work > he glances at the result and nudges it with one more sentence that's the whole skill, and you've had it since you learned to talk the only gap between that and a worker that runs on its own is handing that sentence a schedule and the tools to act check his work, then build the version that keeps working when you stop

Andrej Karpathy spent 2h showing how he actually uses AI day to day he's a co-founder of OpenAI and led AI at Tesla, so when he shows how he works, it’s worth watching and the whole session is just him telling the machine what he wants in simple terms, like he's briefing a coworker watch what's actually happening the entire time: > he describes the task in normal words > it goes off and does the work > he glances at the result and nudges it with one more sentence that's the whole skill, and you've had it since you learned to talk the only gap between that and a worker that runs on its own is handing that sentence a schedule and the tools to act check his work, then build the version that keeps working when you stop

Chrome

1,792,070 Aufrufe • vor 1 Monat

I'm running Llama 4 Maverick at 620 t/s! I'm living in the future! Honestly, a large language model running this fast is something straight out of a sci-fi movie. Speeds like this will enable a whole new world of applications that aren't possible today. For reference, GPT-4o, which is probably the most popular OpenAI model, runs between 60 and 110 t/s. The secret here: I'm not running AI at Meta's Llama 4 Maverick on a GPU. I'm using the SambaNova Cloud (my sponsor) and their custom SN40L chips. They are optimized from the ground up for running AI workflows. Right now, SambaNova Cloud runs DeepSeek, Qwen, Whisper, and the entire family of Llama models on these chips. You can check the speed of each of these models using SambaNova Cloud's Playground (see the attached video). It's completely free, and that's how I'm measuring their speeds. For example, I also tried DeepSeek R1 (the latest version from May) and, oh boy! DeepSeek R1 is a huge 671B parameter model. It's probably the best open reasoning model in the world, and it runs at 140 tokens per second! !!! Inference time on an SN40L is night and day from what you'll get from a GPU. Here is why this is big: If you are running an agentic workflow that uses multiple models simultaneously on a GPU, it will need to swap models in and out of memory (because not every model fits). A single SNL40 chip can simultaneously hold over 100 models (trillions of parameters) in memory. If you are using open models, try the SambaCloud API to see what lightning speed looks like. Here is how: 1. Create a free account at: 2. Check the QuickStart guide: If you try the playground, check the speed you're getting with Llama 4 and DeepSeek, and post the results below. I've seen much higher numbers than I posted here, so I'm curious to see whether geography affects the speed.

I'm running Llama 4 Maverick at 620 t/s! I'm living in the future! Honestly, a large language model running this fast is something straight out of a sci-fi movie. Speeds like this will enable a whole new world of applications that aren't possible today. For reference, GPT-4o, which is probably the most popular OpenAI model, runs between 60 and 110 t/s. The secret here: I'm not running AI at Meta's Llama 4 Maverick on a GPU. I'm using the SambaNova Cloud (my sponsor) and their custom SN40L chips. They are optimized from the ground up for running AI workflows. Right now, SambaNova Cloud runs DeepSeek, Qwen, Whisper, and the entire family of Llama models on these chips. You can check the speed of each of these models using SambaNova Cloud's Playground (see the attached video). It's completely free, and that's how I'm measuring their speeds. For example, I also tried DeepSeek R1 (the latest version from May) and, oh boy! DeepSeek R1 is a huge 671B parameter model. It's probably the best open reasoning model in the world, and it runs at 140 tokens per second! !!! Inference time on an SN40L is night and day from what you'll get from a GPU. Here is why this is big: If you are running an agentic workflow that uses multiple models simultaneously on a GPU, it will need to swap models in and out of memory (because not every model fits). A single SNL40 chip can simultaneously hold over 100 models (trillions of parameters) in memory. If you are using open models, try the SambaCloud API to see what lightning speed looks like. Here is how: 1. Create a free account at: 2. Check the QuickStart guide: If you try the playground, check the speed you're getting with Llama 4 and DeepSeek, and post the results below. I've seen much higher numbers than I posted here, so I'm curious to see whether geography affects the speed.

Santiago

34,148 Aufrufe • vor 1 Jahr

Marc Andreessen worked at IBM at the peak of their power. There were 12 layers of management between him as an intern and the CEO. He describes what that meant: if one layer lies to the layer above it, maybe that's okay. Two or three layers, the lies compound. Six layers, they really compound. Twelve layers... The CEO has absolutely no idea what's happening inside his own company. IBM even had a name for it. They called it the Big Gray Cloud, the cloud of men in gray suits that followed the CEO everywhere and made it physically impossible for him to ever talk to someone actually doing the work. "It was like a visit from the king. The king and the traveling court. A completely impervious bubble." That company controlled 80% of tech. Then it didn't. Elon looked at that model and built the opposite. The most dangerous thing in a large organization isn't incompetence. It's the distance between the truth and the top.

Marc Andreessen worked at IBM at the peak of their power. There were 12 layers of management between him as an intern and the CEO. He describes what that meant: if one layer lies to the layer above it, maybe that's okay. Two or three layers, the lies compound. Six layers, they really compound. Twelve layers... The CEO has absolutely no idea what's happening inside his own company. IBM even had a name for it. They called it the Big Gray Cloud, the cloud of men in gray suits that followed the CEO everywhere and made it physically impossible for him to ever talk to someone actually doing the work. "It was like a visit from the king. The king and the traveling court. A completely impervious bubble." That company controlled 80% of tech. Then it didn't. Elon looked at that model and built the opposite. The most dangerous thing in a large organization isn't incompetence. It's the distance between the truth and the top.

Founder Mode

220,034 Aufrufe • vor 3 Monaten

Jane Street brought in Horace He to explain how to squeeze the most out of a single GPU. And then how to scale it to tens of thousands. Graduated from Cornell in 2020. Works on the PyTorch team. Training Llama 3 takes 40 trillion trillion floating point operations. He wrote the two things those models run on: torch.compile and FlexAttention.

Jane Street brought in Horace He to explain how to squeeze the most out of a single GPU. And then how to scale it to tens of thousands. Graduated from Cornell in 2020. Works on the PyTorch team. Training Llama 3 takes 40 trillion trillion floating point operations. He wrote the two things those models run on: torch.compile and FlexAttention.

zostaff

51,601 Aufrufe • vor 1 Monat

AMD on local AI: "this is the smallest AI development system in the world, capable of running models up to 200 billion parameters locally, not connected to anything" 128GB of unified memory shared by CPU, GPU, and NPU fits in your hand. Runs a model larger than GPT-3 with no internet $3,999 once. roughly $16 a month in power Cloud equivalent: up to $750 a month The question was never whether local AI could compete on power It was whether it could compete on price. Now it does

AMD on local AI: "this is the smallest AI development system in the world, capable of running models up to 200 billion parameters locally, not connected to anything" 128GB of unified memory shared by CPU, GPU, and NPU fits in your hand. Runs a model larger than GPT-3 with no internet $3,999 once. roughly $16 a month in power Cloud equivalent: up to $750 a month The question was never whether local AI could compete on power It was whether it could compete on price. Now it does

plutos

151,966 Aufrufe • vor 1 Monat

"Elon is very mysterious in a way that somehow nobody seems to notice. One of his slogans is, Ad Astra, to the stars, but you can't get to the stars using SpaceX because it's a chemical rocket company. So what the heck? He's not investing in physics, he was going to go into a physics grad school and he has a physics BA, so there's something very very strange about Elon, which is that he's focused on getting to the stars, he doesn't seem to have any interest in physicists, and he has a chemical rocket company. What's going on? His real space company, I think, is not SpaceX, it's xAI, it's Grok. Look at the logo for Grok, I think it's like a black hole profile. Look at its slogan, to understand the universe. I think he's given up on the corruption of science. He keeps talking about the fact that he doesn't like the term researcher, he believes only in engineers. He's constantly really running down scientists. I don't think he doesn't believe in scientists, I think he thinks that science has gotten corrupted and he doesn't want to deal with it. I think what he's doing is he's building Grok to do science in a time when humans are too corrupt to be truthful and real." 😮👀

Apple Lamps

552,905 Aufrufe • vor 4 Monaten

THIS AI DEVELOPER RUNS 120B PARAMETER MODELS LOCALLY AND HIS MACBOOK PRO CAN’T EVEN OPEN THEM he bought a DGX Spark, gave it the model, went back to his desk by end of day the inference was done, tested, committed his team is still waiting for the cloud GPU queue to free up the only thing between them is 128GB of unified memory in a box smaller than your lunch one Spark is a datacenter node, two Sparks is your own cluster bookmark this and send it to whoever is still paying for cloud GPUs

THIS AI DEVELOPER RUNS 120B PARAMETER MODELS LOCALLY AND HIS MACBOOK PRO CAN’T EVEN OPEN THEM he bought a DGX Spark, gave it the model, went back to his desk by end of day the inference was done, tested, committed his team is still waiting for the cloud GPU queue to free up the only thing between them is 128GB of unified memory in a box smaller than your lunch one Spark is a datacenter node, two Sparks is your own cluster bookmark this and send it to whoever is still paying for cloud GPUs

leopardracer

85,470 Aufrufe • vor 1 Monat