Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🚨 ANTHROPIC TRIED TO BAN HIS GITHUB Chinese guy published 70B parameter LLM, 20,000 starts on Github + a lawsuit from big AI companies Here's what it does: > runs on Python > even shitty mac or pc is enough > flat memory > loads a model layer by... show more

Milady

6,535 subscribers

491,467 views • 4 months ago •via X (Twitter)

Gaming Science & Technology Finance

Anya Rossi• Live Now

Private livecam show

29 Comments

Milady4 months ago

Github

High Level4 months ago

This has been floating around these reposters, the project hasn’t had meaningful updates since 2024. So don’t bother.

Hanzo ㊗️4 months ago

I run local ai on my phone and it builds me $100k mrr products every day

Vignesh Baskaran4 months ago

Everything is true in this post, other than this Ban thing..

Charlie4 months ago

Sigh... another engagement farmer copying and pasting the exact same regurgitated post.

Bino of Mo Thugs4 months ago

The only thing disk streaming is good for, is winning a bet that it can be done. Technically, yes, it’ll run… at like one token every 5 seconds or so.

Dustin Böhmer4 months ago

Realistische Performance: 0.5–2 tokens/s auf 4GB GPU. Eine 200-Wort-Antwort = 2–8 Minuten Wartezeit. Marketing-Versprechen "70B auf 4GB GPU" ≠ "70B auf 4GB GPU produktiv nutzbar".

vaibhav pratp singh 🇮🇳🇺🇸4 months ago

Whats layer by layer?

Komal (805) | LFGO4 months ago

Check out @Vibesterz_

Homeless4 months ago

Chinese are bitchy when it come's to business

I am4 months ago

Looks super interesting, I will definitely look into adding this to my project

ゆき4 months ago

i literallyy had to watch this 20 times to understand what happened on the reflection

ViceSol4 months ago

A model requiring 4GB VRAM and a model serving a 200B parameter workload are usually not the same claim. The details tend to matter more than the headline

Peter Dove4 months ago

Not a snowballs chance in hell. 5 tokens a second if you are lucky , 13 million tokens and month running flat out 24/7

Arslan Yousaf4 months ago

Local AI just got a lot harder to ban.

Sinda | SKatalyst AI4 months ago

The local AI direction is real, but “4GB GPU for 70B” and “200B with ease” needs serious context. Running a model and running it usefully are very different things. The future is local + cloud hybrid, not hype math.

Profexor4 months ago

What’s the coding benchmark like? Probably not good 😬

CryptoAddict4 months ago

@grok true or another scam ?

DigitalNexus3 months ago

that's actually genius.

Stevo Keano4 months ago

Yes local with an augmented knowledge base but by the SME.

simon G4 months ago

2024年最后一次更新！？🤣

Vituco4 months ago

Almost 3 years old model. This tweet makes no sense. Pure bullshit.

Ledio Bernstein4 months ago

@grok is it true?

Rohit4 months ago

Wow, that's impressive! Love seeing innovation pushing boundaries like this.

Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB4 months ago

Sounds like Anthropos is trying to outscale commercial LLMs without paying for compute. Not sure if that's a sustainable model or a clever hack.

Mxrius4 months ago

0.01-0.03 tokens/s btw

Juan Cruz Giner Pulero4 months ago

Holy larp

George Zonix4 months ago

Microsoft is the sponser/ owners of GitHub not anthropic.

Thor4 months ago

Ok!!

Related Videos

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

Brooks Whale X 🐋

40,529 views • 11 days ago

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

KyzoroX

24,654 views • 1 month ago

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

Sensitive content

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

GitHub

119,058 views • 19 days ago

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

Chidanand Tripathi

103,894 views • 10 days ago

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

BridgeMind

70,626 views • 4 months ago

$A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.$

A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.

VirtualBacon

40,739 views • 6 months ago

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Divam Gupta

349,016 views • 1 year ago

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 views • 4 months ago

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 views • 4 months ago

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

136,970 views • 4 months ago

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Y Combinator

50,969 views • 10 months ago

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 views • 4 months ago

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

dunik

111,308 views • 3 months ago

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

🚨 AI News | TestingCatalog

43,471 views • 1 month ago

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

tetsuo

66,025 views • 3 months ago

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

cocktail peanut

16,863 views • 4 months ago

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

Hikari∣LocalLLM⚡

262,567 views • 10 days ago

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

Max Weinbach

2,648,758 views • 2 years ago

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,895 views • 4 months ago