Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

🚨 ANTHROPIC TRIED TO BAN HIS GITHUB Chinese guy published 70B parameter LLM, 20,000 starts on Github + a lawsuit from big AI companies Here's what it does: > runs on Python > even shitty mac or pc is enough > flat memory > loads a model layer by... show more

Milady

6,535 subscribers

491,467 просмотров • 4 месяцев назад •via X (Twitter)

Игры Наука и технологии Финансы

Anya Rossi• Live Now

Private livecam show

Комментарии: 29

Фото профиля Milady

Milady4 месяцев назад

Github

Фото профиля High Level

High Level4 месяцев назад

This has been floating around these reposters, the project hasn’t had meaningful updates since 2024. So don’t bother.

Фото профиля Hanzo ㊗️

Hanzo ㊗️4 месяцев назад

I run local ai on my phone and it builds me $100k mrr products every day

Фото профиля Vignesh Baskaran

Vignesh Baskaran4 месяцев назад

Everything is true in this post, other than this Ban thing..

Фото профиля Charlie

Charlie4 месяцев назад

Sigh... another engagement farmer copying and pasting the exact same regurgitated post.

Фото профиля Bino of Mo Thugs

Bino of Mo Thugs4 месяцев назад

The only thing disk streaming is good for, is winning a bet that it can be done. Technically, yes, it’ll run… at like one token every 5 seconds or so.

Фото профиля Dustin Böhmer

Dustin Böhmer4 месяцев назад

Realistische Performance: 0.5–2 tokens/s auf 4GB GPU. Eine 200-Wort-Antwort = 2–8 Minuten Wartezeit. Marketing-Versprechen "70B auf 4GB GPU" ≠ "70B auf 4GB GPU produktiv nutzbar".

Фото профиля vaibhav pratp singh 🇮🇳🇺🇸

vaibhav pratp singh 🇮🇳🇺🇸4 месяцев назад

Whats layer by layer?

Фото профиля Komal (805) | LFGO

Komal (805) | LFGO4 месяцев назад

Check out @Vibesterz_

Фото профиля Homeless

Homeless4 месяцев назад

Chinese are bitchy when it come's to business

Фото профиля I am

I am4 месяцев назад

Looks super interesting, I will definitely look into adding this to my project

Фото профиля ゆき

ゆき4 месяцев назад

i literallyy had to watch this 20 times to understand what happened on the reflection

Фото профиля ViceSol

ViceSol4 месяцев назад

A model requiring 4GB VRAM and a model serving a 200B parameter workload are usually not the same claim. The details tend to matter more than the headline

Фото профиля Peter Dove

Peter Dove4 месяцев назад

Not a snowballs chance in hell. 5 tokens a second if you are lucky , 13 million tokens and month running flat out 24/7

Фото профиля Arslan Yousaf

Arslan Yousaf4 месяцев назад

Local AI just got a lot harder to ban.

Фото профиля Sinda | SKatalyst AI

Sinda | SKatalyst AI4 месяцев назад

The local AI direction is real, but “4GB GPU for 70B” and “200B with ease” needs serious context. Running a model and running it usefully are very different things. The future is local + cloud hybrid, not hype math.

Фото профиля Profexor

Profexor4 месяцев назад

What’s the coding benchmark like? Probably not good 😬

Фото профиля CryptoAddict

CryptoAddict4 месяцев назад

@grok true or another scam ?

Фото профиля DigitalNexus

DigitalNexus3 месяцев назад

that's actually genius.

Фото профиля Stevo Keano

Stevo Keano4 месяцев назад

Yes local with an augmented knowledge base but by the SME.

Фото профиля simon G

simon G4 месяцев назад

2024年最后一次更新！？🤣

Фото профиля Vituco

Vituco4 месяцев назад

Almost 3 years old model. This tweet makes no sense. Pure bullshit.

Фото профиля Ledio Bernstein

Ledio Bernstein4 месяцев назад

@grok is it true?

Фото профиля Rohit

Rohit4 месяцев назад

Wow, that's impressive! Love seeing innovation pushing boundaries like this.

Фото профиля Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB

Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB4 месяцев назад

Sounds like Anthropos is trying to outscale commercial LLMs without paying for compute. Not sure if that's a sustainable model or a clever hack.

Фото профиля Mxrius

Mxrius4 месяцев назад

0.01-0.03 tokens/s btw

Фото профиля Juan Cruz Giner Pulero

Juan Cruz Giner Pulero4 месяцев назад

Holy larp

Фото профиля George Zonix

George Zonix4 месяцев назад

Microsoft is the sponser/ owners of GitHub not anthropic.

Фото профиля Thor

Thor4 месяцев назад

Ok!!

Похожие видео

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

Brooks Whale X 🐋

40,529 просмотров • 11 дней назад

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

KyzoroX

24,654 просмотров • 1 месяц назад

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

Sensitive content

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

GitHub

119,058 просмотров • 19 дней назад

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

Chidanand Tripathi

103,894 просмотров • 10 дней назад

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

BridgeMind

70,626 просмотров • 4 месяцев назад

$A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.$

A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.

VirtualBacon

40,739 просмотров • 6 месяцев назад

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Divam Gupta

349,016 просмотров • 1 год назад

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 просмотров • 4 месяцев назад

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 просмотров • 4 месяцев назад

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

136,970 просмотров • 4 месяцев назад

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Y Combinator

50,969 просмотров • 10 месяцев назад

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 просмотров • 4 месяцев назад

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

dunik

111,308 просмотров • 3 месяцев назад

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

🚨 AI News | TestingCatalog

43,471 просмотров • 1 месяц назад

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

tetsuo

66,025 просмотров • 3 месяцев назад

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

cocktail peanut

16,863 просмотров • 4 месяцев назад

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

Hikari∣LocalLLM⚡

262,567 просмотров • 10 дней назад

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

Max Weinbach

2,648,758 просмотров • 2 лет назад

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,895 просмотров • 4 месяцев назад