Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

🚨 ANTHROPIC TRIED TO BAN HIS GITHUB Chinese guy published 70B parameter LLM, 20,000 starts on Github + a lawsuit from big AI companies Here's what it does: > runs on Python > even shitty mac or pc is enough > flat memory > loads a model layer by... show more

Milady

6,535 subscribers

491,467 Aufrufe • vor 4 Monaten •via X (Twitter)

Gaming Wissenschaft & Technologie Finanzen

Anya Rossi• Live Now

Private livecam show

29 Kommentare

Profilbild von Milady

Miladyvor 4 Monaten

Github

Profilbild von High Level

High Levelvor 4 Monaten

This has been floating around these reposters, the project hasn’t had meaningful updates since 2024. So don’t bother.

Profilbild von Hanzo ㊗️

Hanzo ㊗️vor 4 Monaten

I run local ai on my phone and it builds me $100k mrr products every day

Profilbild von Vignesh Baskaran

Vignesh Baskaranvor 4 Monaten

Everything is true in this post, other than this Ban thing..

Profilbild von Charlie

Charlievor 4 Monaten

Sigh... another engagement farmer copying and pasting the exact same regurgitated post.

Profilbild von Bino of Mo Thugs

Bino of Mo Thugsvor 4 Monaten

The only thing disk streaming is good for, is winning a bet that it can be done. Technically, yes, it’ll run… at like one token every 5 seconds or so.

Profilbild von Dustin Böhmer

Dustin Böhmervor 4 Monaten

Realistische Performance: 0.5–2 tokens/s auf 4GB GPU. Eine 200-Wort-Antwort = 2–8 Minuten Wartezeit. Marketing-Versprechen "70B auf 4GB GPU" ≠ "70B auf 4GB GPU produktiv nutzbar".

Profilbild von vaibhav pratp singh 🇮🇳🇺🇸

vaibhav pratp singh 🇮🇳🇺🇸vor 4 Monaten

Whats layer by layer?

Profilbild von Komal (805) | LFGO

Komal (805) | LFGOvor 4 Monaten

Check out @Vibesterz_

Profilbild von Homeless

Homelessvor 4 Monaten

Chinese are bitchy when it come's to business

Profilbild von I am

I amvor 4 Monaten

Looks super interesting, I will definitely look into adding this to my project

Profilbild von ゆき

ゆきvor 4 Monaten

i literallyy had to watch this 20 times to understand what happened on the reflection

Profilbild von ViceSol

ViceSolvor 4 Monaten

A model requiring 4GB VRAM and a model serving a 200B parameter workload are usually not the same claim. The details tend to matter more than the headline

Profilbild von Peter Dove

Peter Dovevor 4 Monaten

Not a snowballs chance in hell. 5 tokens a second if you are lucky , 13 million tokens and month running flat out 24/7

Profilbild von Arslan Yousaf

Arslan Yousafvor 4 Monaten

Local AI just got a lot harder to ban.

Profilbild von Sinda | SKatalyst AI

Sinda | SKatalyst AIvor 4 Monaten

The local AI direction is real, but “4GB GPU for 70B” and “200B with ease” needs serious context. Running a model and running it usefully are very different things. The future is local + cloud hybrid, not hype math.

Profilbild von Profexor

Profexorvor 4 Monaten

What’s the coding benchmark like? Probably not good 😬

Profilbild von CryptoAddict

CryptoAddictvor 4 Monaten

@grok true or another scam ?

Profilbild von DigitalNexus

DigitalNexusvor 3 Monaten

that's actually genius.

Profilbild von Stevo Keano

Stevo Keanovor 4 Monaten

Yes local with an augmented knowledge base but by the SME.

Profilbild von simon G

simon Gvor 4 Monaten

2024年最后一次更新！？🤣

Profilbild von Vituco

Vitucovor 4 Monaten

Almost 3 years old model. This tweet makes no sense. Pure bullshit.

Profilbild von Ledio Bernstein

Ledio Bernsteinvor 4 Monaten

@grok is it true?

Profilbild von Rohit

Rohitvor 4 Monaten

Wow, that's impressive! Love seeing innovation pushing boundaries like this.

Profilbild von Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMB

Prithvi Jadwani | AI SEO | GEO | REDDIT SEO | GMBvor 4 Monaten

Sounds like Anthropos is trying to outscale commercial LLMs without paying for compute. Not sure if that's a sustainable model or a clever hack.

Profilbild von Mxrius

Mxriusvor 4 Monaten

0.01-0.03 tokens/s btw

Profilbild von Juan Cruz Giner Pulero

Juan Cruz Giner Pulerovor 4 Monaten

Holy larp

Profilbild von George Zonix

George Zonixvor 4 Monaten

Microsoft is the sponser/ owners of GitHub not anthropic.

Profilbild von Thor

Thorvor 4 Monaten

Ok!!

Ähnliche Videos

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

🚨 2.8 trillion parameters. Under 4GB VRAM. No quantization. An open source tool called AirLLM has changed how large language models run on consumer hardware. Instead of loading the entire model at once, it processes one layer at a time. This means your VRAM needs depend only on the size of a single layer rather than the full model. No quantization. No distillation. No pruning. → DeepSeek V3 (671B) runs on just 12GB → Llama 3.1 405B runs on 8GB → Kimi K3 (2.8 trillion parameters) runs on under 4GB → Works with nearly every major open source model The largest model often needs the least VRAM. Kimi K3 is a sparse mixture of experts model, so it only loads the specific experts activated by each token instead of an entire dense layer. This is why a 2.8 trillion parameter model can run in less memory than a 70B model. AirLLM makes frontier scale AI practical on everyday hardware without any of the usual tradeoffs. (Link in the comments)

Brooks Whale X 🐋

40,529 Aufrufe • vor 11 Tagen

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

$3,800 for a Mini PC with 96GB of GPU memory sounds ridiculous. Until you see what AMD is actually doing. The GMKtec EVO-X3 has: Ryzen AI MAX+ 395 128GB LPDDR5X Up to 96GB allocated to the Radeon 8060S 8000 MT/s memory No discrete GPU. That means a tiny box can run local models that would normally push you toward a workstation with a huge GPU. And it can still play games. The interesting part isn't the 126 TOPS number. It's the fact that 96GB of GPU memory now fits inside a mini PC. The question is no longer whether this can run a big model. It's how fast it can run it. The EVO-X3 is available here:

KyzoroX

24,654 Aufrufe • vor 1 Monat

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

Sensitive content

📣 Coming soon to GitHub Copilot: intelligent local model routing. We are excited to announce the next step in our Project HydraFusion vision. GitHub Copilot will soon be able to automatically route tasks to a local model when it's most suitable, helping you save on AI credits.

GitHub

119,058 Aufrufe • vor 19 Tagen

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

The wall of text is the most annoying part of using AI apps. With OpenUI 1.0, the model picks what each answer needs. A chart for numbers, a map for a trip, a form when it needs your input. You choose which components it can use, so it still looks like your product. Open source, any model, even ones you run on your own hardware.

Chidanand Tripathi

103,894 Aufrufe • vor 10 Tagen

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

I tested Gemma 4 12b on my Macbook Pro M5 Max with 128GB of unified memory. It runs VERY slow for a 12b parameter model. 44 tokens/second. The lava lamp that it produced was also terrible. I would stick to Qwen 3.6 27b or Qwen 3.6 35b. Google is still way behind, even with their local models.

BridgeMind

70,626 Aufrufe • vor 4 Monaten

$A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.$

A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.

VirtualBacon

40,739 Aufrufe • vor 6 Monaten

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Introducing Kitten TTS, a SOTA tiny text-to-speech model - Just 15M parameters - Runs without a GPU - Model size less than 25 MB - Multiple high-quality voices - Ultra-fast - even runs on low-end edge devices Github and HF links below

Divam Gupta

349,016 Aufrufe • vor 1 Jahr

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

TWO BOXES THE SIZE OF A MAC MINI JUST RAN A 235 BILLION PARAMETER MODEL ON A DESK It is two NVIDIA DGX Spark units linked by a single cable. A year ago a model this size meant renting a GPU cluster by the hour. Now it sits next to your monitor for around $8,000. Here is the twist most people miss. Linking them does not create one shared 256GB memory pool. The model is split across both boxes, and that is the only reason a 235B model fits at all. It answers at roughly 10 tokens per second, and both chips sit at just 74 degrees while sipping around 50 watts. Every token stays on the desk. Nothing touches a cloud, and nothing leaves the room. The ceiling for what you can run at home just jumped from 70B to 235B. Bookmark this & Watch it run ↓

slash1s

101,317 Aufrufe • vor 4 Monaten

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

HERMES AGENT NOW RUNS ON AN 8GB LAPTOP GPU JUST AS EASILY AS IT RUNS ON A 128GB MINI PC Nous Research shipped the official Hermes Agent Desktop App this week. Someone pointed it at a local llama server running on an RTX 4060 with 16GB system RAM. The integration took two minutes The model behind it: Gemma 4 26B MoE, QAT quantized, running on 8GB of VRAM. A 60k token prompt held a stable 20 tokens a second, flat, no slowdown as context grew. The flags were nothing exotic, just -cmoe -c 248000 on llama.cpp What that 8GB setup does out of the box: reads and patches its own code, runs it in a terminal, debugs errors, manages GitHub repos, spawns sub-agents for parallel work. Browses the web with vision to debug a UI. Schedules cron jobs in plain language. Connects to Notion, Google Workspace, Linear, and Obsidian to manage tasks on its own That's the same agent layer running on a Minisforum MS-S1 MAX with 128GB of unified memory, 96GB of it to the GPU, holding a 120B model at 56 tokens a second instead of a 26B model at 20. Same software, same tool execution, same zero API key. The only thing that changes between an $800 laptop and a $2,000 mini PC is how big a model you can afford to run underneath it The barrier to running a real autonomous agent locally didn't just drop. It dropped all the way down to hardware most people already own

NO1ennn

40,079 Aufrufe • vor 4 Monaten

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with Red Hat and taught by Cedric Clyburn. Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management. Skills you'll gain: - Quantize a model and measure the accuracy tradeoff - Serve a model with vLLM and watch it handle concurrent requests efficiently - Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy Join and learn to serve LLMs efficiently:

Andrew Ng

136,970 Aufrufe • vor 4 Monaten

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Congrats to LemonSlice on their $10.5M seed! They built the world's first interactive talking AI video model—a face layer for voice agents. Trained on a custom, 20B-parameter video diffusion transformer, streaming at 20fps on a single GPU. Infinite-length video generation with no error accumulation.

Y Combinator

50,969 Aufrufe • vor 10 Monaten

$A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓$

A $351 MINI PC IS RUNNING 26-BILLION-PARAMETER AI MODELS AT 20 TOKENS/SEC AND HERMES AGENT ON TOP OF IT This is the Minisforum UM790 Pro. AMD Ryzen 9 7940HS, Radeon 780M iGPU, 48GB DDR5. The BIOS reports the GPU has 4GB of VRAM Here's the part people get wrong. The 780M has no dedicated VRAM at all it borrows from system RAM. Vulkan ignores the BIOS number and reads the full 48GB pool directly That's the whole trick. 21+ GB allocated to model weights on a "4GB" GPU. Because the models are MoE, only 4 billion of the parameters activate per token a fraction of the reads a dense model needs. That's why 20 tok/s works here Gemma 4 26B MoE holds 19.5 tok/s with 196K context. Qwen3.5-35B-A3B holds 20.8. Nemotron Cascade 2 clears 24.8. A dense 31B, by contrast, drops to 4 tok/s it reads the entire model every token, no way around it On top: Hermes Agent, full agentic workflows terminal, file ops, web, 40+ tools against local models only. No API keys. The wall between you and a usable local agent used to be a GPU you couldn't afford. Now it's a BIOS setting most people never check Bookmark this & Try it yourself ↓

slash1s

35,742 Aufrufe • vor 4 Monaten

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

he made 405-billion-parameter models run on a single 8GB gaming GPU. instead of turning it into a paid service, he open-sourced the whole thing. Gavin Li Founder and CEO of Anima AI in the Bay Area. 'airllm' - it keeps one layer on the GPU at a time and streams the rest, so a 70B model fits in 4GB and DeepSeek's full 671B fits in about 12GB 22,700 stars. Apache 2.0. → / he has 808 followers. / his repo has 22,700 stars. bookmark it. that's what shipping something impossible looks like.

dunik

111,308 Aufrufe • vor 3 Monaten

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

PERPLEXITY 🔥: A new Hybrid mode is coming soon on Mac, letting local models handle some subtasks. > Gemma 4 (for 16 GB machines), Qwen3.6, and Perplexity's own model (For 32 GB machines) are currently supported. > "In Hybrid mode, Computer orchestrates in the cloud and delegates suitable subtasks to a model on your Mac." > "Local models run on your Mac's own memory and processor — the more capable your machine, the larger the models it can run." > "Privacy Gate is a small model that runs entirely on your Mac. It inspects anything headed to the cloud for personal or sensitive information — and if it finds any, you decide how to handle it before anything is sent." That would be huge 👀

🚨 AI News | TestingCatalog

43,471 Aufrufe • vor 1 Monat

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

Clone any voice. Keep it local. Voice Clone Lab turns 5–15 minutes of audio into a voice model on your own GPU, cleanup, transcription, fine-tuning, and generation included. Open source: github/tetsuo-ai/voice_clone_lab

tetsuo

66,025 Aufrufe • vor 3 Monaten

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

Run StableDAW on your PC with 1 click (No GPU needed) A full AI music studio powered by Stable Audio 3: generate tracks from text, edit them on a timeline, master them, even DJ them --- 100% local. Created by GANTASMO No GPU required. All OS supported: Windows, Linux, and Mac.

cocktail peanut

16,863 Aufrufe • vor 4 Monaten

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

Hikari∣LocalLLM⚡

262,567 Aufrufe • vor 10 Tagen

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

This is from Apple's State of the Union The local model is a 3B parameter SLM that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. Anything running locally or Apple's Secure Cloud is an Apple model, not OpenAI.

Max Weinbach

2,648,758 Aufrufe • vor 2 Jahren

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

APPLE RESEARCH SCIENTIST JUST SHOWED HOW 4 MAC STUDIOS RUN A TRILLION PARAMETER MODEL LOCALLY ZERO COSTS 13:18 she shows the main thing - connect 4 Mac Studios and you get 1TB of shared memory - exactly enough to run a trillion parameter model right on your desk Apple's library - and four machines start working as one cluster tensor parallelism: every machine holds part of every layer - all process the same token simultaneously - speed increases 3x compared to a single device fine-tuning: one Mac Studio processes 180 tokens per second four together process 600 and not a single byte of data leaves the room one command in the terminal - and a trillion parameter model answers from your desk and runs 24/7 data centers took years to build to run models like this - Apple did it with four Thunderbolt cables

Noisy

48,895 Aufrufe • vor 4 Monaten