Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push... up the frontier specifically for local hardware. Better kernels, inference tricks like spec decoding, quantizing models, model routing. 2. Usability / UX. To use Local AI today, you need to install CLIs, select a model, a quantization, a harness that works best with that model, and you need to configure your inference engine. It's not very accessible. More work needs to be done on making this seamless, so the user doesn't even know the AI is running locally.show more

Alex Cheema

54,817 subscribers

30,854 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

30 Kommentare

Profilbild von DegenApeDev

DegenApeDevvor 3 Monaten

How can I get an exo hoodie?

Profilbild von Alex Cheema

Alex Cheemavor 3 Monaten

We're hosting our first ever in-person event soon in SF. We'll have merch there.

Profilbild von DegenApeDev

DegenApeDevvor 3 Monaten

Oh wow, lucky for whoever goes!

Profilbild von Daniel Isaac

Daniel Isaacvor 3 Monaten

can I get early access?

Profilbild von Blake Fullerton

Blake Fullertonvor 3 Monaten

100%… also spot on with the vLLM part

Profilbild von sparkarena

sparkarenavor 3 Monaten

EXO is doing an amazing work to push local AI to the masses. Particularly if you run heterogeneous deployments. With Spark Arena’s DGX Spark specific OSS projects is quite easy to run the latest vLLM, SGLang, llama.cpp, Atlas … versions on a single or multiple sparks using and also LLM recipes shared by real DGX Spark users around the world. In fact, just ask your agent to do it for you.

Profilbild von nirmata

nirmatavor 3 Monaten

Yes. When you buy a computer, you don't need to configure drivers to get audio output. We need personal compute clusters anyone can buy off the shelf and start generating tokens within minutes.

Profilbild von AI Mastery Guide

AI Mastery Guidevor 3 Monaten

The user shouldn't even know it's running locally is the right bar for usability honestly

Profilbild von #INTERNETofAGENTS

#INTERNETofAGENTSvor 3 Monaten

Make your local machine work fully before you pay for cloud. That’s the hard part and these guys are making it more accessible maybe even easier.

Profilbild von doodlifts

doodliftsvor 3 Monaten

@exolabs Hey I need this mad happy esque exo hoodie immediately, please and thank you

Profilbild von Ryan

Ryanvor 3 Monaten

You need to get GLM-5.2 quality on single gpus if you really want to get local mass adopted

Profilbild von Volatile Markets

Volatile Marketsvor 3 Monaten

Enjoyed this. Excited to see what you have up next. I’m very pleased with @exolabs, Pi, GLM 5.2, and multiple memory systems on pre hooks with post hooks to memory for growth. I’m confident this is early days.. but most of my life I’m early and clunky… then someone comes in and streamlines all of it.. but that discovery and error curve is how I have always been able to stay investing in the right vehicles. Thanks to @exolabs and all the builders helping us break new ground.

Profilbild von Rameswar

Rameswarvor 3 Monaten

100%, most people don't care if it's local or cloud, they care if it opens instantly and just works, the infrastructure should be invisible

Profilbild von Faraday Machines

Faraday Machinesvor 3 Monaten

I need this!!

Profilbild von Nil Ni 👶🦊🗝🎮🐷💎🎆

Nil Ni 👶🦊🗝🎮🐷💎🎆vor 3 Monaten

@exolabs How many Mac Studio do I need to get GLM running with 5-10 tps on a local machine?

Profilbild von nikhil tayal

nikhil tayalvor 3 Monaten

@exolabs Correct. OSS AI will run local and on the cloud. We are addressing the later with ease of use addressed out of the box.

Profilbild von Thomas A. Anderson

Thomas A. Andersonvor 3 Monaten

W/ NVFP4 & MTP 5 Im getting 140 tok/s generation on the full glm-5.2 with the least used expert at 3 bit to make it fit. Uncompressed it would be 1.5 tb instead of 327 while losing only 1 point on GPQA Diamond.

Profilbild von Alex Cheema

Alex Cheemavor 3 Monaten

What hardware

Profilbild von Thomas A. Anderson

Thomas A. Andersonvor 3 Monaten

4 x rtx pro 6000

Profilbild von Forge Cao

Forge Caovor 3 Monaten

Strongly agree — clusters are a dead end for mass adoption. Axis 1 is already paying off via architecture: a 35B-A3B MoE (4-bit) does ~84 tok/s in 22GB on a single 64GB Mac — no cluster needed. Axis 2 is the real gap — I live the model/quant/harness config hell daily.

Profilbild von Saeed Anwar

Saeed Anwarvor 3 Monaten

You nailed the two axes but missed the third: reliability. Mass adoption needs models that don't hallucinate your grandma's medical dosage. Intelligence and speed mean nothing if normies can't trust the output without verification. Who's working on that trust layer?

Profilbild von BowTied Fullstack - Link in bio or NGMI

BowTied Fullstack - Link in bio or NGMIvor 3 Monaten

Agreed, but all I care about is hosting Z GLM 5.2 at home.

Profilbild von Rompel

Rompelvor 3 Monaten

Local doesn't die on the frontier — it dies at "pick a quant and a harness." Ollama got closest by hiding it all behind one pull. Endgame: the app never says "local," it just runs. Frontier is table stakes; invisibility is the moat.

Profilbild von Jai Bhagat

Jai Bhagatvor 3 Monaten

@exolabs Who is working on UX for local AI right now?

Profilbild von Darren Barklie

Darren Barklievor 3 Monaten

This form needs a few more tokens :)

Profilbild von Dan Holzrichter

Dan Holzrichtervor 3 Monaten

Lack of vision really gimps the model

Profilbild von Abhay अभय

Abhay अभयvor 3 Monaten

The ecosystem of opensource currently is fragmented and makes it laborious if not difficult to start building local inferencing infrastructure, once opensource nails this we can be better option for those who have access to hardware

Profilbild von FortSignal

FortSignalvor 3 Monaten

Right

Profilbild von ML Papers, Distilled

ML Papers, Distilledvor 3 Monaten

decode speed's the wall for local, not throughput. but benchmarks still chase inference latency. what's the real blocker in your deployments?

Profilbild von Nick

Nickvor 3 Monaten

Would love access to local dot ai. Running gpt-oss-120b (76 tok/s), Qwen3-30B, and Gemma 4 12B via Ollama on a Mac Studio M3 Ultra, served over Tailscale to the rest of my machines. Claude Code daily in the terminal. RE developer using AI for market intelligence — moving more inference on-device.

Ähnliche Videos

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

63,883 Aufrufe • vor 5 Monaten

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,676 Aufrufe • vor 5 Monaten

Google whistleblower Zach Vorhies joined us on Decentralize TV to discuss how frontier-level AI models are going to be increasingly run LOCALLY on local hardware, without censorship or access restrictions. The local hardware is catching up! (This is the path to decentralized machine cognition...)

Google whistleblower Zach Vorhies joined us on Decentralize TV to discuss how frontier-level AI models are going to be increasingly run LOCALLY on local hardware, without censorship or access restrictions. The local hardware is catching up! (This is the path to decentralized machine cognition...)

HealthRanger

11,923 Aufrufe • vor 2 Monaten

A $170 AI bill in 10 days is not a cloud problem. It is a routing problem. Most devs send every task to the most expensive brain available. summaries small refactors docs scripts log parsing boilerplate That is how the bill gets stupid. The Mac Mini M4 trade is simple: run routine work locally keep private code on your desk pay once for hardware save frontier models for hard calls Local AI does not beat Claude at everything. It does not need to. It only needs to replace the boring prompts that never needed a frontier model in the first place. Cloud AI stays. The default route changes. Full breakdown below ↓

A $170 AI bill in 10 days is not a cloud problem. It is a routing problem. Most devs send every task to the most expensive brain available. summaries small refactors docs scripts log parsing boilerplate That is how the bill gets stupid. The Mac Mini M4 trade is simple: run routine work locally keep private code on your desk pay once for hardware save frontier models for hard calls Local AI does not beat Claude at everything. It does not need to. It only needs to replace the boring prompts that never needed a frontier model in the first place. Cloud AI stays. The default route changes. Full breakdown below ↓

Fluixo

35,295 Aufrufe • vor 2 Monaten

Today we're announcing that hybrid agentic inference is coming to Perplexity Computer. Computer can split tasks between a local model running on your machine and frontier models in the cloud. This keeps private data on your device and maximizes token efficiency. Coming soon.

Today we're announcing that hybrid agentic inference is coming to Perplexity Computer. Computer can split tasks between a local model running on your machine and frontier models in the cloud. This keeps private data on your device and maximizes token efficiency. Coming soon.

Perplexity

354,430 Aufrufe • vor 5 Monaten

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀

Jack Huynh

50,015 Aufrufe • vor 5 Monaten

1/ Excited to launch an experiment today - introducing Window, a way to use your own AI models on the web - including local ones! It's a bet on a new kind of AI app emerging, one that shifts model authentication and management to the user.

1/ Excited to launch an experiment today - introducing Window, a way to use your own AI models on the web - including local ones! It's a bet on a new kind of AI app emerging, one that shifts model authentication and management to the user.

Alex Atallah

390,173 Aufrufe • vor 3 Jahren

Do you see what's happening? Local AI is getting smarter. It's Opus 4.5 level Hardware that hosts those models are flying off the shelf Your window to host local models is shrinking In this video I cover EVERYTHING local AI models and how to host your own, even on Mac Minis:

Do you see what's happening? Local AI is getting smarter. It's Opus 4.5 level Hardware that hosts those models are flying off the shelf Your window to host local models is shrinking In this video I cover EVERYTHING local AI models and how to host your own, even on Mac Minis:

Alex Finn

55,389 Aufrufe • vor 7 Monaten

Engram cofounder Jack Morris just raised $98M to build a new type of AI. He says models don't need to get smarter over time. Instead, they just need to know you better and better over time. Jack describes what he's building: "Our product is a new type of AI. We have a pretty different vision from a lot of the frontier labs, which are working on one model per lab, and trying to make that model smarter every month." "There's another way to think about it, which is that the model doesn't need to get smarter every month. It needs to know you better." "So we're working on a whole different stack, which is a way to train models that train themselves to know your world better and adjust to the things that you say." "So: new ways of training, new ways of running the models."

Engram cofounder Jack Morris just raised $98M to build a new type of AI. He says models don't need to get smarter over time. Instead, they just need to know you better and better over time. Jack describes what he's building: "Our product is a new type of AI. We have a pretty different vision from a lot of the frontier labs, which are working on one model per lab, and trying to make that model smarter every month." "There's another way to think about it, which is that the model doesn't need to get smarter every month. It needs to know you better." "So we're working on a whole different stack, which is a way to train models that train themselves to know your world better and adjust to the things that you say." "So: new ways of training, new ways of running the models."

TBPN

70,797 Aufrufe • vor 4 Monaten

The easiest way to run your agent harnesses using local models: Whether you're using Claude Code, Codex, OpenCode, or Pi, the local setup itself isn't really the hard part. The harder problem is figuring out which local model your machine can actually handle well. You have to think about RAM, model size, quantization, context length, KV cache, speed, accuracy, and a bunch of other trade-offs before you even start. Magnitude is an open-source solution that removes all of that guesswork for you. It profiles your machine, benchmarks what it can realistically run, recommends the best models for your hardware, and then lets you connect them to your preferred agent harness. The entire setup takes just two commands. In this video, I walk through the full process from hardware profiling to running a harness on a local model. Chapters: 00:00 Intro 00:17 What Magnitude is 00:45 Setup in two commands 01:29 Why picking a local model gets confusing fast 02:11 Hardware profiling + model recommendations 02:59 Choosing your harness 03:22 Connecting a harness that's already running 04:18 Outro Get started: (don't forget to star 🌟)

The easiest way to run your agent harnesses using local models: Whether you're using Claude Code, Codex, OpenCode, or Pi, the local setup itself isn't really the hard part. The harder problem is figuring out which local model your machine can actually handle well. You have to think about RAM, model size, quantization, context length, KV cache, speed, accuracy, and a bunch of other trade-offs before you even start. Magnitude is an open-source solution that removes all of that guesswork for you. It profiles your machine, benchmarks what it can realistically run, recommends the best models for your hardware, and then lets you connect them to your preferred agent harness. The entire setup takes just two commands. In this video, I walk through the full process from hardware profiling to running a harness on a local model. Chapters: 00:00 Intro 00:17 What Magnitude is 00:45 Setup in two commands 01:29 Why picking a local model gets confusing fast 02:11 Hardware profiling + model recommendations 02:59 Choosing your harness 03:22 Connecting a harness that's already running 04:18 Outro Get started: (don't forget to star 🌟)

Akshay 🚀

18,722 Aufrufe • vor 1 Monat

Local AI v6.11.2 on Omarchy is out. Look how I'm running GLM-5.3-Flash on a SINGLE RTX 3090 at exl3-3bpw What I'm doing is putting parts of the model on NVMe, parts on DDR4 RAM, and the GPU is used as a station of sorts, we send the math to be done there. I value freedom

Local AI v6.11.2 on Omarchy is out. Look how I'm running GLM-5.3-Flash on a SINGLE RTX 3090 at exl3-3bpw What I'm doing is putting parts of the model on NVMe, parts on DDR4 RAM, and the GPU is used as a station of sorts, we send the math to be done there. I value freedom

0xSero

13,338 Aufrufe • vor 27 Tagen

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:

Alex Finn

125,207 Aufrufe • vor 4 Monaten

Perplexity CEO Aravind Srinivas on the biggest threat to the data center industry: It's not competition. It's not regulation. It's decentralisation. "The biggest threat to a data center is if the intelligence can be packed locally on a chip that's running on the device and then there's no need to inference all of it on like one centralized data center." He outlines how this could work in practice. Personalisation doesn't necessarily require on-device model training. Retrieval augmented generation, tool calls, and local data can already tailor AI to individual users. But the real unlock? Test time training. Aravind Srinivas describes a future where AI lives on your device, watches how you work and gradually automates your repetitive tasks. "Imagine we crack test time training where the AI watches tasks you repeatedly do on your local system, adapts to you over time and starts automating a lot of the things you do." The key insight: in this model, the intelligence belongs to you. It's your data, your device, your personalised AI brain. And if that future arrives, the economics of centralised infrastructure start to collapse. "That really disrupts the whole data center industry. It doesn't make sense to spend all this money, 500 billion, 5 trillion, whatever on building all the centralized data centers across the world that do a lot of the intelligence workloads for people." The companies spending trillions on centralised infrastructure may want to rethink where intelligence actually needs to live.

Perplexity CEO Aravind Srinivas on the biggest threat to the data center industry: It's not competition. It's not regulation. It's decentralisation. "The biggest threat to a data center is if the intelligence can be packed locally on a chip that's running on the device and then there's no need to inference all of it on like one centralized data center." He outlines how this could work in practice. Personalisation doesn't necessarily require on-device model training. Retrieval augmented generation, tool calls, and local data can already tailor AI to individual users. But the real unlock? Test time training. Aravind Srinivas describes a future where AI lives on your device, watches how you work and gradually automates your repetitive tasks. "Imagine we crack test time training where the AI watches tasks you repeatedly do on your local system, adapts to you over time and starts automating a lot of the things you do." The key insight: in this model, the intelligence belongs to you. It's your data, your device, your personalised AI brain. And if that future arrives, the economics of centralised infrastructure start to collapse. "That really disrupts the whole data center industry. It doesn't make sense to spend all this money, 500 billion, 5 trillion, whatever on building all the centralized data centers across the world that do a lot of the intelligence workloads for people." The companies spending trillions on centralised infrastructure may want to rethink where intelligence actually needs to live.

Big Brain AI

90,355 Aufrufe • vor 8 Monaten

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Sagar

13,738 Aufrufe • vor 4 Monaten

Alex Finn reveals that you need to spend $20,000 to get the equivalent of Opus 4.6 on a local model "The advantages are that you aren’t paying for tokens, all you have to pay for is electricity.” "The fact you're not paying for tokens means you can run your model 24/7 365, you can't do that with Opus" “If you run it locally it’s a completely private system. Nothing you say is private if you're not on a local model”

Alex Finn reveals that you need to spend $20,000 to get the equivalent of Opus 4.6 on a local model "The advantages are that you aren’t paying for tokens, all you have to pay for is electricity.” "The fact you're not paying for tokens means you can run your model 24/7 365, you can't do that with Opus" “If you run it locally it’s a completely private system. Nothing you say is private if you're not on a local model”

Mikli

85,206 Aufrufe • vor 8 Monaten

ICYMI: the last month’s biggest local AI model drops... Check out the latest—all ready to run locally on your RTX AI PC.

ICYMI: the last month’s biggest local AI model drops... Check out the latest—all ready to run locally on your RTX AI PC.

NVIDIA RTX Spark

145,162 Aufrufe • vor 10 Monaten

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

Ejaaz

136,637 Aufrufe • vor 4 Monaten

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶

Asuka Zheng🎀@ICML

178,998 Aufrufe • vor 11 Monaten

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.

KyzoroX

35,554 Aufrufe • vor 1 Monat

It's here: We just hit superhuman performance on AI kernel optimization! Real customer models & production settings. Not toy problems (what I typically see). This is the year that Claude writes its own kernels, Codex its own kernels, for every new GPU that it wants to run on -- something that takes months to port between GPU generations today. This has a massive impact to scaling intelligence. More compute means getting the next frontier model sooner.

It's here: We just hit superhuman performance on AI kernel optimization! Real customer models & production settings. Not toy problems (what I typically see). This is the year that Claude writes its own kernels, Codex its own kernels, for every new GPU that it wants to run on -- something that takes months to port between GPU generations today. This has a massive impact to scaling intelligence. More compute means getting the next frontier model sooner.

Sharon Zhou

247,624 Aufrufe • vor 7 Monaten