Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

GLM-5.2 running on Mac Studio Clusters or RTX GPU rigs isn't how Local AI will get mass adopted. For mass adoption we need to focus on two axes: 1. Intelligence/Speed Frontier. This is being tracked on local dot ai. Aggressive co-design is needed across the entire stack to push...

alexocheema's profile picture

Alex Cheema

54,817 subscribers

30,854 görüntüleme • 3 ay önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

30 Yorum

DegenApeDev profil fotoğrafı
DegenApeDev3 ay önce

How can I get an exo hoodie?

Alex Cheema profil fotoğrafı
Alex Cheema3 ay önce

We're hosting our first ever in-person event soon in SF. We'll have merch there.

DegenApeDev profil fotoğrafı
DegenApeDev3 ay önce

Oh wow, lucky for whoever goes!

Daniel Isaac profil fotoğrafı
Daniel Isaac3 ay önce

can I get early access?

Blake Fullerton profil fotoğrafı
Blake Fullerton3 ay önce

100%… also spot on with the vLLM part

sparkarena profil fotoğrafı
sparkarena3 ay önce

EXO is doing an amazing work to push local AI to the masses. Particularly if you run heterogeneous deployments. With Spark Arena’s DGX Spark specific OSS projects is quite easy to run the latest vLLM, SGLang, llama.cpp, Atlas … versions on a single or multiple sparks using and also LLM recipes shared by real DGX Spark users around the world. In fact, just ask your agent to do it for you.

nirmata profil fotoğrafı
nirmata3 ay önce

Yes. When you buy a computer, you don't need to configure drivers to get audio output. We need personal compute clusters anyone can buy off the shelf and start generating tokens within minutes.

AI Mastery Guide profil fotoğrafı
AI Mastery Guide3 ay önce

The user shouldn't even know it's running locally is the right bar for usability honestly

#INTERNETofAGENTS profil fotoğrafı
#INTERNETofAGENTS3 ay önce

Make your local machine work fully before you pay for cloud. That’s the hard part and these guys are making it more accessible maybe even easier.

doodlifts profil fotoğrafı
doodlifts3 ay önce

@exolabs Hey I need this mad happy esque exo hoodie immediately, please and thank you

Ryan profil fotoğrafı
Ryan3 ay önce

You need to get GLM-5.2 quality on single gpus if you really want to get local mass adopted

Volatile Markets profil fotoğrafı
Volatile Markets3 ay önce

Enjoyed this. Excited to see what you have up next. I’m very pleased with @exolabs, Pi, GLM 5.2, and multiple memory systems on pre hooks with post hooks to memory for growth. I’m confident this is early days.. but most of my life I’m early and clunky… then someone comes in and streamlines all of it.. but that discovery and error curve is how I have always been able to stay investing in the right vehicles. Thanks to @exolabs and all the builders helping us break new ground.

Rameswar profil fotoğrafı
Rameswar3 ay önce

100%, most people don't care if it's local or cloud, they care if it opens instantly and just works, the infrastructure should be invisible

Faraday Machines profil fotoğrafı
Faraday Machines3 ay önce

I need this!!

Nil Ni 👶🦊🗝🎮🐷💎🎆 profil fotoğrafı
Nil Ni 👶🦊🗝🎮🐷💎🎆3 ay önce

@exolabs How many Mac Studio do I need to get GLM running with 5-10 tps on a local machine?

nikhil tayal profil fotoğrafı
nikhil tayal3 ay önce

@exolabs Correct. OSS AI will run local and on the cloud. We are addressing the later with ease of use addressed out of the box.

Thomas A. Anderson profil fotoğrafı
Thomas A. Anderson3 ay önce

W/ NVFP4 & MTP 5 Im getting 140 tok/s generation on the full glm-5.2 with the least used expert at 3 bit to make it fit. Uncompressed it would be 1.5 tb instead of 327 while losing only 1 point on GPQA Diamond.

Alex Cheema profil fotoğrafı
Alex Cheema3 ay önce

What hardware

Thomas A. Anderson profil fotoğrafı
Thomas A. Anderson3 ay önce

4 x rtx pro 6000

Forge Cao profil fotoğrafı
Forge Cao3 ay önce

Strongly agree — clusters are a dead end for mass adoption. Axis 1 is already paying off via architecture: a 35B-A3B MoE (4-bit) does ~84 tok/s in 22GB on a single 64GB Mac — no cluster needed. Axis 2 is the real gap — I live the model/quant/harness config hell daily.

Saeed Anwar profil fotoğrafı
Saeed Anwar3 ay önce

You nailed the two axes but missed the third: reliability. Mass adoption needs models that don't hallucinate your grandma's medical dosage. Intelligence and speed mean nothing if normies can't trust the output without verification. Who's working on that trust layer?

BowTied Fullstack - Link in bio or NGMI profil fotoğrafı
BowTied Fullstack - Link in bio or NGMI3 ay önce

Agreed, but all I care about is hosting Z GLM 5.2 at home.

Rompel profil fotoğrafı
Rompel3 ay önce

Local doesn't die on the frontier — it dies at "pick a quant and a harness." Ollama got closest by hiding it all behind one pull. Endgame: the app never says "local," it just runs. Frontier is table stakes; invisibility is the moat.

Jai Bhagat profil fotoğrafı
Jai Bhagat3 ay önce

@exolabs Who is working on UX for local AI right now?

Darren Barklie profil fotoğrafı
Darren Barklie3 ay önce

This form needs a few more tokens :)

Dan Holzrichter profil fotoğrafı
Dan Holzrichter3 ay önce

Lack of vision really gimps the model

Abhay अभय profil fotoğrafı
Abhay अभय3 ay önce

The ecosystem of opensource currently is fragmented and makes it laborious if not difficult to start building local inferencing infrastructure, once opensource nails this we can be better option for those who have access to hardware

FortSignal profil fotoğrafı
FortSignal3 ay önce

Right

ML Papers, Distilled profil fotoğrafı
ML Papers, Distilled3 ay önce

decode speed's the wall for local, not throughput. but benchmarks still chase inference latency. what's the real blocker in your deployments?

Nick profil fotoğrafı
Nick3 ay önce

Would love access to local dot ai. Running gpt-oss-120b (76 tok/s), Qwen3-30B, and Gemma 4 12B via Ollama on a Mac Studio M3 Ultra, served over Tailscale to the rest of my machines. Claude Code daily in the terminal. RE developer using AI for market intelligence — moving more inference on-device.

Benzer Videolar

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.
1:04

Qwen 3.6 35B running at over 100 tokens per second on my $5,399 MacBook Pro M5 Max. This is the best local AI model I have ever run. 128GB of unified memory. No cloud. No API costs. No rate limits. Just raw local inference at speeds I didn't think were possible on a laptop. This model is more intelligent than GPT 5 on benchmarks. Running locally. On a MacBook. For free after the hardware cost. I said local AI would never compete with frontier. I'm starting to rethink that. The gap is closing faster than anyone expected.

BridgeMind

63,883 görüntüleme • 5 ay önce

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."
2:32

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,676 görüntüleme • 5 ay önce

Google whistleblower Zach Vorhies joined us on Decentralize TV to discuss how frontier-level AI models are going to be increasingly run LOCALLY on local hardware, without censorship or access restrictions. The local hardware is catching up! (This is the path to decentralized machine cognition...)
3:12

Google whistleblower Zach Vorhies joined us on Decentralize TV to discuss how frontier-level AI models are going to be increasingly run LOCALLY on local hardware, without censorship or access restrictions. The local hardware is catching up! (This is the path to decentralized machine cognition...)

HealthRanger

11,923 görüntüleme • 2 ay önce

A $170 AI bill in 10 days is not a cloud problem. It is a routing problem. Most devs send every task to the most expensive brain available. summaries small refactors docs scripts log parsing boilerplate That is how the bill gets stupid. The Mac Mini M4 trade is simple: run routine work locally keep private code on your desk pay once for hardware save frontier models for hard calls Local AI does not beat Claude at everything. It does not need to. It only needs to replace the boring prompts that never needed a frontier model in the first place. Cloud AI stays. The default route changes. Full breakdown below ↓
1:01

A $170 AI bill in 10 days is not a cloud problem. It is a routing problem. Most devs send every task to the most expensive brain available. summaries small refactors docs scripts log parsing boilerplate That is how the bill gets stupid. The Mac Mini M4 trade is simple: run routine work locally keep private code on your desk pay once for hardware save frontier models for hard calls Local AI does not beat Claude at everything. It does not need to. It only needs to replace the boring prompts that never needed a frontier model in the first place. Cloud AI stays. The default route changes. Full breakdown below ↓

Fluixo

35,295 görüntüleme • 2 ay önce

Today we're announcing that hybrid agentic inference is coming to Perplexity Computer. Computer can split tasks between a local model running on your machine and frontier models in the cloud. This keeps private data on your device and maximizes token efficiency. Coming soon.
0:59

Today we're announcing that hybrid agentic inference is coming to Perplexity Computer. Computer can split tasks between a local model running on your machine and frontier models in the cloud. This keeps private data on your device and maximizes token efficiency. Coming soon.

Perplexity

354,591 görüntüleme • 5 ay önce

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀
4:49

✨ Personal AI is the next computing platform. AI is shifting from something you access to something you build with, locally, at the edge, and across systems. We’re unlocking new possibilities for developers: • AMD Ryzen AI Halo, a local-first developer system, preorder starting in June, develop AI without limits on your desk • Gorgon Halo with up to 192GB unified memory, supporting 300B+ parameter models locally, run massive models locally We’re excited to partner with clem 🤗🤗, Co-founder and CEO of Hugging Face, to advance open-source AI for Ryzen AI. Our focus is seamless AI, from model to deployment. Cloud, edge, device. One continuum. Multi-agent systems, local inference at scale, open models as infrastructure. This is the next computing era 🚀

Jack Huynh

50,015 görüntüleme • 5 ay önce

1/ Excited to launch an experiment today - introducing Window, a way to use your own AI models on the web - including local ones! It's a bet on a new kind of AI app emerging, one that shifts model authentication and management to the user.
0:52

1/ Excited to launch an experiment today - introducing Window, a way to use your own AI models on the web - including local ones! It's a bet on a new kind of AI app emerging, one that shifts model authentication and management to the user.

Alex Atallah

390,176 görüntüleme • 3 yıl önce

Do you see what's happening? Local AI is getting smarter. It's Opus 4.5 level Hardware that hosts those models are flying off the shelf Your window to host local models is shrinking In this video I cover EVERYTHING local AI models and how to host your own, even on Mac Minis:
21:26

Do you see what's happening? Local AI is getting smarter. It's Opus 4.5 level Hardware that hosts those models are flying off the shelf Your window to host local models is shrinking In this video I cover EVERYTHING local AI models and how to host your own, even on Mac Minis:

Alex Finn

55,389 görüntüleme • 7 ay önce

Engram cofounder Jack Morris just raised $98M to build a new type of AI. He says models don't need to get smarter over time. Instead, they just need to know you better and better over time. Jack describes what he's building: "Our product is a new type of AI. We have a pretty different vision from a lot of the frontier labs, which are working on one model per lab, and trying to make that model smarter every month." "There's another way to think about it, which is that the model doesn't need to get smarter every month. It needs to know you better." "So we're working on a whole different stack, which is a way to train models that train themselves to know your world better and adjust to the things that you say." "So: new ways of training, new ways of running the models."
2:06

Engram cofounder Jack Morris just raised $98M to build a new type of AI. He says models don't need to get smarter over time. Instead, they just need to know you better and better over time. Jack describes what he's building: "Our product is a new type of AI. We have a pretty different vision from a lot of the frontier labs, which are working on one model per lab, and trying to make that model smarter every month." "There's another way to think about it, which is that the model doesn't need to get smarter every month. It needs to know you better." "So we're working on a whole different stack, which is a way to train models that train themselves to know your world better and adjust to the things that you say." "So: new ways of training, new ways of running the models."

TBPN

70,850 görüntüleme • 4 ay önce

The easiest way to run your agent harnesses using local models: Whether you're using Claude Code, Codex, OpenCode, or Pi, the local setup itself isn't really the hard part. The harder problem is figuring out which local model your machine can actually handle well. You have to think about RAM, model size, quantization, context length, KV cache, speed, accuracy, and a bunch of other trade-offs before you even start. Magnitude is an open-source solution that removes all of that guesswork for you. It profiles your machine, benchmarks what it can realistically run, recommends the best models for your hardware, and then lets you connect them to your preferred agent harness. The entire setup takes just two commands. In this video, I walk through the full process from hardware profiling to running a harness on a local model. Chapters: 00:00 Intro 00:17 What Magnitude is 00:45 Setup in two commands 01:29 Why picking a local model gets confusing fast 02:11 Hardware profiling + model recommendations 02:59 Choosing your harness 03:22 Connecting a harness that's already running 04:18 Outro Get started: (don't forget to star 🌟)
4:29

The easiest way to run your agent harnesses using local models: Whether you're using Claude Code, Codex, OpenCode, or Pi, the local setup itself isn't really the hard part. The harder problem is figuring out which local model your machine can actually handle well. You have to think about RAM, model size, quantization, context length, KV cache, speed, accuracy, and a bunch of other trade-offs before you even start. Magnitude is an open-source solution that removes all of that guesswork for you. It profiles your machine, benchmarks what it can realistically run, recommends the best models for your hardware, and then lets you connect them to your preferred agent harness. The entire setup takes just two commands. In this video, I walk through the full process from hardware profiling to running a harness on a local model. Chapters: 00:00 Intro 00:17 What Magnitude is 00:45 Setup in two commands 01:29 Why picking a local model gets confusing fast 02:11 Hardware profiling + model recommendations 02:59 Choosing your harness 03:22 Connecting a harness that's already running 04:18 Outro Get started: (don't forget to star 🌟)

Akshay 🚀

18,722 görüntüleme • 1 ay önce

Local AI v6.11.2 on Omarchy is out. Look how I'm running GLM-5.3-Flash on a SINGLE RTX 3090 at exl3-3bpw What I'm doing is putting parts of the model on NVMe, parts on DDR4 RAM, and the GPU is used as a station of sorts, we send the math to be done there. I value freedom
2:55

Local AI v6.11.2 on Omarchy is out. Look how I'm running GLM-5.3-Flash on a SINGLE RTX 3090 at exl3-3bpw What I'm doing is putting parts of the model on NVMe, parts on DDR4 RAM, and the GPU is used as a station of sorts, we send the math to be done there. I value freedom

0xSero

13,338 görüntüleme • 28 gün önce

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:
18:30

Truly unbelievable GLM 5.2 just released and it's an open weights model you can run locally The insane part is, it's just as good as Opus 4.8 Unlimited, free super intelligence running on your desk In this video I cover how it works, and how to set up your first local model:

Alex Finn

125,273 görüntüleme • 4 ay önce

Perplexity CEO Aravind Srinivas on the biggest threat to the data center industry: It's not competition. It's not regulation. It's decentralisation. "The biggest threat to a data center is if the intelligence can be packed locally on a chip that's running on the device and then there's no need to inference all of it on like one centralized data center." He outlines how this could work in practice. Personalisation doesn't necessarily require on-device model training. Retrieval augmented generation, tool calls, and local data can already tailor AI to individual users. But the real unlock? Test time training. Aravind Srinivas describes a future where AI lives on your device, watches how you work and gradually automates your repetitive tasks. "Imagine we crack test time training where the AI watches tasks you repeatedly do on your local system, adapts to you over time and starts automating a lot of the things you do." The key insight: in this model, the intelligence belongs to you. It's your data, your device, your personalised AI brain. And if that future arrives, the economics of centralised infrastructure start to collapse. "That really disrupts the whole data center industry. It doesn't make sense to spend all this money, 500 billion, 5 trillion, whatever on building all the centralized data centers across the world that do a lot of the intelligence workloads for people." The companies spending trillions on centralised infrastructure may want to rethink where intelligence actually needs to live.
1:15

Perplexity CEO Aravind Srinivas on the biggest threat to the data center industry: It's not competition. It's not regulation. It's decentralisation. "The biggest threat to a data center is if the intelligence can be packed locally on a chip that's running on the device and then there's no need to inference all of it on like one centralized data center." He outlines how this could work in practice. Personalisation doesn't necessarily require on-device model training. Retrieval augmented generation, tool calls, and local data can already tailor AI to individual users. But the real unlock? Test time training. Aravind Srinivas describes a future where AI lives on your device, watches how you work and gradually automates your repetitive tasks. "Imagine we crack test time training where the AI watches tasks you repeatedly do on your local system, adapts to you over time and starts automating a lot of the things you do." The key insight: in this model, the intelligence belongs to you. It's your data, your device, your personalised AI brain. And if that future arrives, the economics of centralised infrastructure start to collapse. "That really disrupts the whole data center industry. It doesn't make sense to spend all this money, 500 billion, 5 trillion, whatever on building all the centralized data centers across the world that do a lot of the intelligence workloads for people." The companies spending trillions on centralised infrastructure may want to rethink where intelligence actually needs to live.

Big Brain AI

90,355 görüntüleme • 8 ay önce

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!
0:28

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Sagar

13,738 görüntüleme • 4 ay önce

Alex Finn reveals that you need to spend $20,000 to get the equivalent of Opus 4.6 on a local model "The advantages are that you aren’t paying for tokens, all you have to pay for is electricity.” "The fact you're not paying for tokens means you can run your model 24/7 365, you can't do that with Opus" “If you run it locally it’s a completely private system. Nothing you say is private if you're not on a local model”
1:36

Alex Finn reveals that you need to spend $20,000 to get the equivalent of Opus 4.6 on a local model "The advantages are that you aren’t paying for tokens, all you have to pay for is electricity.” "The fact you're not paying for tokens means you can run your model 24/7 365, you can't do that with Opus" “If you run it locally it’s a completely private system. Nothing you say is private if you're not on a local model”

Mikli

85,206 görüntüleme • 8 ay önce

ICYMI: the last month’s biggest local AI model drops... Check out the latest—all ready to run locally on your RTX AI PC.
1:54

ICYMI: the last month’s biggest local AI model drops... Check out the latest—all ready to run locally on your RTX AI PC.

NVIDIA RTX Spark

145,162 görüntüleme • 10 ay önce

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched
1:30

i will die on this hill: Apple is a frontier AI company. they will inject AI into every place you scroll, type, read and listen imagine if you could “google” a past memory or ask siri to prove you’re right in a group chat ai models that run on-device with access to your entire life. the craziest part is the models don’t even need to be big! a 20B frontier model can run smoothly on apple’s chip architecture completely privately there are 3.5 billion apple devices, the gravitational pull this company has on societal culture is unmatched

Ejaaz

136,637 görüntüleme • 4 ay önce

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶
7:41

just built a local AI waifu on my two macbooks by using the oss parallax inference library!!! not super smooth yet but parallax is still the easiest way to run llms locally on top of mlx - and no bigco can steal my waifu. still need folks to help her see/hear/draw since mlx has no legit wrapper yet. i’m running a campaign giving away mac minis + dgx spark. please please share your localhost ai model setup by using parallax (you don't need a gpu to try self-hosting you can just start with an extra macbook you borrowed from your friend like i did) or any fun app built on top of yourself host model to link below. come build sovereign intelligence with me 🫶

Asuka Zheng🎀@ICML

178,998 görüntüleme • 11 ay önce

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.
0:27

We used to buy more VRAM. Now we’re finding ways around the VRAM limit. This little DDR5 SO-DIMM setup is paired with an old NVIDIA Tesla P40 in the video. The P40 is an old datacenter GPU with 24GB of VRAM. By today’s standards, it’s not fast. But combine that with a machine carrying 128GB of system memory, and suddenly you have a very different local AI box. The GPU handles what it’s good at. The system memory gives the model somewhere to go when VRAM runs out. You obviously pay for that with bandwidth and inference speed. But this is the part I find interesting: Local AI is making “how much memory do I have?” almost as important as “how powerful is my GPU?” We’ve spent years chasing faster GPUs. Maybe the next wave of local AI hardware will be about finding smarter ways to use all the memory we already have. More VRAM is still better. But it’s no longer the only way to run a bigger model.

KyzoroX

35,554 görüntüleme • 1 ay önce

It's here: We just hit superhuman performance on AI kernel optimization! Real customer models & production settings. Not toy problems (what I typically see). This is the year that Claude writes its own kernels, Codex its own kernels, for every new GPU that it wants to run on -- something that takes months to port between GPU generations today. This has a massive impact to scaling intelligence. More compute means getting the next frontier model sooner.
2:48

It's here: We just hit superhuman performance on AI kernel optimization! Real customer models & production settings. Not toy problems (what I typically see). This is the year that Claude writes its own kernels, Codex its own kernels, for every new GPU that it wants to run on -- something that takes months to port between GPU generations today. This has a massive impact to scaling intelligence. More compute means getting the next frontier model sooner.

Sharon Zhou

247,624 görüntüleme • 8 ay önce