Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.

Dria

14,575 subscribers

26,004 views • 6 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,525 views • 21 days ago

🇵🇸 🇩🇪 “We are well aware that today a two State solution appears more distant than ever. We are well aware that steps are being taken as we speak to bring it out of reach. We are meeting today because we are convinced there can be a better future.”

🇵🇸 🇩🇪 “We are well aware that today a two State solution appears more distant than ever. We are well aware that steps are being taken as we speak to bring it out of reach. We are meeting today because we are convinced there can be a better future.”

State of Palestine

75,902 views • 8 months ago

‘Please look, study’ - Putin presented with a catalogue of all destroyed and desecrated Soviet/Russian monuments in Ukraine and Europe ‘This is being done to erase the historical memory of who we are, where we came from. It is being done so that we have no future. It is also being done to erase the memory of those who look towards Russia’

‘Please look, study’ - Putin presented with a catalogue of all destroyed and desecrated Soviet/Russian monuments in Ukraine and Europe ‘This is being done to erase the historical memory of who we are, where we came from. It is being done so that we have no future. It is also being done to erase the memory of those who look towards Russia’

Zlatti71

14,766 views • 1 year ago

Geometric Context Transformer for Streaming 3D Reconstruction Contributions: • We introduce LingBot-Map, a streaming 3D foundation model built around Geometric Context Attention (GCA), which maintains three complementary context types – anchor, pose-reference window, and trajectory memory – for efficient and consistent long-sequence streaming inference. • We propose an efficient training recipe based on progressive training and context parallelism with a relative loss formulation for stable long-sequence optimization. • We demonstrate that LingBot-Map achieves state-of-the-art performance on multiple benchmarks (Oxford Spires, Tanks and Temples, ETH3D, and 7-Scenes), significantly outperforming existing streaming approaches in reconstruction quality and inference speed.

Geometric Context Transformer for Streaming 3D Reconstruction Contributions: • We introduce LingBot-Map, a streaming 3D foundation model built around Geometric Context Attention (GCA), which maintains three complementary context types – anchor, pose-reference window, and trajectory memory – for efficient and consistent long-sequence streaming inference. • We propose an efficient training recipe based on progressive training and context parallelism with a relative loss formulation for stable long-sequence optimization. • We demonstrate that LingBot-Map achieves state-of-the-art performance on multiple benchmarks (Oxford Spires, Tanks and Temples, ETH3D, and 7-Scenes), significantly outperforming existing streaming approaches in reconstruction quality and inference speed.

MrNeRF

24,549 views • 1 month ago

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

Deirdre Bosa

83,925 views • 25 days ago

Today Ukraine honors the memory of millions of victims of the Holocaust. We know and remember that indifference kills along with hatred. That is why it is so important that everyone who values life should show determination. Eternal memory to all victims of the Holocaust!

Today Ukraine honors the memory of millions of victims of the Holocaust. We know and remember that indifference kills along with hatred. That is why it is so important that everyone who values life should show determination. Eternal memory to all victims of the Holocaust!

Volodymyr Zelenskyy / Володимир Зеленський

1,925,329 views • 3 years ago

Today, together with Prime Minister of Portugal Luís Montenegro, we honored the memory of our fallen warriors. Eternal memory to all our defenders, to every Ukrainian who gave their life so that Ukraine could live. 🇺🇦🇺🇦🇺🇦

Today, together with Prime Minister of Portugal Luís Montenegro, we honored the memory of our fallen warriors. Eternal memory to all our defenders, to every Ukrainian who gave their life so that Ukraine could live. 🇺🇦🇺🇦🇺🇦

Volodymyr Zelenskyy / Володимир Зеленський

115,863 views • 5 months ago

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

Sean Hannity 🇺🇸

67,586 views • 2 days ago

Today, we remember the Screaming Eagles that made the ultimate sacrifice. We remember their stories and resolve to live in a way to honor their memory. Be safe this Memorial Day. #airassault

Today, we remember the Screaming Eagles that made the ultimate sacrifice. We remember their stories and resolve to live in a way to honor their memory. Be safe this Memorial Day. #airassault

101st Airborne Div.

11,162 views • 1 year ago

I move that we adjourn in the Honor and memory of Charlie Kirk…

I move that we adjourn in the Honor and memory of Charlie Kirk…

Nick Freitas

39,742 views • 7 months ago

New course: Agent Memory: Building Memory-Aware Agents, built in partnership with Oracle and taught by Richmond Alake and Nacho Martínez. Many agents work well within a single session but their memory resets once the session ends. Consider a research agent working on dozens of papers across multiple days: without memory, it has no way to store and retrieve what it learned across sessions. This short course teaches you to build a memory system that enables agents to persist memory and thereby learn across sessions. You'll design a Memory Manager that handles different memory types, implement semantic tool retrieval that scales without bloating the context, and build write-back pipelines that let your agent autonomously update and refine what it knows over time. Skills you'll gain: - Build persistent memory stores for different agent memory types - Implement a Memory Manager that orchestrates how your agent reads, writes, and retrieves memory - Treat tools as procedural memory and retrieve only relevant ones at inference time using semantic search Join and learn to build agents that remember and improve over time!

New course: Agent Memory: Building Memory-Aware Agents, built in partnership with Oracle and taught by Richmond Alake and Nacho Martínez. Many agents work well within a single session but their memory resets once the session ends. Consider a research agent working on dozens of papers across multiple days: without memory, it has no way to store and retrieve what it learned across sessions. This short course teaches you to build a memory system that enables agents to persist memory and thereby learn across sessions. You'll design a Memory Manager that handles different memory types, implement semantic tool retrieval that scales without bloating the context, and build write-back pipelines that let your agent autonomously update and refine what it knows over time. Skills you'll gain: - Build persistent memory stores for different agent memory types - Implement a Memory Manager that orchestrates how your agent reads, writes, and retrieves memory - Treat tools as procedural memory and retrieve only relevant ones at inference time using semantic search Join and learn to build agents that remember and improve over time!

Andrew Ng

157,523 views • 2 months ago

As we honor their memory today, let us pledge that their lives, their sacrifices, their valor, shall be justified and remembered. Support the Honored SEAL Legacy Scholarship now:

As we honor their memory today, let us pledge that their lives, their sacrifices, their valor, shall be justified and remembered. Support the Honored SEAL Legacy Scholarship now:

SEAL Legacy

27,363 views • 3 years ago

Jensen Huang: “We need memory for… HBM memory for performance. We need LPDDR for low-power memory.” Reporter: Is memory constraint an issue for NVIDIA this year? Jensen Huang: “The entire supply chain is challenging this year because demand is so much more. Supply has been growing 100% every year, but demand is going faster than that. We are going to have a very good year.” We are entering a Memory Supercycle $MU

Jensen Huang: “We need memory for… HBM memory for performance. We need LPDDR for low-power memory.” Reporter: Is memory constraint an issue for NVIDIA this year? Jensen Huang: “The entire supply chain is challenging this year because demand is so much more. Supply has been growing 100% every year, but demand is going faster than that. We are going to have a very good year.” We are entering a Memory Supercycle $MU

NIK

217,174 views • 4 months ago

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Sarvam AI

16,310 views • 6 months ago

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

Jon Saad-Falcon

146,030 views • 12 days ago

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,487 views • 11 months ago

Special words from Owen and Ed today in memory of Maddy Cusack. “We hope they find the peace and the justice that they are still striving for” #MC8 ❤️

Special words from Owen and Ed today in memory of Maddy Cusack. “We hope they find the peace and the justice that they are still striving for” #MC8 ❤️

Chris Lawn

48,761 views • 1 year ago

NVIDIA $NVDA CEO JENSEN HUANG JUST SAID: "We need many many TRILLIONS OF DOLLARS OF CAPACITY...We work with EVERY MEMORY VENDOR, BECAUSE WE USE SO MUCH MEMORY. We are going to work with every connector vendor, EVERY SINGLE SILICON PHOTONICS COMPANY." $AAOI $COHR $LITE $CIEN $FN $MU $SNDK $EWY

NVIDIA $NVDA CEO JENSEN HUANG JUST SAID: "We need many many TRILLIONS OF DOLLARS OF CAPACITY...We work with EVERY MEMORY VENDOR, BECAUSE WE USE SO MUCH MEMORY. We are going to work with every connector vendor, EVERY SINGLE SILICON PHOTONICS COMPANY." $AAOI $COHR $LITE $CIEN $FN $MU $SNDK $EWY

KawzInvests 🦑

50,630 views • 2 months ago

We are the 60,000 that run and the hundreds of thousands that support. We are the red arrows that roar and the river that flows. We will stand as one for the biggest run of all. We are the Great North Run. #WeAreTheGNR

We are the 60,000 that run and the hundreds of thousands that support. We are the red arrows that roar and the river that flows. We will stand as one for the biggest run of all. We are the Great North Run. #WeAreTheGNR

Great Run

14,210 views • 10 months ago