Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.

driaforall's profile picture

Dria

14,507 subscribers

26,040 Aufrufe • vor 6 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”
1:31

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,827 Aufrufe • vor 27 Tagen

🇵🇸 🇩🇪 “We are well aware that today a two State solution appears more distant than ever. We are well aware that steps are being taken as we speak to bring it out of reach. We are meeting today because we are convinced there can be a better future.”
0:28

🇵🇸 🇩🇪 “We are well aware that today a two State solution appears more distant than ever. We are well aware that steps are being taken as we speak to bring it out of reach. We are meeting today because we are convinced there can be a better future.”

State of Palestine

75,902 Aufrufe • vor 8 Monaten

‘Please look, study’ - Putin presented with a catalogue of all destroyed and desecrated Soviet/Russian monuments in Ukraine and Europe ‘This is being done to erase the historical memory of who we are, where we came from. It is being done so that we have no future. It is also being done to erase the memory of those who look towards Russia’
1:29

‘Please look, study’ - Putin presented with a catalogue of all destroyed and desecrated Soviet/Russian monuments in Ukraine and Europe ‘This is being done to erase the historical memory of who we are, where we came from. It is being done so that we have no future. It is also being done to erase the memory of those who look towards Russia’

Zlatti71

14,766 Aufrufe • vor 1 Jahr

Geometric Context Transformer for Streaming 3D Reconstruction Contributions: • We introduce LingBot-Map, a streaming 3D foundation model built around Geometric Context Attention (GCA), which maintains three complementary context types – anchor, pose-reference window, and trajectory memory – for efficient and consistent long-sequence streaming inference. • We propose an efficient training recipe based on progressive training and context parallelism with a relative loss formulation for stable long-sequence optimization. • We demonstrate that LingBot-Map achieves state-of-the-art performance on multiple benchmarks (Oxford Spires, Tanks and Temples, ETH3D, and 7-Scenes), significantly outperforming existing streaming approaches in reconstruction quality and inference speed.
1:10

Geometric Context Transformer for Streaming 3D Reconstruction Contributions: • We introduce LingBot-Map, a streaming 3D foundation model built around Geometric Context Attention (GCA), which maintains three complementary context types – anchor, pose-reference window, and trajectory memory – for efficient and consistent long-sequence streaming inference. • We propose an efficient training recipe based on progressive training and context parallelism with a relative loss formulation for stable long-sequence optimization. • We demonstrate that LingBot-Map achieves state-of-the-art performance on multiple benchmarks (Oxford Spires, Tanks and Temples, ETH3D, and 7-Scenes), significantly outperforming existing streaming approaches in reconstruction quality and inference speed.

MrNeRF

24,549 Aufrufe • vor 2 Monaten

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."
0:50

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

Deirdre Bosa

83,925 Aufrufe • vor 1 Monat

Today Ukraine honors the memory of millions of victims of the Holocaust. We know and remember that indifference kills along with hatred. That is why it is so important that everyone who values ​​life should show determination. Eternal memory to all victims of the Holocaust!
1:00

Today Ukraine honors the memory of millions of victims of the Holocaust. We know and remember that indifference kills along with hatred. That is why it is so important that everyone who values ​​life should show determination. Eternal memory to all victims of the Holocaust!

Volodymyr Zelenskyy / Володимир Зеленський

1,925,337 Aufrufe • vor 3 Jahren

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.
1:01

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

Sean Hannity 🇺🇸

67,586 Aufrufe • vor 8 Tagen

Today, together with Prime Minister of Portugal Luís Montenegro, we honored the memory of our fallen warriors. Eternal memory to all our defenders, to every Ukrainian who gave their life so that Ukraine could live. 🇺🇦🇺🇦🇺🇦
0:27

Today, together with Prime Minister of Portugal Luís Montenegro, we honored the memory of our fallen warriors. Eternal memory to all our defenders, to every Ukrainian who gave their life so that Ukraine could live. 🇺🇦🇺🇦🇺🇦

Volodymyr Zelenskyy / Володимир Зеленський

115,863 Aufrufe • vor 5 Monaten

Today, we remember the Screaming Eagles that made the ultimate sacrifice. We remember their stories and resolve to live in a way to honor their memory. Be safe this Memorial Day. #airassault
1:03

Today, we remember the Screaming Eagles that made the ultimate sacrifice. We remember their stories and resolve to live in a way to honor their memory. Be safe this Memorial Day. #airassault

101st Airborne Div.

11,162 Aufrufe • vor 1 Jahr

I move that we adjourn in the Honor and memory of Charlie Kirk…
3:08

I move that we adjourn in the Honor and memory of Charlie Kirk…

Nick Freitas

40,434 Aufrufe • vor 7 Monaten

New course: Agent Memory: Building Memory-Aware Agents, built in partnership with Oracle and taught by Richmond Alake and Nacho Martínez. Many agents work well within a single session but their memory resets once the session ends. Consider a research agent working on dozens of papers across multiple days: without memory, it has no way to store and retrieve what it learned across sessions. This short course teaches you to build a memory system that enables agents to persist memory and thereby learn across sessions. You'll design a Memory Manager that handles different memory types, implement semantic tool retrieval that scales without bloating the context, and build write-back pipelines that let your agent autonomously update and refine what it knows over time. Skills you'll gain: - Build persistent memory stores for different agent memory types - Implement a Memory Manager that orchestrates how your agent reads, writes, and retrieves memory - Treat tools as procedural memory and retrieve only relevant ones at inference time using semantic search Join and learn to build agents that remember and improve over time!
1:37

New course: Agent Memory: Building Memory-Aware Agents, built in partnership with Oracle and taught by Richmond Alake and Nacho Martínez. Many agents work well within a single session but their memory resets once the session ends. Consider a research agent working on dozens of papers across multiple days: without memory, it has no way to store and retrieve what it learned across sessions. This short course teaches you to build a memory system that enables agents to persist memory and thereby learn across sessions. You'll design a Memory Manager that handles different memory types, implement semantic tool retrieval that scales without bloating the context, and build write-back pipelines that let your agent autonomously update and refine what it knows over time. Skills you'll gain: - Build persistent memory stores for different agent memory types - Implement a Memory Manager that orchestrates how your agent reads, writes, and retrieves memory - Treat tools as procedural memory and retrieve only relevant ones at inference time using semantic search Join and learn to build agents that remember and improve over time!

Andrew Ng

158,028 Aufrufe • vor 2 Monaten

As we honor their memory today, let us pledge that their lives, their sacrifices, their valor, shall be justified and remembered. Support the Honored SEAL Legacy Scholarship now:
3:30

As we honor their memory today, let us pledge that their lives, their sacrifices, their valor, shall be justified and remembered. Support the Honored SEAL Legacy Scholarship now:

SEAL Legacy

27,363 Aufrufe • vor 3 Jahren

Jensen Huang: “We need memory for… HBM memory for performance. We need LPDDR for low-power memory.” Reporter: Is memory constraint an issue for NVIDIA this year? Jensen Huang: “The entire supply chain is challenging this year because demand is so much more. Supply has been growing 100% every year, but demand is going faster than that. We are going to have a very good year.” We are entering a Memory Supercycle $MU
0:50

Jensen Huang: “We need memory for… HBM memory for performance. We need LPDDR for low-power memory.” Reporter: Is memory constraint an issue for NVIDIA this year? Jensen Huang: “The entire supply chain is challenging this year because demand is so much more. Supply has been growing 100% every year, but demand is going faster than that. We are going to have a very good year.” We are entering a Memory Supercycle $MU

NIK

217,217 Aufrufe • vor 4 Monaten

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,487 Aufrufe • vor 1 Jahr

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.
1:13

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

Jon Saad-Falcon

148,925 Aufrufe • vor 18 Tagen

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.
1:30

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Sarvam AI

16,310 Aufrufe • vor 6 Monaten

Special words from Owen and Ed today in memory of Maddy Cusack. “We hope they find the peace and the justice that they are still striving for” #MC8 ❤️
1:21

Special words from Owen and Ed today in memory of Maddy Cusack. “We hope they find the peace and the justice that they are still striving for” #MC8 ❤️

Chris Lawn

48,767 Aufrufe • vor 1 Jahr

NVIDIA $NVDA CEO JENSEN HUANG JUST SAID: "We need many many TRILLIONS OF DOLLARS OF CAPACITY...We work with EVERY MEMORY VENDOR, BECAUSE WE USE SO MUCH MEMORY. We are going to work with every connector vendor, EVERY SINGLE SILICON PHOTONICS COMPANY." $AAOI $COHR $LITE $CIEN $FN $MU $SNDK $EWY
0:31

NVIDIA $NVDA CEO JENSEN HUANG JUST SAID: "We need many many TRILLIONS OF DOLLARS OF CAPACITY...We work with EVERY MEMORY VENDOR, BECAUSE WE USE SO MUCH MEMORY. We are going to work with every connector vendor, EVERY SINGLE SILICON PHOTONICS COMPANY." $AAOI $COHR $LITE $CIEN $FN $MU $SNDK $EWY

KawzInvests 🦑

50,630 Aufrufe • vor 3 Monaten

Introducing: Walrus Memory. 🦭 A portable memory layer that lets your AI agents carry context across every app you run them in. No more starting from zero. No more being locked into one platform.…
0:30

Introducing: Walrus Memory. 🦭 A portable memory layer that lets your AI agents carry context across every app you run them in. No more starting from zero. No more being locked into one platform.…

Walrus 🦭/acc

17,582,748 Aufrufe • vor 12 Tagen