Загрузка видео...

Не удалось загрузить видео

На главную

Today we are shipping dnet, a distributed inference framework that lets Apple Silicon clusters run models that exceed their physical memory. We fuse pipelined-ring parallelism, disk streaming and UMA-aware scheduling so “out of memory” stops being the limit.

driaforall's profile picture

Dria

14,507 subscribers

26,040 просмотров • 8 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”
1:31

We asked James Wang why Cerebras can run large models so quickly. His answer: Inference speed is mostly a memory problem. Instead of constantly pulling weights from external memory, Cerebras splits the model across multiple wafers and pipelines the layers together. “Inference is all about memory bandwidth.” “You don’t want to store weights external to the wafer, because the second it’s outside, it becomes much slower.” “We put the weights on the inside.” “The team built a new software stack that basically lets us split the models by layer and store them layer by layer across multiple wafers.” “That allows us to never have to read from external memory, and that’s what makes it so fast.” “For context, Claude is about 100 tokens per second.”

MTS

27,935 просмотров • 2 месяцев назад

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.
1:01

“That was the first time I realized we were different.” Gad Saad recalls a childhood memory from Beirut that shaped his understanding of the world. Full episode out Tuesday on YouTube, Spotify, and Apple Podcasts.

Sean Hannity 🇺🇸

67,586 просмотров • 1 месяц назад

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.
0:20

"Mac Minis for example are a very good fit" - Andrej Karpathy Andrej Karpathy shouted out my work on EXO Labs in his keynote at Y Combinator AI SUS! Here's the breakdown: Right now most AI workloads run in the cloud where requests from different users are continuously batched together. These workloads are FLOPS-bound and favors hardware with the best unit economics of $ per FLOP, i.e. enterprise GPUs. The personal computing revolution will shift these workloads to personal devices with lower batch sizes (mostly batch_size=1). batch_size=1 inference is memory-bound, because all of the model parameters need to be loaded into the GPU every time a token is generated. Apple Silicon with its Unified Memory architecture has a lot of memory and memory bandwidth per $ compared to other hardware: - M4 Pro Mac Mini, 24GB @ 273GB/s, $58.33/GB, $5.13/GB/s - H100, 80GB @ 3350GB/s, $625/GB, $14.93/GB/s The unit economics of Apple Silicon are becoming more compelling with every release of the Mac. The future of AI inference looks more like open weights models (OpenAI open weights model soonTM) run at low batch_size on personal devices.

Alex Cheema

103,566 просмотров • 1 год назад

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.
2:41

Everyone wrote Apple off as the AI loser, but one hardware spec might flip that story upside down (Save this). @jason called Apple a screaming buy on the back of a single chip detail. The rumored M7 Ultra, expected around 2028, is designed to support up to 1.5TB of unified memory, enough to run frontier class trillion parameter AI models locally, with no cloud required. The Street's bear case on Apple is straightforward. Apple has no frontier model of its own, Siri has stumbled for years and the company effectively rents OpenAI's models for its hardest queries. That narrative treats Apple as the one Magnificent Seven name that missed the AI wave entirely but the bull case flips that framing on its head. If frontier AI models keep shrinking and getting cheaper to run, Apple doesn't need the smartest model in the world, it just needs to own the device that model runs on. And unified memory is the mechanism that makes this possible. Unlike traditional systems where the CPU and GPU each need separate memory, Apple's architecture lets the CPU, GPU and Neural Engine draw from one shared pool. A fully specced M7 Ultra could theoretically run something on the scale of a 1.2 trillion parameter model locally and that capability plugs directly into the one advantage Apple has spent over a decade building: privacy. Apple has already shipped Private Cloud Compute, a system designed so even Apple can't access user data processed off device. Apple doubled down on this at WWDC 2026, framing on device privacy as non-negotiable while rivals default to the cloud. If the best AI models get small enough to run on Apple silicon, the moat stops being the model and becomes the hardware it has to sit on. Milk Road Pro remains bullish on Apple and it remains as one of our core positions, if you want the full thesis + our full AI trades, come join us using the link below for just a $1.

Milk Road AI

36,711 просмотров • 5 дней назад

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.
1:30

Our co-founder Pratyush Kumar highlighted Sarvam's "contrarian bet" of being a full-stack AI platform. That means we build every layer: the models, how they run (inference), how agents work with each other (orchestration), and the AI applications people use. This lets us build AI that actually works for India's unique context and diverse language needs.

Sarvam AI

16,310 просмотров • 7 месяцев назад

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.
1:13

The dominant story in AI has been the growing cloud: bigger clusters, larger models, more gigawatts. We believe the future is in the opposite direction: on-device inference, smaller models, watts instead of gigawatts. Today we're releasing OpenJarvis v1.0: a personal AI assistant that lives, learns, and works on your device.

Jon Saad-Falcon

155,561 просмотров • 1 месяц назад

A place of remembrance and deep reverence. Here are the names of those who gave their lives for our independence and freedom. Today, together with First Lady Olena Zelenska / Олена Зеленська and Prime Minister of Canada Mark Carney, we honored the fallen defenders of Ukraine at the Wall of Remembrance. We cherish the memory of each and every one of them. We honor their heroism and everything they did so that Ukraine may live. Eternal gratitude and glory to the heroes.
0:48

A place of remembrance and deep reverence. Here are the names of those who gave their lives for our independence and freedom. Today, together with First Lady Olena Zelenska / Олена Зеленська and Prime Minister of Canada Mark Carney, we honored the fallen defenders of Ukraine at the Wall of Remembrance. We cherish the memory of each and every one of them. We honor their heroism and everything they did so that Ukraine may live. Eternal gratitude and glory to the heroes.

Volodymyr Zelenskyy / Володимир Зеленський

173,461 просмотров • 11 месяцев назад

The United States of America is a generous country and we take care of our own. It's disgraceful that fraudsters out there are taking advantage of programs like Medicaid. That stops today. Dr. Oz CMS
1:45

The United States of America is a generous country and we take care of our own. It's disgraceful that fraudsters out there are taking advantage of programs like Medicaid. That stops today. Dr. Oz CMS

JD Vance

989,267 просмотров • 4 месяцев назад

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the default scrollback limit from 10MB to 50MB, because on average a full scrollback will still compress smaller than the prior limit. More history, for free. ("Unlimited", disk-paged history is on the roadmap too) Let's talk about cool implementation details, cause this was fun. First, the data structure and memory layout ("PageList") I wrote two years ago finally pays off! One of its traits is that screen memory is backed by a linked list of page-aligned, page-sized (or page-multiple-sized) blocks. Because each block is page-aligned and page-sized, we can use madvise to discard its physical backing while keeping the virtual address space reserved. Compressed pages therefore disappear from resident memory, but decompression is still guaranteed because the address space remains valid and we simply fault new pages back in as needed. We use the same trick for our memory pools, too. Unallocated pool pages don't count as resident memory, saving another couple of MB per terminal. This functionality is also available to libghostty-vt consumers via new `ghostty_terminal_compress` APIs. The consumer decides when the appropriate time to compress is and the APIs advise on compressability.
3:33

Ghostty is getting automatic scrollback compression, resulting in 70 to 90% less physical memory usage. It happens incrementally when idle, so it had no measurable effect on IO throughput. I'm not aware of any other mainstream terminal that does this. Demo video below! The gains let us increase the default scrollback limit from 10MB to 50MB, because on average a full scrollback will still compress smaller than the prior limit. More history, for free. ("Unlimited", disk-paged history is on the roadmap too) Let's talk about cool implementation details, cause this was fun. First, the data structure and memory layout ("PageList") I wrote two years ago finally pays off! One of its traits is that screen memory is backed by a linked list of page-aligned, page-sized (or page-multiple-sized) blocks. Because each block is page-aligned and page-sized, we can use madvise to discard its physical backing while keeping the virtual address space reserved. Compressed pages therefore disappear from resident memory, but decompression is still guaranteed because the address space remains valid and we simply fault new pages back in as needed. We use the same trick for our memory pools, too. Unallocated pool pages don't count as resident memory, saving another couple of MB per terminal. This functionality is also available to libghostty-vt consumers via new `ghostty_terminal_compress` APIs. The consumer decides when the appropriate time to compress is and the APIs advise on compressability.

Mitchell Hashimoto

164,018 просмотров • 15 дней назад

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM
0:29

Linear scaling achieved with multiple DeepSeek v3.1 instances. 4x macs = 4x throughput. 2x M3 Ultra Mac Studios = 1x DeepSeek @ 14 tok/sec 4x M3 Ultra Mac Studios = 2x DeepSeek @ 28 tok/sec DeepSeek V3.1 is a 671B parameter model - so at its native 8-bit quantization, it requires ~700GB of memory to run the model. EXO puts half of the layers on each device, combining their memory. EXO uses MLX distributed with TB5 interconnect, optimized for Apple Silicon. If we need higher throughput, adding two more devices lets us serve more users at once. EXO Labs handles all of this seamlessly - adding more devices to the cluster for linear scaling as we need it. The new EXO 1.0 will be open-source soonTM

Matt Beton

158,485 просмотров • 11 месяцев назад

🇺🇸 SEC RUBIO: 9/11 COULD HAPPEN AGAIN, WE MUST STAY VIGILANT “As we look back, let us hope that without terrorist attacks, we are always vigilant. These kinds of attacks and our enemies are out there, and that it could happen again. but also inspired by the memory of the sense of unity that reigned in our country, a sense of unity that drove us to make the changes necessary so that a day like that would never happen again. And that is how we remember September 11th of 2001." Source: Secretary Marco Rubio
1:16

🇺🇸 SEC RUBIO: 9/11 COULD HAPPEN AGAIN, WE MUST STAY VIGILANT “As we look back, let us hope that without terrorist attacks, we are always vigilant. These kinds of attacks and our enemies are out there, and that it could happen again. but also inspired by the memory of the sense of unity that reigned in our country, a sense of unity that drove us to make the changes necessary so that a day like that would never happen again. And that is how we remember September 11th of 2001." Source: Secretary Marco Rubio

Mario Nawfal

126,292 просмотров • 10 месяцев назад

ISRAELI PM NETANYAHU: “Despite the fake news that is unfortunately being spread since the start of the war 20 days ago, we are winning, and Iran is being decimated.” “Iran's missile and drone arsenal is being massively degraded and will be destroyed. Hundreds of their launchers have been destroyed, their stockpiles of missiles are being hit hard, and so are the industries that produce them.” “In Rising Lion, we destroyed missiles, and we destroyed a lot of the nuclear infrastructure. But what we're destroying now are the factories that produce the components to make these missiles and to make the nuclear weapons that they're trying to produce. We're wiping out their industrial base in a way that we didn't do before.”
0:48

ISRAELI PM NETANYAHU: “Despite the fake news that is unfortunately being spread since the start of the war 20 days ago, we are winning, and Iran is being decimated.” “Iran's missile and drone arsenal is being massively degraded and will be destroyed. Hundreds of their launchers have been destroyed, their stockpiles of missiles are being hit hard, and so are the industries that produce them.” “In Rising Lion, we destroyed missiles, and we destroyed a lot of the nuclear infrastructure. But what we're destroying now are the factories that produce the components to make these missiles and to make the nuclear weapons that they're trying to produce. We're wiping out their industrial base in a way that we didn't do before.”

Fox News

531,407 просмотров • 4 месяцев назад

Today, at the 41st Air India 182 Memorial, we honoured the 329 innocent people murdered, including 268 Canadians. They were mothers and fathers, sons and daughters, neighbours and friends. We honour them by refusing euphemism and calling out the perpetrators and terror groups that still threaten innocent people at home and abroad. Their memory endures. So does Canada.
2:14

Today, at the 41st Air India 182 Memorial, we honoured the 329 innocent people murdered, including 268 Canadians. They were mothers and fathers, sons and daughters, neighbours and friends. We honour them by refusing euphemism and calling out the perpetrators and terror groups that still threaten innocent people at home and abroad. Their memory endures. So does Canada.

Shuv Majumdar

21,461 просмотров • 1 месяц назад

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!
0:39

Micron is going to $4,000 and once you understand what inference actually is, the number stops sounding crazy (Save this). Dylan Patel just said that by 2030, OpenAI and Anthropic alone will need over 100 gigawatts of compute combined and by 2040, we may not even be measuring AI infrastructure in gigawatts anymore. We may be talking about terawatts. Every single one of those gigawatts needs memory to function. Without it, the compute is worthless. Most people heard that and thought about Nvidia but they should be thinking about Micron. Every AI model generating a response has two phases. The first is prefill, processing your prompt which is compute-heavy and the second is decode generating each word one token at a time and that phase is almost entirely memory-bound, not compute-bound. During decode, the GPU's processing units sit idle more than 95% of the time, waiting for data to arrive from memory. Google confirmed it in a research paper that decode-phase bottlenecks are dominated by memory bandwidth and capacity not raw compute. The GPU is not the bottleneck but the memory feeding the GPU is. This matters because inference is now where all the money lives. Training a model happens once, Inference happens billions of times a day every ChatGPT response, every Claude output, every agentic workflow running in the background and every one of those token streams is a billing event tied directly to memory performance. Adding more GPUs does not fix this because GPUs are already underutilized in inference because they are sitting idle waiting on memory. Adding more memory bandwidth and capacity is what directly reduces token cost, reduces latency, and allows the same cluster to serve dramatically more users simultaneously. Longer context windows compound the problem further, a model running a 1 million token context window requires dramatically more memory per session than a 10,000 token window, and every new model generation pushes context longer. The market treats memory as a downstream beneficiary of Nvidia orders. The correct framework is the opposite, Micron is the upstream constraint on how much value every Nvidia GPU can actually generate at inference scale. Micron guided Q4 to $50 billion in revenue, has HBM4 ramping at twice the pace of the prior generation, and CEO Sanjay Mehrotra has said supply will not catch demand before the end of 2027. At 8x forward earnings on $112 projected FY2027 EPS, Micron is the most undervalued infrastructure company in the entire AI stack. Inference is memory. Memory is Micron and the inference ramp has barely started. Milk Road Pro members are already up massively on this position and we're just getting started. If you want the full breakdown of what we're buying and why, come join us for just a dollar using the link below!

Milk Road AI

128,522 просмотров • 24 дней назад

Emilia Clarke describes the six-step memory palace method she used to learn Russian for Ponies "I had this amazing language tutor called Fabian, who was incredible. We did this very complicated, six-step process that involved a memory palace and making stories out of the individual sounds of the words" "We would break them down into individual syllables, and each syllable would have an image that came out of it" "For example, 'pochemu' is a word, so for 'po-che-mu,' I would visualize what those things were. Maybe there's a cow for 'mu' and a dog's paw for 'po.' We would make that into a story, so I learn the story, and then we put the story in a memory palace" "I've learned the story to learn it in the memory palace, so 'po' is over here and 'mu' is over here. In every scene that I'm doing Russian, you're witnessing my brain going into the room, finding the story, and doing the words - which is why I can't speak it"
1:19

Emilia Clarke describes the six-step memory palace method she used to learn Russian for Ponies "I had this amazing language tutor called Fabian, who was incredible. We did this very complicated, six-step process that involved a memory palace and making stories out of the individual sounds of the words" "We would break them down into individual syllables, and each syllable would have an image that came out of it" "For example, 'pochemu' is a word, so for 'po-che-mu,' I would visualize what those things were. Maybe there's a cow for 'mu' and a dog's paw for 'po.' We would make that into a story, so I learn the story, and then we put the story in a memory palace" "I've learned the story to learn it in the memory palace, so 'po' is over here and 'mu' is over here. In every scene that I'm doing Russian, you're witnessing my brain going into the room, finding the story, and doing the words - which is why I can't speak it"

sara

1,741,037 просмотров • 17 дней назад

tested out antirez' ds4.c this morning. so impressive and delivers. on a M3 max, 128GB, stock ds4 settings: - 14–15 t/s at 62K pre-filled actual coding conversation - memory usage was flat during gen ~85GB res - disk cache is ~8GB for a full 100K context window - thermals were normal, light fan activity - inference server is rock solid so far biggest constraint: anytime there's a compact, we pay the wait-time price of a fresh prefill (~1min per 10k context) before we are back in action. sequential inference + multiple agents in parallel performance is unclear, will report back. I'm so amped.
1:14

tested out antirez' ds4.c this morning. so impressive and delivers. on a M3 max, 128GB, stock ds4 settings: - 14–15 t/s at 62K pre-filled actual coding conversation - memory usage was flat during gen ~85GB res - disk cache is ~8GB for a full 100K context window - thermals were normal, light fan activity - inference server is rock solid so far biggest constraint: anytime there's a compact, we pay the wait-time price of a fresh prefill (~1min per 10k context) before we are back in action. sequential inference + multiple agents in parallel performance is unclear, will report back. I'm so amped.

pradeep

162,339 просмотров • 2 месяцев назад

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.
1:09

DeepMind CEO Demis Hassabis says the AI bottleneck is already moving from model ideas to physical supply chains. "How much of the restriction at this point is physical? I am talking memory shortage, GPU shortage, and electricity." Demis: "That is constraining a lot of the deployment for sure. We see so much more demand for our models - Gemini and the other models that we have - than we can possibly serve." Then the less obvious part: "It does constrain a little bit the research, because you need a lot of chips to be able to experiment on new ideas at big enough scale that you can actually see if they are going to work." And on memory: "The whole supply chain is kind of strained... It still in the end actually comes down to a few suppliers of a few of the key components. Anywhere where there is some kind of constraint on the capacity, there is a sort of choke point." Important framing: The next AI race is not just who has the best model architecture. It is who can secure memory, chips, power, suppliers, and enough experimental capacity to find the next breakthrough before everyone else.

Karl Mehta

25,716 просмотров • 27 дней назад

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."
2:32

$NBIS cofounder Roman Chernin describes how their recent acquisitions of Eigen AI and Clarifai were all about speed, incredible talent, and acceleration: "The philosophy is very simple. We need to build so many things, and we need to move so fast, that we're always looking for people who can accelerate us. It should be exceptional talent, and/or something that has a great adoption." "Our two recent acquisitions [were] two teams that work on inference optimization. A big part of our business is how efficiently we convert GPUs into tokens. And these two teams — Eigen AI and Clarifai — one is focused on model optimization, the engine of inference. How you run specific models and all the techniques around spec decoding, quantization, and so on." "And the other is system optimization. All the routing, KV caching, and orchestration across the big cluster of compute and so on." "We have a very strong internal team working on inference. But we felt that we needed to move faster, bring more capabilities. Because the market is so fast."

TBPN

30,486 просмотров • 2 месяцев назад