Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Groq founder Jonathan Ross explains why their architecture is better for inference than GPUs. Inference doesn't require as much compute power as training. What matters in inference is speed, cost, and energy consumption. $NVDA was going to lose market share as AI workloads shift to inference. They saw this... show more

Oguz Erkan

120,107 subscribers

97,186 views • 5 months ago •via X (Twitter)

News & Politics Science & Technology Finance

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

So now everyone is talking about how AMD is going to have edges over NVDA in inference? Listen to Jense explain inference problem and how Nvidia is also dominating it.

So now everyone is talking about how AMD is going to have edges over NVDA in inference? Listen to Jense explain inference problem and how Nvidia is also dominating it.

The AI Investor

27,271 views • 10 months ago

This is why Nvidia bought Groq. If Groq deploys a ton of low-cost inference chips, demand for training will increase massively. The more inference you have, the more training you need, and vice versa. Nvidia can keep its high margins selling chips for training forever.

This is why Nvidia bought Groq. If Groq deploys a ton of low-cost inference chips, demand for training will increase massively. The more inference you have, the more training you need, and vice versa. Nvidia can keep its high margins selling chips for training forever.

Lin

92,571 views • 5 months ago

$META is betting on $AMD for AI inference. As AI models become more commoditized, total cost of ownership for token generation matters more than ever. $AMD GPUs are far more cost-efficient than $NVDA for inference. With 3.5B daily users ready to use its AI models, $META is going all-in on $AMD

$META is betting on $AMD for AI inference. As AI models become more commoditized, total cost of ownership for token generation matters more than ever. $AMD GPUs are far more cost-efficient than $NVDA for inference. With 3.5B daily users ready to use its AI models, $META is going all-in on $AMD

Daniel

20,558 views • 4 months ago

No wonder Jensen got so defensive over $NVDA GPUs against custom chips. $GOOG will start using different custom chips for inference and training. $AMZN has already adopted this approach as they have Trainium for training and Inferentia for inference. Hyperscalers obviously see better price/performance in this approach, and this is where the industry is headed. $AVGO and $MRVL are poised to be big winners, while $NVDA dominance will be under increasing pressure.

No wonder Jensen got so defensive over $NVDA GPUs against custom chips. $GOOG will start using different custom chips for inference and training. $AMZN has already adopted this approach as they have Trainium for training and Inferentia for inference. Hyperscalers obviously see better price/performance in this approach, and this is where the industry is headed. $AVGO and $MRVL are poised to be big winners, while $NVDA dominance will be under increasing pressure.

Oguz Erkan

223,244 views • 1 month ago

The market won’t be able to keep $NVDA below $200 much longer. Jensen is clearly building for the inference era with Rubin handling attention-heavy workloads and Groq powering low-latency token generation. The AI engine just keeps humming.

The market won’t be able to keep $NVDA below $200 much longer. Jensen is clearly building for the inference era with Rubin handling attention-heavy workloads and Groq powering low-latency token generation. The AI engine just keeps humming.

Shay Boloor

142,460 views • 3 months ago

Jensen basically confirmed my takes earlier about the Groq deal.. if it was a much better inference stack than NVDA GPU then they wouldn't have sold it for $20bn, also not simply about the SRAM architecture that NVDA can replicate in no time.

Jensen basically confirmed my takes earlier about the Groq deal.. if it was a much better inference stack than NVDA GPU then they wouldn't have sold it for $20bn, also not simply about the SRAM architecture that NVDA can replicate in no time.

The AI Investor

68,892 views • 5 months ago

$AMZN CEO: “If you want to allow customers to use AI as expansively as they want, you need to bring down the cost of inference.” Hyperscalers are preparing for the age of inference. $AMZN has Trainium, $GOOG has TPU and $MSFT recently announced Maia 200. Once the bulk of AI workloads shifts from training to inference, they’ll depend less on $NVDA and margins will expand thanks to vertical integration. $AMZN is leading the race here as their chip business has already reached $10 billion annual run rate. Imagine being bearish on $AMZN.

$AMZN CEO: “If you want to allow customers to use AI as expansively as they want, you need to bring down the cost of inference.” Hyperscalers are preparing for the age of inference. $AMZN has Trainium, $GOOG has TPU and $MSFT recently announced Maia 200. Once the bulk of AI workloads shifts from training to inference, they’ll depend less on $NVDA and margins will expand thanks to vertical integration. $AMZN is leading the race here as their chip business has already reached $10 billion annual run rate. Imagine being bearish on $AMZN.

Oguz Erkan

130,592 views • 4 months ago

$NVDA CEO Jensen Huang said “buy $QCOM stock” while discussing edge AI and the need for AI chips inside smartphones. Qualcomm is positioned to benefit as more AI inference moves to devices.

$NVDA CEO Jensen Huang said “buy $QCOM stock” while discussing edge AI and the need for AI chips inside smartphones. Qualcomm is positioned to benefit as more AI inference moves to devices.

Shay Boloor

183,206 views • 12 days ago

$NVDA CEO Jensen Huang praised $NBIS during his keynote. Nebius is positioning itself as a cloud utility for the agentic AI era built to deliver inference and AI compute efficiently at scale.

$NVDA CEO Jensen Huang praised $NBIS during his keynote. Nebius is positioning itself as a cloud utility for the agentic AI era built to deliver inference and AI compute efficiently at scale.

Shay Boloor

345,774 views • 19 days ago

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

What is AI inference engineering, why is it such an in-demand skill, and how do you break into the field? With author of Inference Engineering Philip Kiely and head of training at Baseten Charlie O'Neill 0:00: What is inference? 2:47: History of inference 4:59: Downstream effects of AI research on inference 13:54: What you'll learn from Inference Engineering 16:14: Advice for engineers transitioning into AI 19:00: Open source models driving inference growth 20:55: Specialization vs. frontier closed models 23:51: "Big Token" and the importance of open source AI 27:18: Where to get Inference Engineering

Madison Kanna

119,268 views • 2 months ago

Sygaldry (Sygaldry) is addressing the rising compute costs and energy requirements for AI by building quantum-accelerated AI servers to exponentially speed up training and inference. Congrats to @chadrigetti and @idaliafriedson on the launch!

Sygaldry (Sygaldry) is addressing the rising compute costs and energy requirements for AI by building quantum-accelerated AI servers to exponentially speed up training and inference. Congrats to @chadrigetti and @idaliafriedson on the launch!

Y Combinator

38,044 views • 1 year ago

It’s official: McLaren F1 x Groq Bringing inference speed at a winning cost to the grid and beyond. See you in Singapore. 🧡🏁

It’s official: McLaren F1 x Groq Bringing inference speed at a winning cost to the grid and beyond. See you in Singapore. 🧡🏁

Groq Inc

219,363 views • 8 months ago

1/ $AMD is a 10x opportunity everybody ignores. Lisa Su believes that inference will be a bigger market than training. Training is the cost center of AI workload while inference is where you make money. $AMD has an advantage here. Let me explain why: 🧵

1/ $AMD is a 10x opportunity everybody ignores. Lisa Su believes that inference will be a bigger market than training. Training is the cost center of AI workload while inference is where you make money. $AMD has an advantage here. Let me explain why: 🧵

Oguz Erkan

413,014 views • 1 year ago

Peter Thiel says the quiet part loud: AI chips will get commoditized. This is why I avoid $NVDA. It made an immense amount of money marking up its GPUs by 10x and others had to pay as alternatives weren’t good enough. Now they are getting good enough. $AMD has already caught up with $NVDA in hardware performance, and ASICS are proving unexpectedly competitive due to their efficiency. $GOOG trained Gemini 3 solely on TPUs, and Anthropic is running a significant part of its training and inference workloads on Trainium clusters. As alternatives become even stronger, $NVDA margins and volumes will erode, and profits will shift from hardware to the application layer.

Peter Thiel says the quiet part loud: AI chips will get commoditized. This is why I avoid $NVDA. It made an immense amount of money marking up its GPUs by 10x and others had to pay as alternatives weren’t good enough. Now they are getting good enough. $AMD has already caught up with $NVDA in hardware performance, and ASICS are proving unexpectedly competitive due to their efficiency. $GOOG trained Gemini 3 solely on TPUs, and Anthropic is running a significant part of its training and inference workloads on Trainium clusters. As alternatives become even stronger, $NVDA margins and volumes will erode, and profits will shift from hardware to the application layer.

Oguz Erkan

451,686 views • 6 months ago

Our approach to AI infra is simple: build the most fungible and flexible fleet to meet the real world's needs across inference and training as Scott Guthrie shared with Alex Kantrowitz. And we are already doing it at scale today, as we power the biggest AI workloads like Copilot and ChatGPT, APIs that power 3P products & enterprise workloads and high scale training.

Our approach to AI infra is simple: build the most fungible and flexible fleet to meet the real world's needs across inference and training as Scott Guthrie shared with Alex Kantrowitz. And we are already doing it at scale today, as we power the biggest AI workloads like Copilot and ChatGPT, APIs that power 3P products & enterprise workloads and high scale training.

Satya Nadella

114,478 views • 8 months ago

IN NEWS: Baseten raises a $150M series D round. Tuhin Srivastava (Founder & CEO, Baseten) on the future of inference: “I think the token price goes down and inference should get cheaper over time. And that really just means there is going to be more inference.” “Every time we lower prices or optimize models to make it cheaper, four months later customers are spending more anyway.” “Inference prices will go down, but if the world is run by AI in 10 years, there is going to be a lot of inference. It better be cheap.”

IN NEWS: Baseten raises a $150M series D round. Tuhin Srivastava (Founder & CEO, Baseten) on the future of inference: “I think the token price goes down and inference should get cheaper over time. And that really just means there is going to be more inference.” “Every time we lower prices or optimize models to make it cheaper, four months later customers are spending more anyway.” “Inference prices will go down, but if the world is run by AI in 10 years, there is going to be a lot of inference. It better be cheap.”

TBPN

17,057 views • 9 months ago

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

OpenAI's Noam Brown says that while AI model performance scales roughly equivalently with more training or inference compute, the cost of inference is on the order of 100 billion times cheaper

Tsarathustra

250,828 views • 1 year ago

Lisa Su explains why GPUs from $AMD & $NVDA will continue to dominate AI accelerators for years since developers need programmability & flexibility that ASICs like $GOOGL TPUs can’t match. Even as ASICs improve for inference, GPUs remain the default platform.

Lisa Su explains why GPUs from $AMD & $NVDA will continue to dominate AI accelerators for years since developers need programmability & flexibility that ASICs like $GOOGL TPUs can’t match. Even as ASICs improve for inference, GPUs remain the default platform.

Shay Boloor

78,755 views • 5 months ago