Loading video...

Video Failed to Load

Go Home

Hot take: Tokens per second is a completely meaningless metric

AdamHoltererer's profile picture

Adam Holter

3,438 subscribers

302,013 views • 7 days ago •via X (Twitter)

Health & WellnessScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

"50 tokens per second is too slow" The slow 👇
0:26

"50 tokens per second is too slow" The slow 👇

Javier • priv/acc

11,219 views • 1 month ago

This is from my conversation with a Former $NVDA Director. If you want more TOKENS per SECOND because you are limited by POWER, you buy the next $NVDA generation. - Limited to 100 MW: $NVDA H100 = 300M tokens/second - Limited to 100 MW: $NVDA GB200 NVL72 = 12B tokens/second
2:22

This is from my conversation with a Former $NVDA Director. If you want more TOKENS per SECOND because you are limited by POWER, you buy the next $NVDA generation. - Limited to 100 MW: $NVDA H100 = 300M tokens/second - Limited to 100 MW: $NVDA GB200 NVL72 = 12B tokens/second

Rihard Jarc

57,622 views • 1 year ago

Nvidia's Chief Scientist Bill Dally says there's a path to serving relatively large models at 10,000 to 20,000 tokens per user per second. For context, Opus 4.6 is ~43 and Grok 4.2 Beta is ~251 tokens/user/s 🤯
1:30

Nvidia's Chief Scientist Bill Dally says there's a path to serving relatively large models at 10,000 to 20,000 tokens per user per second. For context, Opus 4.6 is ~43 and Grok 4.2 Beta is ~251 tokens/user/s 🤯

Marcelo P. Lima

84,684 views • 6 months ago

My mind is blown. GroqInc is serving LLaMA 3 at over 800 tokens per second! 800. Tokens. Per. Second. This unlocks so many incredible use-cases. It's one thing to see my demo — it's another thing entirely to experience it for yourself. Do yourself a favor and try it asap.
0:21

My mind is blown. GroqInc is serving LLaMA 3 at over 800 tokens per second! 800. Tokens. Per. Second. This unlocks so many incredible use-cases. It's one thing to see my demo — it's another thing entirely to experience it for yourself. Do yourself a favor and try it asap.

Matt Shumer

589,626 views • 2 years ago

“AI COULD SOON THINK 100× TO 1,000× FASTER THAN HUMANS.” Emad Mostaque says AI systems are about to become dramatically faster as specialized AI hardware improves. He says humans currently operate at roughly 20–30 tokens per second, while current AI models are around 40 tokens per second. He points to Taalas' technology, which he says demonstrated an 8-billion-parameter model generating around 14,000 tokens per second. Mostaque says current AI systems are effectively “thinking at the speed of a human”, but that will soon change. He predicts AI systems could reach 1,000 tokens per second, followed by 10,000 tokens per second by the end of next year. His broader point is that AI could eventually “think a thousand times faster” than humans.
1:58

“AI COULD SOON THINK 100× TO 1,000× FASTER THAN HUMANS.” Emad Mostaque says AI systems are about to become dramatically faster as specialized AI hardware improves. He says humans currently operate at roughly 20–30 tokens per second, while current AI models are around 40 tokens per second. He points to Taalas' technology, which he says demonstrated an 8-billion-parameter model generating around 14,000 tokens per second. Mostaque says current AI systems are effectively “thinking at the speed of a human”, but that will soon change. He predicts AI systems could reach 1,000 tokens per second, followed by 10,000 tokens per second by the end of next year. His broader point is that AI could eventually “think a thousand times faster” than humans.

SciTech Era

12,824 views • 1 month ago

15,000 tokens per second feels absolutely insane. Is this the future? This is if you haven't heard of it from Taalas Inc..
0:23

15,000 tokens per second feels absolutely insane. Is this the future? This is if you haven't heard of it from Taalas Inc..

David Gomes

262,584 views • 5 months ago

Everyone’s obsessed with cheaper tokens. Meanwhile, enterprise AI spend keeps going up. Chamath says costs are doubling every 45 days for ~5% productivity gains. Glean’s founder says the metric that actually matters is cost per successful task, not cost per token.
1:09

Everyone’s obsessed with cheaper tokens. Meanwhile, enterprise AI spend keeps going up. Chamath says costs are doubling every 45 days for ~5% productivity gains. Glean’s founder says the metric that actually matters is cost per successful task, not cost per token.

Not Jerome Powell

45,215 views • 2 months ago

Qwen3.5-35B-A3B running locally on an M4 chip at 49.5 tokens per second. A 35B model. On a laptop. In real time. LOCAL AI IS GETTING SCARY FAST.
1:10

Qwen3.5-35B-A3B running locally on an M4 chip at 49.5 tokens per second. A 35B model. On a laptop. In real time. LOCAL AI IS GETTING SCARY FAST.

0xMarioNawfal

478,419 views • 7 months ago

Watch Gemma-4-26B running 18 concurrent sessions on a single NVIDIA AI DGX Spark — delivering a cumulative 300 tokens per second. 🔥
0:53

Watch Gemma-4-26B running 18 concurrent sessions on a single NVIDIA AI DGX Spark — delivering a cumulative 300 tokens per second. 🔥

Mia

23,221 views • 3 months ago

“Energy density per unit raw material” is the right metric, and the winner is nuclear fission!
2:55

“Energy density per unit raw material” is the right metric, and the winner is nuclear fission!

Isaiah Taylor - making nuclear reactors

27,632 views • 1 year ago

Gemma 4 running locally on a Nintendo Switch :) 1.5 tokens per second haha, but it runs! Google Gemma Google AI Developers Google DeepMind
2:57

Gemma 4 running locally on a Nintendo Switch :) 1.5 tokens per second haha, but it runs! Google Gemma Google AI Developers Google DeepMind

Maddie D. Reese

188,407 views • 6 months ago

.Ahmad stopped by MTS with a NVIDIA DGX Station (Dell Technologies Pro Max with GB300). 🔥 Thousands of tokens per second all running locally.
0:58

.Ahmad stopped by MTS with a NVIDIA DGX Station (Dell Technologies Pro Max with GB300). 🔥 Thousands of tokens per second all running locally.

NVIDIA RTX Spark

12,130 views • 25 days ago

This is Ultrafast. Our premium speed tier, Ultrafast offers up to 8x faster token generation (300 tokens per second) in Codex and up to 6x in the API.
0:45

This is Ultrafast. Our premium speed tier, Ultrafast offers up to 8x faster token generation (300 tokens per second) in Codex and up to 6x in the API.

OpenAI

1,358,798 views • 13 days ago

Hayek explains why social justice is a meaningless term: “Justice is an attribute of individual action. I can be just or unjust towards my fellow men.” “But the conception of a social justice—to expect a just result from an impersonal process which nobody can control—is meaningless and completely impossible.”
0:42

Hayek explains why social justice is a meaningless term: “Justice is an attribute of individual action. I can be just or unjust towards my fellow men.” “But the conception of a social justice—to expect a just result from an impersonal process which nobody can control—is meaningless and completely impossible.”

F. A. Hayek Quotes

33,349 views • 5 months ago