Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

.Reiner Pope works out from first principles how much frontier models are overtrained relative to Chinchilla optimal. One of the cleverest deductions from the lecture, and the one I enjoyed the most.

Dwarkesh Patel

240,375 subscribers

54,976 views • 2 months ago •via X (Twitter)

Education News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography

Dwarkesh Patel

1,285,605 views • 2 months ago

Sagats optimal 3 bar crMP dr conversion depending on where you are on the screen (The last one isn’t max damage but the optimal is much harder and not worth it most of the time)

Sagats optimal 3 bar crMP dr conversion depending on where you are on the screen (The last one isn’t max damage but the optimal is much harder and not worth it most of the time)

Chico

20,885 views • 10 months ago

Port one of my first custom models to the game. Works surprisingly well

Port one of my first custom models to the game. Works surprisingly well

Macu🇻🇪

68,250 views • 1 year ago

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…

The Punisher

1,385,593 views • 2 years ago

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”

Breaking911

959,830 views • 11 months ago

One of the most moving moments from Pope Francis’ funeral: Patriarchs, major archbishops, and metropolitans from the Eastern Catholics Churches approach Pope Francis’ coffin as a choir chants a Greek prayer from the Byzantine Funeral Office.

One of the most moving moments from Pope Francis’ funeral: Patriarchs, major archbishops, and metropolitans from the Eastern Catholics Churches approach Pope Francis’ coffin as a choir chants a Greek prayer from the Byzantine Funeral Office.

Courtney Mares

294,693 views • 1 year ago

Robert Prevost, the choice of the world's Catholic cardinals to serve as leader of the 1.4-billion-member Church, is the first pope from the United States and a relative unknown on the global stage

Robert Prevost, the choice of the world's Catholic cardinals to serve as leader of the 1.4-billion-member Church, is the first pope from the United States and a relative unknown on the global stage

Reuters

52,316 views • 1 year ago

Could the next Pope be the first American-born one? Steve Bannon: "I do think one of the dark horses - and unfortunately one of the most progressive - is Cardinal Prevost... he's certainly on the shortlist." 📺 Piers Morgan

Could the next Pope be the first American-born one? Steve Bannon: "I do think one of the dark horses - and unfortunately one of the most progressive - is Cardinal Prevost... he's certainly on the shortlist." 📺 Piers Morgan

Piers Morgan Uncensored

191,518 views • 1 year ago

The Hound was one character whom I enjoyed watching from the beginning to the end

The Hound was one character whom I enjoyed watching from the beginning to the end

Unique Movie Moments 🐬

74,923 views • 7 months ago

One of the most pleasantly surprising result from the TikTok ban and the migration to xiaohongshu, is how many young Americans are fluent in Mandarin Chinese.

One of the most pleasantly surprising result from the TikTok ban and the migration to xiaohongshu, is how many young Americans are fluent in Mandarin Chinese.

Zhao DaShuai 东北进修🇨🇳

204,831 views • 1 year ago

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻

Kerala Box Office

29,333 views • 1 month ago

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history

Reuters

73,184 views • 4 months ago

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.

Vatican News

35,618 views • 2 months ago

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby

Chaya’s Clan

120,532 views • 1 year ago

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,110 views • 1 month ago

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.

Dwarkesh Patel

27,876 views • 1 month ago

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.

Dr. Phil

32,680 views • 5 months ago

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼

Trey Hannam

58,140 views • 1 month ago

Congolese Catholics in Rome have spoken of their hopes that the next pope will be from their country. Cardinal Ambongo from DRC is one of the key candidates for the papacy and one of 17 Africans taking part in the conclave to elect the successor to Pope Francis at the Vatican.

Congolese Catholics in Rome have spoken of their hopes that the next pope will be from their country. Cardinal Ambongo from DRC is one of the key candidates for the papacy and one of 17 Africans taking part in the conclave to elect the successor to Pope Francis at the Vatican.

BBC News Africa

297,029 views • 1 year ago