Загрузка видео...

Не удалось загрузить видео

На главную

.Reiner Pope works out from first principles how much frontier models are overtrained relative to Chinchilla optimal. One of the cleverest deductions from the lecture, and the one I enjoyed the most.

dwarkesh_sp's profile picture

Dwarkesh Patel

243,524 subscribers

55,007 просмотров • 3 месяцев назад •via X (Twitter)

ОбразованиеНовости и политикаНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography
2:13:40

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography

Dwarkesh Patel

1,291,043 просмотров • 3 месяцев назад

Common Wealth: A unified theory of society in the age of AI Four works on personhood, economics, law, and the state. One continuous argument for how to build a better society from first principles.
1:14

Common Wealth: A unified theory of society in the age of AI Four works on personhood, economics, law, and the state. One continuous argument for how to build a better society from first principles.

Intelligent Internet

28,198 просмотров • 19 дней назад

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…
11:05

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…

The Punisher

1,385,646 просмотров • 2 лет назад

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”
0:52

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”

Breaking911

959,869 просмотров • 1 год назад

The Hound was one character whom I enjoyed watching from the beginning to the end
0:57

The Hound was one character whom I enjoyed watching from the beginning to the end

Unique Movie Moments 🐬

74,940 просмотров • 9 месяцев назад

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history
0:41

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history

Reuters

73,189 просмотров • 6 месяцев назад

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻
0:30

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻

Kerala Box Office

29,333 просмотров • 3 месяцев назад

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.
1:53

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.

Vatican News

35,831 просмотров • 3 месяцев назад

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.
4:07

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.

Dwarkesh Patel

27,876 просмотров • 3 месяцев назад

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby
9:05

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby

Chaya’s Clan

120,532 просмотров • 2 лет назад

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.
17:21

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.

Dr. Phil

32,680 просмотров • 7 месяцев назад

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼
4:47

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼

Trey Hannam

59,215 просмотров • 3 месяцев назад

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."
2:08

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,110 просмотров • 3 месяцев назад