Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

.Reiner Pope works out from first principles how much frontier models are overtrained relative to Chinchilla optimal. One of the cleverest deductions from the lecture, and the one I enjoyed the most.

dwarkesh_sp's profile picture

Dwarkesh Patel

243,524 subscribers

55,007 Aufrufe • vor 3 Monaten •via X (Twitter)

BildungNachrichten & PolitikWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography
2:13:40

Did a very different format with Reiner Pope – a blackboard lecture where he walks through how frontier LLMs are trained and served. It's shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there - it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube so you can see the chalkboard. 0:00:00 – How batch size affects token cost and speed 0:31:59 – How MoE models are laid out across GPU racks 0:47:02 – How pipeline parallelism spreads model layers across racks 1:03:27 – Why Ilya said, “As we now know, pipelining is not wise.” 1:18:49 – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal 1:32:52 – Deducing long context memory costs from API pricing 2:03:52 – Convergent evolution between neural nets and cryptography

Dwarkesh Patel

1,290,422 Aufrufe • vor 3 Monaten

Common Wealth: A unified theory of society in the age of AI Four works on personhood, economics, law, and the state. One continuous argument for how to build a better society from first principles.
1:14

Common Wealth: A unified theory of society in the age of AI Four works on personhood, economics, law, and the state. One continuous argument for how to build a better society from first principles.

Intelligent Internet

28,198 Aufrufe • vor 14 Tagen

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…
11:05

Pope Francis Declares ‘Jesus is Satan’, Vows To Usher in ‘One World Religion’ Pope Francis is one of the most hated men in the world today. Those who hate him most are not atheists, or protestants, or Muslims, but Christians and his own followers. Notice the Pope is reading from a script? Who wrote the script…

The Punisher

1,385,646 Aufrufe • vor 2 Jahren

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”
0:52

CNN TO BERNIE: “You are one of the most widely admired Americans. I think you're only second to the Pope!”

Breaking911

959,869 Aufrufe • vor 1 Jahr

The Hound was one character whom I enjoyed watching from the beginning to the end
0:57

The Hound was one character whom I enjoyed watching from the beginning to the end

Unique Movie Moments 🐬

74,940 Aufrufe • vor 9 Monaten

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history
0:41

Nick Reiner, the troubled son of slain Hollywood filmmaker Rob Reiner, pleaded not guilty to murder charges stemming from the fatal stabbing of his parents in their home, one of the most shocking celebrity homicide cases in Los Angeles history

Reuters

73,189 Aufrufe • vor 5 Monaten

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻
0:30

From one of the most hated song to one of the favorite track from #Karuppu 🔥🤌🏻

Kerala Box Office

29,333 Aufrufe • vor 3 Monaten

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.
1:53

One year has passed since the white smoke of 8 May 2025 announced to the world the election of Pope Francis’ successor. Our video retraces the most significant moments of the American Pontiff’s first year, from audiences to apostolic journeys to Türkiye, Lebanon, and Africa.

Vatican News

35,813 Aufrufe • vor 3 Monaten

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.
4:07

On a chip, even something as basic as selecting one input from a register file has to be built out of AND and OR gates wired together. Simple-seeming operations end up surprisingly expensive. Reiner Pope showed me how this stacks up. In older generations of chips, most of the die area was used for shuffling data around these circuits, not for the actual compute! This is one of the problems modern chip design tries to solve.

Dwarkesh Patel

27,876 Aufrufe • vor 2 Monaten

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby
9:05

One of the most incredible, eloquent, intelligent, powerful and inspiring videos I’ve ever enjoyed from near the beginning of the war. This is the incredible Noa Tishby

Chaya’s Clan

120,532 Aufrufe • vor 2 Jahren

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.
17:21

Nick Reiner faces charges of parricide — the killing of a parent by their child. It is one of the rarest crimes, and one of the most disturbing because it violates one of the most basic human bonds. Parents are supposed to protect their children. Children are not supposed to fear — or kill — their parents. This video examines the Nick Reiner murder trial, the meaning of parricide, and the warning signs and psychological factors experts say often precede these rare cases.

Dr. Phil

32,680 Aufrufe • vor 7 Monaten

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."
2:08

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,110 Aufrufe • vor 3 Monaten

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼
4:47

Let’s break down one of the best hitters in the country and steal what works. Brenden Lawson had one of the most impressive bats I saw all year. #2 in OPS in the whole SEC. What stood out the most? And what can your hitters actually take from it? Drop it below. 📝👇🏼

Trey Hannam

59,215 Aufrufe • vor 2 Monaten