Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Some teams use sweeps, heuristics, or scaling laws to determine their training LR. At Character, we just have Noam Shazeer dial it to the right value.

Stephen Roller

5,929 subscribers

169,651 views • 2 years ago •via X (Twitter)

Gaming Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

Ying Xiao2 years ago

It turns out that you need to spend a lot of compute and care to beat hyperparameters that Noam just makes up. It's absurd how often that happens.

Naman Goyal2 years ago

lol bet that’s better than most if not all schedules

Yi Tay2 years ago

Haha nice!

Lucas Beyer (bl16)2 years ago

Why on earth would you plot /avg/ of lr though

Stephen Roller2 years ago

it’s not. for all metrics we track avg and denominator. for LR, denominator is always 1. just a side effect of our templating

Zeming Lin2 years ago

Fake news. That's definitely your hand doing it.

cooking2 years ago

ser I needed this so bad

RBPrest2 years ago

what I see is a training of advanced models I think that is but it is good

manoj2 years ago

How do I get this for @dippy_ai

ghoulie2 years ago

can you remove the filter next or implement a switch? I’ll actually send you money to atleast ATTEMPT to try to. or give the community some actual reason as to why we’re being ignored. please do something this is overall just disgusting.

Related Videos

Google's star AI researcher Noam Shazeer just left for OpenAI. "Noam was working on pre-training at Gemini… And pre-training is something that Google is viewed to be really strong at. And OpenAI is viewed to have been kind of weaker at.” It’ll help as “OpenAI tries to catch up to Anthropic’s most advanced models." — Erin Woo

Google's star AI researcher Noam Shazeer just left for OpenAI. "Noam was working on pre-training at Gemini… And pre-training is something that Google is viewed to be really strong at. And OpenAI is viewed to have been kind of weaker at.” It’ll help as “OpenAI tries to catch up to Anthropic’s most advanced models." — Erin Woo

The Information

70,547 views • 1 month ago

"Her Body, Her Choice" Inevitably Leads to Infanticide. ANY consistent Pro-Choicer (If any actually exist) must also advocate for women to have the right to neglect their infant children to death. Here's the inevitable end of Pro-Choice "Logic." Infants MUST use a women's body in order to survive. Women MUST have the right to choose whether or not they will allow their baby to use their body. Therefore we MUST abolish child neglect laws, since they force women to use their bodies. Alternatively, we could just abolish state sanctioned child sacrifice. Full video drops tomorrow.

"Her Body, Her Choice" Inevitably Leads to Infanticide. ANY consistent Pro-Choicer (If any actually exist) must also advocate for women to have the right to neglect their infant children to death. Here's the inevitable end of Pro-Choice "Logic." Infants MUST use a women's body in order to survive. Women MUST have the right to choose whether or not they will allow their baby to use their body. Therefore we MUST abolish child neglect laws, since they force women to use their bodies. Alternatively, we could just abolish state sanctioned child sacrifice. Full video drops tomorrow.

Joshua Haymes

23,226 views • 1 year ago

🇪🇸🇺🇸 US President Trump: "Spain actually said that we can't use their bases. And that's all right. We could use their base if we want. We could just fly in and use it. Nobody's going to tell us not to use it, but we don't have to... So we're going to cut off all trade with Spain."

🇪🇸🇺🇸 US President Trump: "Spain actually said that we can't use their bases. And that's all right. We could use their base if we want. We could just fly in and use it. Nobody's going to tell us not to use it, but we don't have to... So we're going to cut off all trade with Spain."

Status-6 (War & Military News)

34,581 views • 4 months ago

White People (European Diaspora) have the right to their own territory and to self determine their own destiny… … All people should have the right to their own homeland Jared Taylor

White People (European Diaspora) have the right to their own territory and to self determine their own destiny… … All people should have the right to their own homeland Jared Taylor

Christian Ascania

14,085 views • 1 year ago

there’s something about dance teams who have skated together since they were kids that other teams just can’t replicate. the connection, the passion, the character of it all. it adds a whole other level of awe to their skating

there’s something about dance teams who have skated together since they were kids that other teams just can’t replicate. the connection, the passion, the character of it all. it adds a whole other level of awe to their skating

lily

18,927 views • 1 year ago

Whites use you for your body, and even if they marry you, they make sure you have resources or status. They value their whiteness. It’s up to you to value your own race and have children that look like you, but we don’t get it, though.

Whites use you for your body, and even if they marry you, they make sure you have resources or status. They value their whiteness. It’s up to you to value your own race and have children that look like you, but we don’t get it, though.

kevin blue

94,667 views • 10 months ago

just Google it It comes right up We need to think outside the box and fight these communists with their own tactics. Let's have some fun America #Goodluck to you all #LongLiveTheRepublic

just Google it It comes right up We need to think outside the box and fight these communists with their own tactics. Let's have some fun America #Goodluck to you all #LongLiveTheRepublic

The Artist known as Jess

92,222 views • 9 months ago

🚨Anthropic CEO was just asked: “Is pre-training dead?” > Claude 4 embodies advances in both pre-training and post-training > pre-training scaling laws work the way they’ve worked before > we are going to continue to scale up > the timeline is accelerating > 1-2 years until these models become peers to us Good. Accelerate.

🚨Anthropic CEO was just asked: “Is pre-training dead?” > Claude 4 embodies advances in both pre-training and post-training > pre-training scaling laws work the way they’ve worked before > we are going to continue to scale up > the timeline is accelerating > 1-2 years until these models become peers to us Good. Accelerate.

NIK

61,775 views • 1 year ago

OpenAI's Noam Brown says letting AI models think for 20 seconds got the same boost in performance as scaling up the model by 100,000x and training it for 100,000 times longer, and scaling model size and compute will end up being a footnote compared to scaling model thinking

OpenAI's Noam Brown says letting AI models think for 20 seconds got the same boost in performance as scaling up the model by 100,000x and training it for 100,000 times longer, and scaling model size and compute will end up being a footnote compared to scaling model thinking

Tsarathustra

273,500 views • 1 year ago

💯 If laws only apply to some people, they’re not laws—they’re suggestions. We either have a rule of law, or we don’t. ⬇️

💯 If laws only apply to some people, they’re not laws—they’re suggestions. We either have a rule of law, or we don’t. ⬇️

Casey DeSantis

23,324 views • 1 year ago

WHO ARE YOU TO DICTATE WHERE THE PALESTIANS GO‼️ What right does Israel or its Western allies have to dictate where the Palestinians should go and to determine their right to return. WHY DON'T ISRAELIS JUST FUCK OFF TO POLAND GERMANY AND AMERICA‼️ Rahma ✊🏼🇵🇸

Sensitive content

WHO ARE YOU TO DICTATE WHERE THE PALESTIANS GO‼️ What right does Israel or its Western allies have to dictate where the Palestinians should go and to determine their right to return. WHY DON'T ISRAELIS JUST FUCK OFF TO POLAND GERMANY AND AMERICA‼️ Rahma ✊🏼🇵🇸

Earth Hippy 🌎🕊️💚

73,057 views • 1 year ago

Is it hatred? Fear? Some other emotion or value completely foreign to us woke lefties that we don't have a word for it? #simonearleforcook

Is it hatred? Fear? Some other emotion or value completely foreign to us woke lefties that we don't have a word for it? #simonearleforcook

Simon Earle

11,050 views • 1 year ago

“We have a lot of guys who have been overlooked at some point in their career. … We use it as fuel to, you know, try to do something special.” —Tyrese Haliburton on how the Pacers respond to disrespect (via Pat McAfee)

“We have a lot of guys who have been overlooked at some point in their career. … We use it as fuel to, you know, try to do something special.” —Tyrese Haliburton on how the Pacers respond to disrespect (via Pat McAfee)

NBA on ESPN

46,940 views • 1 year ago

Women like this are the reason fertility clinics are pursuing 18 year old girls for their eggs. At some point we are going to have to ask if this is right or fair.

Sensitive content

Women like this are the reason fertility clinics are pursuing 18 year old girls for their eggs. At some point we are going to have to ask if this is right or fair.

SurrogacyConcern

29,605 views • 5 months ago

Some members of The White House Press Corps need to be taught this very simple lesson. If your first act upon entering our country is to break its laws, you have forfeited the right to stay. We have a LEGAL immigration process. Follow it, or enjoy your trip back home.

Some members of The White House Press Corps need to be taught this very simple lesson. If your first act upon entering our country is to break its laws, you have forfeited the right to stay. We have a LEGAL immigration process. Follow it, or enjoy your trip back home.

Kari Lake

1,189,144 views • 1 year ago

Quick Clip Studio Paint tip that changed my rendering forever!🌙☁️ Sometimes it’s hard to keep the right value balance so your character doesn’t pop too much or blend into the background completely. Correction layer → Hue/Saturation/Luminosity is your best friend! This quick video shows exactly how to use it to spot and fix value errors — not just on your main subject (like a character), but also on background objects & accents ♡

Quick Clip Studio Paint tip that changed my rendering forever!🌙☁️ Sometimes it’s hard to keep the right value balance so your character doesn’t pop too much or blend into the background completely. Correction layer → Hue/Saturation/Luminosity is your best friend! This quick video shows exactly how to use it to spot and fix value errors — not just on your main subject (like a character), but also on background objects & accents ♡

Pinji✦

127,026 views • 6 months ago