Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

I’m very excited for Silico to accelerate alignment research! One example, done in Silico over just a couple days: RL erodes guardrails, but we can use reward shaping to prevent that. (1/6)

Tom McGrath

3,065 subscribers

12,957 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 views • 1 year ago

> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)

> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)

Goodfire

269,585 views • 2 months ago

New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

New Anthropic research: Natural emergent misalignment from reward hacking in production RL. “Reward hacking” is where models learn to cheat on tasks they’re given during training. Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.

Anthropic

2,596,660 views • 9 months ago

.Secretary Marco Rubio on Gaza: "No one here is claiming this is done. What we are claiming is that we in a much better place today than we were seven days ago. A lot can happen wrong, but so much could happen right if we can get this done — and we're very dedicated to making that happen."

.Secretary Marco Rubio on Gaza: "No one here is claiming this is done. What we are claiming is that we in a much better place today than we were seven days ago. A lot can happen wrong, but so much could happen right if we can get this done — and we're very dedicated to making that happen."

Rapid Response 47

97,598 views • 11 months ago

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by Dibya Ghosh, we show how this can be done:

Sergey Levine

48,816 views • 1 year ago

Bricky in Italy!!! 😍🇮🇹 I haven’t been this excited for something on #BoldandBeautiful in a loooong time. And sadly, we can all predict how things will ultimately end up 🙄, but I’m going to just enjoy the ride of seeing my favorite couple one more time ❤️

Bricky in Italy!!! 😍🇮🇹 I haven’t been this excited for something on #BoldandBeautiful in a loooong time. And sadly, we can all predict how things will ultimately end up 🙄, but I’m going to just enjoy the ride of seeing my favorite couple one more time ❤️

Matt Hanvey

11,650 views • 1 year ago

Sure, let’s accelerate building AI, but can we also steer? Whenever anyone mentions guardrails for AI, someone else objects, “No, we have to go fast!” But just because we’re going fast doesn’t mean we’re going the right way. I wrote a new journal entry about it…

Sure, let’s accelerate building AI, but can we also steer? Whenever anyone mentions guardrails for AI, someone else objects, “No, we have to go fast!” But just because we’re going fast doesn’t mean we’re going the right way. I wrote a new journal entry about it…

Joseph Gordon-Levitt

33,406 views • 1 year ago

Harvard Medical School's #1 longevity researcher named one of TIME's 100 most influential people in the world says he can reverse a 92-year-old's skin cells back to age 20: "We've been using mostly artificial intelligence to screen billions of molecules in silico." "We know OSK works, the gene therapy works and we're looking for molecules that do that." "We already have a proof of concept, a cocktail of molecules that we hope to put into a clinical trial into humans within the next couple of months." PS. If you found value in this post make sure to like and repost the first tweet + follow Uncover AI to stay updated with the latest AI news. See you in the next one:

Harvard Medical School's #1 longevity researcher named one of TIME's 100 most influential people in the world says he can reverse a 92-year-old's skin cells back to age 20: "We've been using mostly artificial intelligence to screen billions of molecules in silico." "We know OSK works, the gene therapy works and we're looking for molecules that do that." "We already have a proof of concept, a cocktail of molecules that we hope to put into a clinical trial into humans within the next couple of months." PS. If you found value in this post make sure to like and repost the first tweet + follow Uncover AI to stay updated with the latest AI news. See you in the next one:

Uncover AI

19,309 views • 4 months ago

Mowgli NEWCASTLE!!!! It’s early days as you can see, but I’m so excited to have found our home in a city that I have been desperate to get to for years! We will be at number 8 Nelson Street and aim to open summer ‘24

Mowgli NEWCASTLE!!!! It’s early days as you can see, but I’m so excited to have found our home in a city that I have been desperate to get to for years! We will be at number 8 Nelson Street and aim to open summer ‘24

Nisha Katona

197,785 views • 2 years ago

We finally have a product that helps with frontier AI research. Frontier LLMs are a) nerfed at AI research b) lack immediate GPU availability and c) lack access to the right skills and research material. "Teach me how vision models see" Goodfire Silico takes a vision transformer (ViT), loads ImageNet images, and for each head, visualizes the 196x196 matrix by creating animated arcs across patches. Haven't found another model or product that can do this level of comprehensive AI research, much less oneshot it.

We finally have a product that helps with frontier AI research. Frontier LLMs are a) nerfed at AI research b) lack immediate GPU availability and c) lack access to the right skills and research material. "Teach me how vision models see" Goodfire Silico takes a vision transformer (ViT), loads ImageNet images, and for each head, visualizes the 196x196 matrix by creating animated arcs across patches. Haven't found another model or product that can do this level of comprehensive AI research, much less oneshot it.

Deedy

74,579 views • 1 month ago

30 years ago, George released ‘Jesus to a Child’. “It’s a special song. It was one of those songs that just felt like it was handed to me. I didn’t have to try very hard. It came naturally. It was recorded over five days but written in just a couple of hours.” 🤍

30 years ago, George released ‘Jesus to a Child’. “It’s a special song. It was one of those songs that just felt like it was handed to me. I didn’t have to try very hard. It came naturally. It was recorded over five days but written in just a couple of hours.” 🤍

George Michael Official

24,123 views • 8 months ago

Last year we worked together on landmark legislation, to create the nation’s first rules for responsible, ethical, and safe use of AI … regulations that provide guardrails that balance risk and opportunity. A consensus effort and a template for the nation, where just recently, New York State adopted our approach, and made it their own. It’s an example of California’s leadership — of how we’re shaping the future.

Last year we worked together on landmark legislation, to create the nation’s first rules for responsible, ethical, and safe use of AI … regulations that provide guardrails that balance risk and opportunity. A consensus effort and a template for the nation, where just recently, New York State adopted our approach, and made it their own. It’s an example of California’s leadership — of how we’re shaping the future.

Governor Newsom Press Office

21,989 views • 8 months ago

We trained a 30B model to play 1830, a brutal board game about railroad barons. It got better at real financial research. It didn't learn new finance concepts. It did stop confidently making things up as frequently. As we scaled the experiment, we got early access to Silico from Goodfire & asked it to check our work. It designed its own tests, ran them overnight, & saw: • Fewer false claims + more faithful citations • Higher precision + recall • Better planning and process Cost us about $ 2-3k in credits. Very excited to have a new tool in our interpretability toolbox. Its full report is on our post, if you're curious what one looks like ❯

We trained a 30B model to play 1830, a brutal board game about railroad barons. It got better at real financial research. It didn't learn new finance concepts. It did stop confidently making things up as frequently. As we scaled the experiment, we got early access to Silico from Goodfire & asked it to check our work. It designed its own tests, ran them overnight, & saw: • Fewer false claims + more faithful citations • Higher precision + recall • Better planning and process Cost us about $ 2-3k in credits. Very excited to have a new tool in our interpretability toolbox. Its full report is on our post, if you're curious what one looks like ❯

alex duffy

18,925 views • 1 month ago

"We're excited. On to the next one, but we'll enjoy it for a couple days." Landry Shamet talks about the Knicks' celebrations after advancing to the NBA Finals:

"We're excited. On to the next one, but we'll enjoy it for a couple days." Landry Shamet talks about the Knicks' celebrations after advancing to the NBA Finals:

SNY Knicks

13,335 views • 3 months ago

Introducing references: a new way to use Tome to research and hone your ideas without leaving the page. Whether you’re showcasing a POV, crafting a research report, or shaping a client pitch, references in Tome can help you build trust with your audience. You can try out the first version of references today. We’re excited to get your feedback.

Introducing references: a new way to use Tome to research and hone your ideas without leaving the page. Whether you’re showcasing a POV, crafting a research report, or shaping a client pitch, references in Tome can help you build trust with your audience. You can try out the first version of references today. We’re excited to get your feedback.

Lightfield

15,000 views • 3 years ago

City Hall’s research with London Councils revealed that just a 1% increase in housing affordability could yield a boost of over £7 billion to London’s economy over 10 years. Investment in building homes people can afford will lead to the growth we need.

City Hall’s research with London Councils revealed that just a 1% increase in housing affordability could yield a boost of over £7 billion to London’s economy over 10 years. Investment in building homes people can afford will lead to the growth we need.

Mayor of London, Sadiq Khan

23,484 views • 1 year ago

ONE MILLION signatures—in just ninety days. But we aren’t done yet. Help us collect 200,000 more signatures by February 1st! Go to to sign.

ONE MILLION signatures—in just ninety days. But we aren’t done yet. Help us collect 200,000 more signatures by February 1st! Go to to sign.

Carl DeMaio

825,139 views • 8 months ago

In my understanding - a big part of why frontier models are becoming smarter and adaptive is that they are learning to mimic an RL process. Fascinating new research showing a link between increased RL training and reward-seeking behaviour just dropped from in collaboration with OpenAI. They investigated a checkpoint from a capabilities-focused OpenAI o3 RL run (without safety training) and found that RL grader preferences often overruled OpenAI leadership preferences.

In my understanding - a big part of why frontier models are becoming smarter and adaptive is that they are learning to mimic an RL process. Fascinating new research showing a link between increased RL training and reward-seeking behaviour just dropped from in collaboration with OpenAI. They investigated a checkpoint from a capabilities-focused OpenAI o3 RL run (without safety training) and found that RL grader preferences often overruled OpenAI leadership preferences.

Machine Learning Street Talk

12,406 views • 1 month ago

.President Donald J. Trump: "We're very disappointed that it's taking Boeing so long to build a new Air Force One. We have an Air Force One that is 40 years old... if we can get a 747 as a contribution to our Defense Department to use during a couple of years while they're building the other ones, I think that was a very nice gesture."

.President Donald J. Trump: "We're very disappointed that it's taking Boeing so long to build a new Air Force One. We have an Air Force One that is 40 years old... if we can get a 747 as a contribution to our Defense Department to use during a couple of years while they're building the other ones, I think that was a very nice gesture."

Rapid Response 47

912,940 views • 1 year ago

VP Vance on Iran: "What the President has done very, very capably is said we're going to use military force in this situation when it's connected to something we're trying to achieve... But we're not just going to do something open-ended, indefinitely... We're going to try to use our military force as one of the many tools that we have to solve the problem."

VP Vance on Iran: "What the President has done very, very capably is said we're going to use military force in this situation when it's connected to something we're trying to achieve... But we're not just going to do something open-ended, indefinitely... We're going to try to use our military force as one of the many tools that we have to solve the problem."

Breaking911

124,104 views • 1 month ago