Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Alex Rives

14,396 subscribers

206,827 views • 1 year ago •via X (Twitter)

Science & Technology Education Health & Wellness

Anya Rossi• Live Now

Private livecam show

10 Comments

Alex Rives1 year ago

ESM C models establish a frontier of performance as a function of parameter scale. We see large improvements across all parameter scales over previous state of the art models. Read more:

Alex Rives1 year ago

Information about protein structure in ESM C representations improves predictably with increasing training compute, demonstrating linear scaling across multiple orders of magnitude. (We overtrained the 300M and 600M models past the predicted point of compute optimality).

Alex Rives1 year ago

ESM C comes with major performance and efficiency benefits over ESM2. The 300M parameter ESM C delivers similar performance to ESM2 650M. The 600M delivers similar performance to ESM2 3B and approaches the capabilities of the ESM2 15B, with far greater efficiency. The 6B parameter ESM C outperforms all ESM2 models by a wide margin.

Alex Rives1 year ago

Today we’re releasing ESM C 300M, and 600M with open weights. ESM C 6B is available immediately on EvolutionaryScale Forge for academic use, and AWS Sagemaker for commercial use. ESM C will be on NVIDIA BioNemo soon. We’re excited to see what you build with ESM!

Nathan Benaich1 year ago

cool work!

Alex Rives1 year ago

Thanks Nathan!

Kevin K. Yang 楊凱筌1 year ago

Why is it called Cambrian?

Alex Rives1 year ago

wouldn't want to spoil the mystery!

Surge Biswas1 year ago

Congrats Alex and the evoscale team! Exciting

Alex Rives1 year ago

Thank you Surge!

Related Videos

We're thrilled to present ESM3 in Science Magazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence. Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering. ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins. ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein. We estimate this is equivalent to simulating five hundred million years of evolution.

We're thrilled to present ESM3 in Science Magazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence. Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering. ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins. ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein. We estimate this is equivalent to simulating five hundred million years of evolution.

Alex Rives

227,177 views • 1 year ago

What could scaling unlock for biology? Introducing ProGen3- our next AI foundation models for protein generation. We develop compute-optimal scaling laws up to 46B parameters on 1.5T tokens with real evidence in the wet lab. +we solve a new set of challenges for drug discovery

What could scaling unlock for biology? Introducing ProGen3- our next AI foundation models for protein generation. We develop compute-optimal scaling laws up to 46B parameters on 1.5T tokens with real evidence in the wet lab. +we solve a new set of challenges for drug discovery

Ali Madani

80,634 views • 1 year ago

Proteins are the machinery of life. Scientists have cataloged billions of protein sequences—but their biology is still mostly unknown. Today we're releasing a world model of protein biology: a scientific engine for prediction, design, and discovery that consists of ESMFold2, ESMC, and ESM Atlas. Together, they're helping to open up a new way for researchers to design proteins and speed up scientific discovery. Our mission is to cure or prevent disease. To do that, we need to accelerate science. That's why we're releasing all three openly.

Proteins are the machinery of life. Scientists have cataloged billions of protein sequences—but their biology is still mostly unknown. Today we're releasing a world model of protein biology: a scientific engine for prediction, design, and discovery that consists of ESMFold2, ESMC, and ESM Atlas. Together, they're helping to open up a new way for researchers to design proteins and speed up scientific discovery. Our mission is to cure or prevent disease. To do that, we need to accelerate science. That's why we're releasing all three openly.

biohub

71,955 views • 23 days ago

Neat new system to evolve protein-protein interactions - while altering both sides of the interface! Watch for a description of the clever method, read the paper to get the deets on how these interfaces evolved & predictive models trained on screen data.

Neat new system to evolve protein-protein interactions - while altering both sides of the interface! Watch for a description of the clever method, read the paper to get the deets on how these interfaces evolved & predictive models trained on screen data.

Julia Bauman

58,527 views • 2 years ago

Nobel laureate John Jumper (John Jumper) on AlphaFold and the future of science. At AI Startup School in San Francisco. 01:02 - Journey from Physics to Computational Biology 02:01 - Transition to Machine Learning and AI 02:59 - Joining Google DeepMind 03:47 - The Mission: Accelerating Scientific Discovery 04:54 - Understanding Protein Structures 07:44 - Challenges in Protein Structure Determination 11:21 - The Role of Data, Compute, and Research 13:28 - AlphaFold: A Breakthrough in Protein Prediction 14:09 - The Importance of Research and Ideas 17:48 - Real-World Applications and Impact 18:02 - Open Sourcing AlphaFold 21:07 - Unexpected Uses and Innovations 23:15 - Case Study: Molecular Syringe 25:28 - Future of AI in Science

Nobel laureate John Jumper (John Jumper) on AlphaFold and the future of science. At AI Startup School in San Francisco. 01:02 - Journey from Physics to Computational Biology 02:01 - Transition to Machine Learning and AI 02:59 - Joining Google DeepMind 03:47 - The Mission: Accelerating Scientific Discovery 04:54 - Understanding Protein Structures 07:44 - Challenges in Protein Structure Determination 11:21 - The Role of Data, Compute, and Research 13:28 - AlphaFold: A Breakthrough in Protein Prediction 14:09 - The Importance of Research and Ideas 17:48 - Real-World Applications and Impact 18:02 - Open Sourcing AlphaFold 21:07 - Unexpected Uses and Innovations 23:15 - Case Study: Molecular Syringe 25:28 - Future of AI in Science

Y Combinator

149,700 views • 11 months ago

My team is hiring in multimodal representation learning. We are working on tactile representations to equip robots with a physical understanding of the world. Bored of scraping disembodied internet data and looking for a new challenge? Apply here:

My team is hiring in multimodal representation learning. We are working on tactile representations to equip robots with a physical understanding of the world. Bored of scraping disembodied internet data and looking for a new challenge? Apply here:

Mustafa Mukadam

56,700 views • 2 years ago

In two new papers we have found that the ESM2 language model generalizes beyond natural proteins, and enables programmable generation of complex and modular protein structures.

In two new papers we have found that the ESM2 language model generalizes beyond natural proteins, and enables programmable generation of complex and modular protein structures.

Alex Rives

202,817 views • 3 years ago

Introducing Ray2, a new frontier in video generative models. Scaled to 10x compute, #Ray2 creates realistic videos with natural and coherent motion, unlocking new freedoms of creative expression and visual storytelling. Available now. Learn more

Introducing Ray2, a new frontier in video generative models. Scaled to 10x compute, #Ray2 creates realistic videos with natural and coherent motion, unlocking new freedoms of creative expression and visual storytelling. Available now. Learn more

Luma

4,155,902 views • 1 year ago

The next frontier after language models is world models, and learning to predict the next state of a world. Hear from our researchers how these models really work at a low-level and the mountain we're climbing. A general-purpose world simulator (AKA sci-fi) is now in sight.

The next frontier after language models is world models, and learning to predict the next state of a world. Hear from our researchers how these models really work at a low-level and the mountain we're climbing. A general-purpose world simulator (AKA sci-fi) is now in sight.

Odyssey

25,857 views • 6 months ago

HRT, one of the largest quantitative trading firms in the world, just revealed the scale of their AI operation. "two of the biggest ingredients you need to build deep learning models are compute and data. HRT has both in incredible orders of magnitude." "we have vast quantities of data, on the order of petabytes, just in terms of raw tick data we've written down since the beginning of electronic trading." "we have a very high compute-per-researcher ratio." then the line that puts it in perspective: "there probably aren't too many places on Earth you can do cutting-edge deep learning research at the level we do it, just because of how few places have access to as much compute." "it's basically the hyperscalers of the world and then we're in the next tranche." a trading firm putting itself in the same compute tier as Google, Microsoft, and Meta. not for chatbots, for trading.

HRT, one of the largest quantitative trading firms in the world, just revealed the scale of their AI operation. "two of the biggest ingredients you need to build deep learning models are compute and data. HRT has both in incredible orders of magnitude." "we have vast quantities of data, on the order of petabytes, just in terms of raw tick data we've written down since the beginning of electronic trading." "we have a very high compute-per-researcher ratio." then the line that puts it in perspective: "there probably aren't too many places on Earth you can do cutting-edge deep learning research at the level we do it, just because of how few places have access to as much compute." "it's basically the hyperscalers of the world and then we're in the next tranche." a trading firm putting itself in the same compute tier as Google, Microsoft, and Meta. not for chatbots, for trading.

Goshawk Trades

14,548 views • 17 days ago

It’s here! 🤩 Introducing the new Mammoth Practice and Training Facility, a state-of-the-art building we’ve brought to life in just 13 months.

It’s here! 🤩 Introducing the new Mammoth Practice and Training Facility, a state-of-the-art building we’ve brought to life in just 13 months.

Utah Mammoth

136,413 views • 9 months ago

Emilia Clarke shares the hilarious reality of learning a new language for Game of Thrones😂⚔️

Emilia Clarke shares the hilarious reality of learning a new language for Game of Thrones😂⚔️

Virgin Radio UK

22,744 views • 1 month ago

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 views • 1 year ago

Simulation is a key enabler for embodied AI. It allows us to massively scale up data collection in order to experience years in minutes. Nvidia's Isaac Lab allows for rapid learning of new skills in just a few days, which directly transfer to the real world.

Simulation is a key enabler for embodied AI. It allows us to massively scale up data collection in order to experience years in minutes. Nvidia's Isaac Lab allows for rapid learning of new skills in just a few days, which directly transfer to the real world.

RIVR

25,295 views • 1 year ago

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Galbot

37,804 views • 1 month ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,311 views • 3 years ago

A new generation of grapplers are learning about Keenan in the Gi 💀

A new generation of grapplers are learning about Keenan in the Gi 💀

BJJotter

29,394 views • 1 year ago

🎧 New Podcast 🎧 How is AI impacting different areas of scientific research? In this episode we discuss AlphaFold, the machine learning programme from DeepMind that can determine a protein structure from its amino acids alone. Listen here 👇

🎧 New Podcast 🎧 How is AI impacting different areas of scientific research? In this episode we discuss AlphaFold, the machine learning programme from DeepMind that can determine a protein structure from its amino acids alone. Listen here 👇

Royal Institution (now on Bluesky)

43,320 views • 2 years ago

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

Fangchen Liu

68,288 views • 1 year ago