Loading video...

Video Failed to Load

Go Home

We release Diamond Maps💎 unlocking accurate and efficient guidance for diffusion models. Our experiments show that our methods scale incredibly well. Excited to see what people will build with this! Accurate guidance has been a notoriously hard problem, but in this work, we’re bringing TWO (!) solutions to the...

peholderrieth's profile picture

Peter Holderrieth

5,018 subscribers

59,916 views • 2 months ago •via X (Twitter)

EducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Diffusion models are an amazing tool for cofolding, they allow us to predict a protein and the molecule bound to it at once. But they are not exactly fast and require a lot of denoising steps to get accurate predictions. So we distilled ours. Meet DeCAF-Pearl: the first flow map model for all-atom cofolding. Instead of inching along the denoising trajectory, a flow map learns to jump across it. DeCAF-Pearl runs structure generation ~5x faster than Pearl, our SOTA model, while still maintaining the performance of the teacher model. That speed up allows us to run larger experiments and generate more synthetic data to improve our models. Getting there meant reparameterizing into noise-level space to stabilize gradients, committing to clean-structure prediction to keep the rigid-alignment loss biomolecules needed, and building DeCAF-Search, one steering algorithm for every compute budget. For more technical details, read out blog post: And the paper:
0:11

Diffusion models are an amazing tool for cofolding, they allow us to predict a protein and the molecule bound to it at once. But they are not exactly fast and require a lot of denoising steps to get accurate predictions. So we distilled ours. Meet DeCAF-Pearl: the first flow map model for all-atom cofolding. Instead of inching along the denoising trajectory, a flow map learns to jump across it. DeCAF-Pearl runs structure generation ~5x faster than Pearl, our SOTA model, while still maintaining the performance of the teacher model. That speed up allows us to run larger experiments and generate more synthetic data to improve our models. Getting there meant reparameterizing into noise-level space to stabilize gradients, committing to clean-structure prediction to keep the rigid-alignment loss biomolecules needed, and building DeCAF-Search, one steering algorithm for every compute budget. For more technical details, read out blog post: And the paper:

Sergey Edunov

36,985 views • 27 days ago

Can we use motion to prompt diffusion models? Our #ICLR2024 paper does just that. We propose Motion Guidance, a technique that allows users to edit an image by specifying “where things should move.”
0:15

Can we use motion to prompt diffusion models? Our #ICLR2024 paper does just that. We propose Motion Guidance, a technique that allows users to edit an image by specifying “where things should move.”

Daniel Geng

16,306 views • 2 years ago

1/ Happy to share VADER: Video Diffusion Alignment via Reward Gradients. We adapt foundational video diffusion models using pre-trained reward models to generate high-quality, aligned videos for various end-applications. Below we generated a short movie using VADER 😀, we used ChatGPT to write a script and an off-the-shelf AI music generator to generate the sound. Our code & weights are open-sourced:
0:16

1/ Happy to share VADER: Video Diffusion Alignment via Reward Gradients. We adapt foundational video diffusion models using pre-trained reward models to generate high-quality, aligned videos for various end-applications. Below we generated a short movie using VADER 😀, we used ChatGPT to write a script and an off-the-shelf AI music generator to generate the sound. Our code & weights are open-sourced:

Mihir Prabhudesai

13,368 views • 1 year ago

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.
0:10

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.

Mihir Prabhudesai

104,934 views • 1 year ago

We are also releasing self-contained lecture notes that explain flow matching and diffusion models from scratch. This goes from "zero" to the state-of-the-art in modern Generative AI. 📖 Read the notes here: Joint work with Ezra Erives.
0:15

We are also releasing self-contained lecture notes that explain flow matching and diffusion models from scratch. This goes from "zero" to the state-of-the-art in modern Generative AI. 📖 Read the notes here: Joint work with Ezra Erives.

Peter Holderrieth

475,051 views • 3 months ago

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.
0:19

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,449 views • 2 years ago

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation paper page: Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.
0:17

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation paper page: Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

AK

375,090 views • 3 years ago

After 5 months of planning, our first Live2D Jakarta Chapter event was a huge success! We brought an interactive Live2D showcase to the biggest anime con in Indonesia, featuring over 20 finest models created by our local artists and riggers. This showcase also included a custom-made plugin for VTube Studio that adds touch interactivity to Live2D models, allowing visitors to interact with VTuber models in a more engaging way! We hope this demo could open up new potential for Vtubers in IRL events. Special thanks to Live2D Inc. for trusting me to host this event! The showcase attracted many curious visitors and gave us the opportunity to introduce this tech to a wider audience, and we hope we are opening up new opportunities for everyone. Look forward to our next one! (Model: Key Oriesa by karamomo🍑 | Live2D Rigging | Food Illustration)
0:19

After 5 months of planning, our first Live2D Jakarta Chapter event was a huge success! We brought an interactive Live2D showcase to the biggest anime con in Indonesia, featuring over 20 finest models created by our local artists and riggers. This showcase also included a custom-made plugin for VTube Studio that adds touch interactivity to Live2D models, allowing visitors to interact with VTuber models in a more engaging way! We hope this demo could open up new potential for Vtubers in IRL events. Special thanks to Live2D Inc. for trusting me to host this event! The showcase attracted many curious visitors and gave us the opportunity to introduce this tech to a wider audience, and we hope we are opening up new opportunities for everyone. Look forward to our next one! (Model: Key Oriesa by karamomo🍑 | Live2D Rigging | Food Illustration)

Ran 🐲 Live2D Animator

164,863 views • 1 month ago

Production Design for Komatsu Village in Episode 1 of Blue Eye Samurai. We did models like this for our previs department to scout the sets for our directors and showrunners. This was the first set we crafted for the show. Thanks for looking!
0:15

Production Design for Komatsu Village in Episode 1 of Blue Eye Samurai. We did models like this for our previs department to scout the sets for our directors and showrunners. This was the first set we crafted for the show. Thanks for looking!

Jason Scheier

46,416 views • 2 years ago

Welcome to the team, Chris. This is an important moment for our country. We are working together to build our communities, grow a strong economy, and create new opportunities for people in Acadie—Annapolis and across Canada.
0:14

Welcome to the team, Chris. This is an important moment for our country. We are working together to build our communities, grow a strong economy, and create new opportunities for people in Acadie—Annapolis and across Canada.

Mark Carney

478,455 views • 8 months ago

We had our official #LostRecords launch party yesterday and the amazing Nora Kelly and her band performed See You in Hell and other awesome songs for us. It was so emotional. I’m lucky I’ve been able to work on this dream project of mine with such a talented team!
0:14

We had our official #LostRecords launch party yesterday and the amazing Nora Kelly and her band performed See You in Hell and other awesome songs for us. It was so emotional. I’m lucky I’ve been able to work on this dream project of mine with such a talented team!

Michel Koch

13,726 views • 1 year ago

We are so honored to have our work featured in the “Abracadabra” Lady Gaga music video! Bringing this design by Olivier Theysken to life was quite a task! Many thanks to Peri Rosenzweig with HardStyle for the opportunity. We are beyond thrilled and so grateful!
0:08

We are so honored to have our work featured in the “Abracadabra” Lady Gaga music video! Bringing this design by Olivier Theysken to life was quite a task! Many thanks to Peri Rosenzweig with HardStyle for the opportunity. We are beyond thrilled and so grateful!

Renee Masoomian

87,689 views • 1 year ago

Too many people to thank in one tweet (will save that for tonight's podcast) but was absolutely awesome to be with everyone from our Badlands team and community in NYC last night. These events are always a bit overwhelming in all the good ways and it is nice to be reminded of how many great people we get to come across thanks to this team (even if they are bad) and this podcast....we're back to our draft coverage and we'll see you again in August!
0:04

Too many people to thank in one tweet (will save that for tonight's podcast) but was absolutely awesome to be with everyone from our Badlands team and community in NYC last night. These events are always a bit overwhelming in all the good ways and it is nice to be reminded of how many great people we get to come across thanks to this team (even if they are bad) and this podcast....we're back to our draft coverage and we'll see you again in August!

Joe Caporoso

18,900 views • 2 months ago

Working with multiple models in Chat? The model picker in VS Code is now organized by provider, making it easier to browse, search, and switch between models. You'll also see provider names next to your recent models for quicker recognition. 💡 Tip: Use /models for quick access.
0:14

Working with multiple models in Chat? The model picker in VS Code is now organized by provider, making it easier to browse, search, and switch between models. You'll also see provider names next to your recent models for quicker recognition. 💡 Tip: Use /models for quick access.

Visual Studio Code

32,935 views • 1 month ago

CES was an enormous success. We saw thousands of people and a few dogs interacting with our agents and robots. We booked this booth at the last minute, mainly to book next year’s booth. Truly extraordinary what our team was able to put together on such short notice. Next year we reserved one of the largest booths at CES in the highest traffic location for AI and Robotics. We plan to unveil something extraordinary. Until the next show, we will be debuting the first custom models that will emerge from the first humanoid robotics hot rod shop. These models will be walking around, talking and interacting at tradeshows all over the world over the next few months. We are in the process of acquiring every humanoid robot available on the market to modify these as well. Stay tuned for more updates, the future is here and it’s rapidly accelerating.
0:15

CES was an enormous success. We saw thousands of people and a few dogs interacting with our agents and robots. We booked this booth at the last minute, mainly to book next year’s booth. Truly extraordinary what our team was able to put together on such short notice. Next year we reserved one of the largest booths at CES in the highest traffic location for AI and Robotics. We plan to unveil something extraordinary. Until the next show, we will be debuting the first custom models that will emerge from the first humanoid robotics hot rod shop. These models will be walking around, talking and interacting at tradeshows all over the world over the next few months. We are in the process of acquiring every humanoid robot available on the market to modify these as well. Stay tuned for more updates, the future is here and it’s rapidly accelerating.

Parallel

16,844 views • 1 year ago

World Models are the path for some AI Models in the future. But how can we efficiently train these models to not only see the world the way humans do but to see the world in a new and unique way. By visualizing, what is normally sequenced audio patterns, we can derive much more insights. Here we see Paganini in a visual form that can than be described and transcribed into a World Model. We can observe connections in a manner that may not have been clear prior to the digitalization of music and sound in this way. The company with the most valuable potential in building a World Model is Tesla. Not that this type of visualization is being used, but that the mechanisms are in place, and the technology is in place for the company to thrive in this new form of AI.
0:19

World Models are the path for some AI Models in the future. But how can we efficiently train these models to not only see the world the way humans do but to see the world in a new and unique way. By visualizing, what is normally sequenced audio patterns, we can derive much more insights. Here we see Paganini in a visual form that can than be described and transcribed into a World Model. We can observe connections in a manner that may not have been clear prior to the digitalization of music and sound in this way. The company with the most valuable potential in building a World Model is Tesla. Not that this type of visualization is being used, but that the mechanisms are in place, and the technology is in place for the company to thrive in this new form of AI.

Brian Roemmele

57,424 views • 7 months ago

We’re excited to introduce ShinkaEvolve: An open-source framework that evolves programs for scientific discovery with unprecedented sample-efficiency. Blog: Code: Like AlphaEvolve and its variants, our framework leverages LLMs to find state-of-the-art solutions to complex problems, but using orders of magnitude fewer resources! Many evolutionary AI systems are powerful but act like brute-force engines, burning thousands of samples to find good solutions. This makes discovery slow and expensive. We took inspiration from the efficiency of nature. ‘Shinka’ (進化) is Japanese for evolution, and we designed our system to be just as resourceful. On the classic circle packing optimization problem, ShinkaEvolve discovered a new state-of-the-art solution using only 150 samples. This is a big leap in efficiency compared to previous methods that required thousands of evaluations. We applied ShinkaEvolve to a diverse set of hard problems with real-world applications: 1/ AIME Math Reasoning: It evolved sophisticated agentic scaffolds that significantly outperform strong baselines, discovering an entire Pareto frontier of solutions trading performance for efficiency. 2/ Competitive Programming: On ALE-Bench (a benchmark for NP-Hard optimization problems), ShinkaEvolve took the best existing agent's solutions and improved them, turning a 5th place solution on one task into a 2nd place leaderboard rank in a competitive programming competition. 3/ LLM Training: We even turned ShinkaEvolve inward to improve LLMs themselves. It tackled the open challenge of designing load balancing losses for Mixture-of-Experts (MoE) models. It discovered a novel loss function that leads to better expert specialization and consistently improves model performance and perplexity. ShinkaEvolve achieves its remarkable sample-efficiency through three key innovations that work together: (1) an adaptive parent sampling strategy to balance exploration and exploitation, (2) novelty-based rejection filtering to avoid redundant work, and (3) a bandit-based LLM ensemble that dynamically picks the best model for the job. By making ShinkaEvolve open-source and highly sample-efficient, our goal is to democratize access to advanced, open-ended discovery tools. Our vision for ShinkaEvolve is to be an easy-to-use companion tool to help scientists and engineers with their daily work. We believe that building more efficient, nature-inspired systems is key to unlocking the future of AI-driven scientific research. We are excited to see what the community builds with it! Learn more in our technical report:
0:11

We’re excited to introduce ShinkaEvolve: An open-source framework that evolves programs for scientific discovery with unprecedented sample-efficiency. Blog: Code: Like AlphaEvolve and its variants, our framework leverages LLMs to find state-of-the-art solutions to complex problems, but using orders of magnitude fewer resources! Many evolutionary AI systems are powerful but act like brute-force engines, burning thousands of samples to find good solutions. This makes discovery slow and expensive. We took inspiration from the efficiency of nature. ‘Shinka’ (進化) is Japanese for evolution, and we designed our system to be just as resourceful. On the classic circle packing optimization problem, ShinkaEvolve discovered a new state-of-the-art solution using only 150 samples. This is a big leap in efficiency compared to previous methods that required thousands of evaluations. We applied ShinkaEvolve to a diverse set of hard problems with real-world applications: 1/ AIME Math Reasoning: It evolved sophisticated agentic scaffolds that significantly outperform strong baselines, discovering an entire Pareto frontier of solutions trading performance for efficiency. 2/ Competitive Programming: On ALE-Bench (a benchmark for NP-Hard optimization problems), ShinkaEvolve took the best existing agent's solutions and improved them, turning a 5th place solution on one task into a 2nd place leaderboard rank in a competitive programming competition. 3/ LLM Training: We even turned ShinkaEvolve inward to improve LLMs themselves. It tackled the open challenge of designing load balancing losses for Mixture-of-Experts (MoE) models. It discovered a novel loss function that leads to better expert specialization and consistently improves model performance and perplexity. ShinkaEvolve achieves its remarkable sample-efficiency through three key innovations that work together: (1) an adaptive parent sampling strategy to balance exploration and exploitation, (2) novelty-based rejection filtering to avoid redundant work, and (3) a bandit-based LLM ensemble that dynamically picks the best model for the job. By making ShinkaEvolve open-source and highly sample-efficient, our goal is to democratize access to advanced, open-ended discovery tools. Our vision for ShinkaEvolve is to be an easy-to-use companion tool to help scientists and engineers with their daily work. We believe that building more efficient, nature-inspired systems is key to unlocking the future of AI-driven scientific research. We are excited to see what the community builds with it! Learn more in our technical report:

Sakana AI

359,425 views • 9 months ago

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,039 views • 1 year ago

I'm excited to share our paper "Flow Matching for Posterior Inference with Simulator Feedback" 🤩 The key idea is to incorporate physics-constraints with guiding so that the posterior isn't biased. Highly accurate solutions, better than pure learning!
0:10

I'm excited to share our paper "Flow Matching for Posterior Inference with Simulator Feedback" 🤩 The key idea is to incorporate physics-constraints with guiding so that the posterior isn't biased. Highly accurate solutions, better than pure learning!

Thuerey Group at TUM

21,288 views • 1 year ago

IT’S OUT! Head over to the Gorillaz YouTube channel now (link in bio) to watch the full film. This film is a true love letter to the golden era of 2D animation. We worked incredibly hard alongside Jamie Hewlett to make it as period-accurate as possible and, like all our projects, it was hand-drawn by an incredible team of artists. We even used a variety of practical effects in the film, which was a lot of fun to do. We had a brilliant time making this film and learnt a huge amount while studying the masters of the 60s and 70s. If you’ve been following the release of The Mountain, you know how special and meaningful this project is. We’re so proud to be a part of it and want to say a massive thank you to Jamie and Eleven Management for trusting us to tell this beautiful story. We couldn’t be prouder of the work we did together, and we're so happy to be sharing it with the world! #Gorillaz #THELINE #TheMountain
0:19

IT’S OUT! Head over to the Gorillaz YouTube channel now (link in bio) to watch the full film. This film is a true love letter to the golden era of 2D animation. We worked incredibly hard alongside Jamie Hewlett to make it as period-accurate as possible and, like all our projects, it was hand-drawn by an incredible team of artists. We even used a variety of practical effects in the film, which was a lot of fun to do. We had a brilliant time making this film and learnt a huge amount while studying the masters of the 60s and 70s. If you’ve been following the release of The Mountain, you know how special and meaningful this project is. We’re so proud to be a part of it and want to say a massive thank you to Jamie and Eleven Management for trusting us to tell this beautiful story. We couldn’t be prouder of the work we did together, and we're so happy to be sharing it with the world! #Gorillaz #THELINE #TheMountain

THE LINE

258,203 views • 4 months ago