Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Diffusion Transformers aren't just generative models, but also powerful multi-modal encoders. ConceptAttention creates rich heatmaps of text concepts in images from DiT representations. This even works on real images, and can be applied to tasks like segmentation! Demo 👇

Alec Helbling

10,167 subscribers

24,380 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Text-to-image diffusion transformer models learn to align text and image representations as a byproduct of their conditional denoising task. By taking the dot product between the text and image representations of a DiT model (like Flux 2), you can create rich saliency maps.

Text-to-image diffusion transformer models learn to align text and image representations as a byproduct of their conditional denoising task. By taking the dot product between the text and image representations of a DiT model (like Flux 2), you can create rich saliency maps.

Alec Helbling

94,065 views • 5 months ago

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,746 views • 3 years ago

JUST IN: Meta AI introduces Voicebox, an all-in-one generative speech model. Voicebox is an impressive breakthrough! It could do for speech what other models like GPT-3 and Stable Diffusion have done for text and images. Some key details: - Voicebox can synthesize speech across 6 languages - It's a general-purpose model that can perform tasks it wasn't trained on. It can perform noise removal, content editing, style conversion, and more - Supports in-context text-to-speech synthesis and cross-lingual style transfer - It's 20x faster than current models and outperforms single-purpose models through in-context learning paper: blog:

JUST IN: Meta AI introduces Voicebox, an all-in-one generative speech model. Voicebox is an impressive breakthrough! It could do for speech what other models like GPT-3 and Stable Diffusion have done for text and images. Some key details: - Voicebox can synthesize speech across 6 languages - It's a general-purpose model that can perform tasks it wasn't trained on. It can perform noise removal, content editing, style conversion, and more - Supports in-context text-to-speech synthesis and cross-lingual style transfer - It's 20x faster than current models and outperforms single-purpose models through in-context learning paper: blog:

elvis

88,495 views • 3 years ago

😀 ToonCrafter: Generative Cartoon Interpolation 🔆 Can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Model on 🤗👇

😀 ToonCrafter: Generative Cartoon Interpolation 🔆 Can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Model on 🤗👇

Gradio

93,074 views • 2 years ago

The latent space of earlier generative models like GANS can linearly encode concepts of the data. What if the data was model weights? We present weights2weights, a subspace in diffusion weights that behaves as an interpretable latent space over customized diffusion models.

The latent space of earlier generative models like GANS can linearly encode concepts of the data. What if the data was model weights? We present weights2weights, a subspace in diffusion weights that behaves as an interpretable latent space over customized diffusion models.

Amil Dravid

94,226 views • 2 years ago

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,368 views • 2 years ago

(1/3) Can we turn text-to-image models into photorealistic 3D generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?

(1/3) Can we turn text-to-image models into photorealistic 3D generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?

Matthias Niessner

34,749 views • 2 years ago

Diffusion models have amazing image creation abilities. But how well does their generative knowledge transfer to discriminative tasks? We present Diffusion Classifier: strong classification results with pretrained conditional diffusion models, *with no additional training*! 1/9

Diffusion models have amazing image creation abilities. But how well does their generative knowledge transfer to discriminative tasks? We present Diffusion Classifier: strong classification results with pretrained conditional diffusion models, with no additional training! 1/9

Alex Li

95,304 views • 2 years ago

Introducing AI CLI Generate images, video, and text from your terminal. Pipe them together. Works with any agent. ai image "sunset" | ai video "animate" → Hundreds of models → Multi-model comparison → Inline previews, no native deps → AI SDK + AI Gateway

Introducing AI CLI Generate images, video, and text from your terminal. Pipe them together. Works with any agent. ai image "sunset" | ai video "animate" → Hundreds of models → Multi-model comparison → Inline previews, no native deps → AI SDK + AI Gateway

Chris Tate

61,232 views • 1 month ago

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Bilawal Sidhu

92,760 views • 2 years ago

Seedream 4 can place text in your images. The text can even behind an object. Here's a quick example of the potential!

Seedream 4 can place text in your images. The text can even behind an object. Here's a quick example of the potential!

Jerrod Lew

38,706 views • 8 months ago

Want to generate images from the world's best models like Nano Banana or GPT Image side by side? Now you can with Image Lab 🚀 Run a single prompt across up to 25 models, with up to 20 images from each, and see results in seconds. You've seen our leaderboards. Now generate and evaluate the models yourself. Link in thread 🧵👇

Want to generate images from the world's best models like Nano Banana or GPT Image side by side? Now you can with Image Lab 🚀 Run a single prompt across up to 25 models, with up to 20 images from each, and see results in seconds. You've seen our leaderboards. Now generate and evaluate the models yourself. Link in thread 🧵👇

Artificial Analysis

12,250 views • 3 months ago

This also shows up in the representations learned by the model. We plot the model’s representations of human and robot images. As pre-training is scaled up, the representation of humans and robots become more aligned: to a scaled-up model, human videos "look" like robot demos.

This also shows up in the representations learned by the model. We plot the model’s representations of human and robot images. As pre-training is scaled up, the representation of humans and robots become more aligned: to a scaled-up model, human videos "look" like robot demos.

Physical Intelligence

120,595 views • 5 months ago

Soon anybody can be animated everywhere! En3D can generating 3D humans either from text or images and animate them!

Soon anybody can be animated everywhere! En3D can generating 3D humans either from text or images and animate them!

Dreaming Tulpa 🥓👑

196,729 views • 2 years ago

1/ 🔥 New paper: Differentiable Vector Quantization (DiVeQ) 🔥 Vector quantization (VQ) is a key building block in modern AI. It links continuous data like images and audio to discrete representations (tokens) used by transformers.

1/ 🔥 New paper: Differentiable Vector Quantization (DiVeQ) 🔥 Vector quantization (VQ) is a key building block in modern AI. It links continuous data like images and audio to discrete representations (tokens) used by transformers.

Arno Solin

78,445 views • 1 month ago

Decentralized Diffusion Models power stronger models trained on more accessible infrastructure. DDMs mitigate the networking bottleneck that locks training into expensive and power-hungry centralized clusters. They scale gracefully to billions of parameters and generate photorealistic images with just a week of training on eight independent GPU nodes. They’re easy to implement, adopt DiT hyperparameters directly and outperform standard models FLOP-for-FLOP.

Decentralized Diffusion Models power stronger models trained on more accessible infrastructure. DDMs mitigate the networking bottleneck that locks training into expensive and power-hungry centralized clusters. They scale gracefully to billions of parameters and generate photorealistic images with just a week of training on eight independent GPU nodes. They’re easy to implement, adopt DiT hyperparameters directly and outperform standard models FLOP-for-FLOP.

David McAllister

46,378 views • 1 year ago

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

kling 3.0 is crazy... this model works very different than any other model this is from giving it just 2 images and a multi prompt then it handled all the scenes by itself... now imagine what it could do if you gave it more references and an even more detailed multi prompt going to test this a lot more but kling definetly cooked on this one

Miko

95,568 views • 4 months ago

Here's an introduction to SeaArt FILM. It can generate images from text, specializing in creating styles that look like movie scenes. And what if you could take those completed images and turn them into videos? #SeaArtAI SeaArt.Ai🐋

Here's an introduction to SeaArt FILM. It can generate images from text, specializing in creating styles that look like movie scenes. And what if you could take those completed images and turn them into videos? #SeaArtAI SeaArt.Ai🐋

Baxtable

15,328 views • 1 year ago

Adobe just released edit text in images in Firefly Boards 🤯 You can select text of different complexity that is part of your image and edit it with AI. Best to use similar length of font. Isn't this absolutely wild?

Adobe just released edit text in images in Firefly Boards 🤯 You can select text of different complexity that is part of your image and edit it with AI. Best to use similar length of font. Isn't this absolutely wild?

Kris Kashtanova

63,209 views • 9 months ago