Loading video...

Video Failed to Load

Go Home

Meta presents MarDini Masked Autoregressive Diffusion for Video Generation at Scale

_akhaliq's profile picture

AK

517,220 subscribers

70,073 views • 1 year ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7
0:17

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7

Marianne Arriola @ICML

150,079 views • 1 year ago

Meta presents Exploring MLLM-Diffusion Information Transfer with MetaCanvas
0:10

Meta presents Exploring MLLM-Diffusion Information Transfer with MetaCanvas

AK

22,237 views • 8 months ago

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models "Block diffusion sequentially generates blocks of tokens by performing diffusion within each block and conditioning on previous blocks. By combining strength from autoregressive and diffusion models, block diffusion overcomes the limitations of both approaches by supporting variable-length, higher-quality generation and improving inference efficiency with KV caching and parallel sampling."
0:19

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models "Block diffusion sequentially generates blocks of tokens by performing diffusion within each block and conditioning on previous blocks. By combining strength from autoregressive and diffusion models, block diffusion overcomes the limitations of both approaches by supporting variable-length, higher-quality generation and improving inference efficiency with KV caching and parallel sampling."

Tanishq Mathew Abraham, Ph.D.

21,813 views • 1 year ago

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.
0:19

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,449 views • 2 years ago

Snap presents MoA Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts
0:11

Snap presents MoA Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts

AK

47,488 views • 2 years ago

This symmetric diffusion paper at ICLR is nice (simple idea in retrospect): SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models We'd actually implemented this idea internally at Orbital, and it works nicely even for very large crystal structures:
0:16

This symmetric diffusion paper at ICLR is nice (simple idea in retrospect): SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models We'd actually implemented this idea internally at Orbital, and it works nicely even for very large crystal structures:

Mark Neumann

18,188 views • 1 year ago

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.
0:13

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

AK

124,048 views • 2 years ago

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,062 views • 1 year ago

High-resolution image and video generation is hitting a wall because attention in DiTs scales quadratically with token count. But does every pixel need to be in full resolution? Introducing Foveated Diffusion: a new approach for efficient diffusion-based generation that allocates compute where it matters most. 1/7🧵
0:17

High-resolution image and video generation is hitting a wall because attention in DiTs scales quadratically with token count. But does every pixel need to be in full resolution? Introducing Foveated Diffusion: a new approach for efficient diffusion-based generation that allocates compute where it matters most. 1/7🧵

Gordon Wetzstein

164,415 views • 4 months ago

Introducing Diffusion Forcing, which unifies next-token prediction (eg LLMs) and full-seq. diffusion (eg SORA)! It offers improved performance & new sampling strategies in vision and robotics, such as stable, infinite video generation, better diffusion planning, and more! (1/8)
0:17

Introducing Diffusion Forcing, which unifies next-token prediction (eg LLMs) and full-seq. diffusion (eg SORA)! It offers improved performance & new sampling strategies in vision and robotics, such as stable, infinite video generation, better diffusion planning, and more! (1/8)

Boyuan Chen

208,757 views • 2 years ago

Seedance 2.0 is going to be a game changer for video creation. The powerful video generation tool empowers creativity and drives efficiency. Built for social content, product ads, and marketing at scale. Get 60% OFF on Pollo AI! Time is running out. Don't miss it!
0:14

Seedance 2.0 is going to be a game changer for video creation. The powerful video generation tool empowers creativity and drives efficiency. Built for social content, product ads, and marketing at scale. Get 60% OFF on Pollo AI! Time is running out. Don't miss it!

Pollo AI

16,533,552 views • 4 months ago

this looks insane, LLaDA app just dropped by apolinario (poli) A text generation method different from the traditional left-to-right approach a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance
0:17

this looks insane, LLaDA app just dropped by apolinario (poli) A text generation method different from the traditional left-to-right approach a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance

AK

26,694 views • 1 year ago

The Stable Video Diffusion model just dropped 🔥 The new model supports: – Text-to-Video – Image-to-Video – 14 or 25 frames at 576 x 1024 – Multi-View Generation – Frame Interpolation – 3D Scene Understanding – Camera Control via LoRA Paper: Code: SVD model: SVD-XT model:
0:10

The Stable Video Diffusion model just dropped 🔥 The new model supports: – Text-to-Video – Image-to-Video – 14 or 25 frames at 576 x 1024 – Multi-View Generation – Frame Interpolation – 3D Scene Understanding – Camera Control via LoRA Paper: Code: SVD model: SVD-XT model:

Dreaming Tulpa 🥓👑

457,724 views • 2 years ago

Today we released the code for our CVPR 2026 paper, Flowception. Flowception bridges fully bidirectional sequence modeling and autoregressive generation by inserting frames via learned order, then denoising them with continuous flow. Website: Code:
0:10

Today we released the code for our CVPR 2026 paper, Flowception. Flowception bridges fully bidirectional sequence modeling and autoregressive generation by inserting frames via learned order, then denoising them with continuous flow. Website: Code:

John Nguyen @ ICML

18,980 views • 3 months ago

Selected as a best paper finalist at #CVPR2026: PixelDiT from NVIDIA Research In most image generation models, a pretrained autoencoder compresses the image before any diffusion happens, causing quality loss that accumulates across the entire pipeline. PixelDiT, or Pixel Diffusion Transformers, removes this step entirely. It's a single-stage model that learns the diffusion process directly in pixel space, end-to-end.
0:18

Selected as a best paper finalist at #CVPR2026: PixelDiT from NVIDIA Research In most image generation models, a pretrained autoencoder compresses the image before any diffusion happens, causing quality loss that accumulates across the entire pipeline. PixelDiT, or Pixel Diffusion Transformers, removes this step entirely. It's a single-stage model that learns the diffusion process directly in pixel space, end-to-end.

NVIDIA AI

28,275 views • 2 months ago

Wow. AI-generated animation from a video. This experiment combines video of dance choreography by Erica Klein and hok with AnimateDiff and a text prompt. #AnimateDiff feels like the missing link in diffusion video generation -no flickering! More soon. #stablediffusion
0:18

Wow. AI-generated animation from a video. This experiment combines video of dance choreography by Erica Klein and hok with AnimateDiff and a text prompt. #AnimateDiff feels like the missing link in diffusion video generation -no flickering! More soon. #stablediffusion

Nathan Shipley

172,681 views • 2 years ago

VEED's AI Video API is LIVE on n8n.io ⚡️ Create high-impact video creation at scale • Connect VEED with 1,000+ apps on n8n • Build end-to-end AI video workflows • Power any video use case at scale 🚨 Get 3 FREE templates with access RT + comment “VEED” for link in DM
0:13

VEED's AI Video API is LIVE on n8n.io ⚡️ Create high-impact video creation at scale • Connect VEED with 1,000+ apps on n8n • Build end-to-end AI video workflows • Power any video use case at scale 🚨 Get 3 FREE templates with access RT + comment “VEED” for link in DM

VEED | AI Video Creation

24,025 views • 6 months ago