正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Meta presents MarDini Masked Autoregressive Diffusion for Video Generation at Scale

AK

524,814 subscribers

70,089 次观看 • 2 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

NCMs are pretty fast! See the language generation speed vs. autoregressive and diffusion baselines below.

NCMs are pretty fast! See the language generation speed vs. autoregressive and diffusion baselines below.

Azalia Mirhoseini

10,604 次观看 • 10 天前

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7

Marianne Arriola

150,803 次观看 • 1 年前

Meta presents Exploring MLLM-Diffusion Information Transfer with MetaCanvas

Meta presents Exploring MLLM-Diffusion Information Transfer with MetaCanvas

AK

22,237 次观看 • 10 个月前

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models "Block diffusion sequentially generates blocks of tokens by performing diffusion within each block and conditioning on previous blocks. By combining strength from autoregressive and diffusion models, block diffusion overcomes the limitations of both approaches by supporting variable-length, higher-quality generation and improving inference efficiency with KV caching and parallel sampling."

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models "Block diffusion sequentially generates blocks of tokens by performing diffusion within each block and conditioning on previous blocks. By combining strength from autoregressive and diffusion models, block diffusion overcomes the limitations of both approaches by supporting variable-length, higher-quality generation and improving inference efficiency with KV caching and parallel sampling."

Tanishq Mathew Abraham, Ph.D.

21,813 次观看 • 1 年前

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,471 次观看 • 2 年前

Snap presents MoA Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts

Snap presents MoA Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts

AK

47,488 次观看 • 2 年前

This symmetric diffusion paper at ICLR is nice (simple idea in retrospect): SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models We'd actually implemented this idea internally at Orbital, and it works nicely even for very large crystal structures:

This symmetric diffusion paper at ICLR is nice (simple idea in retrospect): SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models We'd actually implemented this idea internally at Orbital, and it works nicely even for very large crystal structures:

Mark Neumann

18,200 次观看 • 1 年前

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

AK

124,085 次观看 • 2 年前

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,062 次观看 • 1 年前

High-resolution image and video generation is hitting a wall because attention in DiTs scales quadratically with token count. But does every pixel need to be in full resolution? Introducing Foveated Diffusion: a new approach for efficient diffusion-based generation that allocates compute where it matters most. 1/7🧵

High-resolution image and video generation is hitting a wall because attention in DiTs scales quadratically with token count. But does every pixel need to be in full resolution? Introducing Foveated Diffusion: a new approach for efficient diffusion-based generation that allocates compute where it matters most. 1/7🧵

Gordon Wetzstein

172,039 次观看 • 7 个月前

Introducing Diffusion Forcing, which unifies next-token prediction (eg LLMs) and full-seq. diffusion (eg SORA)! It offers improved performance & new sampling strategies in vision and robotics, such as stable, infinite video generation, better diffusion planning, and more! (1/8)

Introducing Diffusion Forcing, which unifies next-token prediction (eg LLMs) and full-seq. diffusion (eg SORA)! It offers improved performance & new sampling strategies in vision and robotics, such as stable, infinite video generation, better diffusion planning, and more! (1/8)

Boyuan Chen

208,952 次观看 • 2 年前

Meta announces SceneScript Reconstructing Scenes With An Autoregressive Structured Language Model We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our

Meta announces SceneScript Reconstructing Scenes With An Autoregressive Structured Language Model We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our

AK

46,099 次观看 • 2 年前

Seedance 2.0 is going to be a game changer for video creation. The powerful video generation tool empowers creativity and drives efficiency. Built for social content, product ads, and marketing at scale. Get 60% OFF on Pollo AI! Time is running out. Don't miss it!

Seedance 2.0 is going to be a game changer for video creation. The powerful video generation tool empowers creativity and drives efficiency. Built for social content, product ads, and marketing at scale. Get 60% OFF on Pollo AI! Time is running out. Don't miss it!

Pollo AI

16,533,552 次观看 • 6 个月前

The Stable Video Diffusion model just dropped 🔥 The new model supports: – Text-to-Video – Image-to-Video – 14 or 25 frames at 576 x 1024 – Multi-View Generation – Frame Interpolation – 3D Scene Understanding – Camera Control via LoRA Paper: Code: SVD model: SVD-XT model:

The Stable Video Diffusion model just dropped 🔥 The new model supports: – Text-to-Video – Image-to-Video – 14 or 25 frames at 576 x 1024 – Multi-View Generation – Frame Interpolation – 3D Scene Understanding – Camera Control via LoRA Paper: Code: SVD model: SVD-XT model:

Dreaming Tulpa 🥓👑

457,887 次观看 • 2 年前

this looks insane, LLaDA app just dropped by apolinario (poli) A text generation method different from the traditional left-to-right approach a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance

this looks insane, LLaDA app just dropped by apolinario (poli) A text generation method different from the traditional left-to-right approach a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance

AK

26,694 次观看 • 1 年前

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Akshay 🚀

64,011 次观看 • 11 个月前

Introducing Digit 5. Agility’s next-generation humanoid, engineered for cooperatively safe work at scale, allowing it to work in close proximity to people without the physical safety barriers required by traditional automation. Explore Digit 5: Watch the full video:

Introducing Digit 5. Agility’s next-generation humanoid, engineered for cooperatively safe work at scale, allowing it to work in close proximity to people without the physical safety barriers required by traditional automation. Explore Digit 5: Watch the full video:

Agility

474,454 次观看 • 1 个月前