Loading video...

Video Failed to Load

Go Home

Controllability matters for video generation. We thus propose SparseCtrl that enables keyframe (sketch, depth, img) animation/transition/prediction/interpolation by feeding TEMPORALLY SPARSE condition maps. It’s also compatible with #AnimateDiff. Page:

CeyuanY's profile picture

Ceyuan Yang

1,915 subscribers

39,542 views • 2 years ago •via X (Twitter)

Science & TechnologyEducation#AnimateDiff
LIVE

Anya Rossi• Live Now

Private livecam show

5 Comments

歸藏(guizang.ai)'s profile picture
歸藏(guizang.ai)2 years ago

@camenduru I'm not entirely sure if I got this right, but it sounds like this research turns the Controlnet, which used to be active in every frame, into something that's only active in a few key frames. This enhances video control while reducing resource consumption, right?

k_nearest's profile picture
k_nearest2 years ago

When do you think this will be up on Arxiv? I’d like to read it in my study group when it’s available.

Yinghao Xu's profile picture
Yinghao Xu2 years ago

Amazing!

Jay Guthrie's profile picture
Jay Guthrie2 years ago

@camenduru I WISH THE CODE WAS OUT 😭😭😭😭

k_nearest's profile picture
k_nearest2 years ago

Looks cool!

Related Videos

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.
0:36

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.

ComfyUI

108,107 views • 5 months ago

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
1:03

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation

AK

36,271 views • 11 months ago

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
1:01

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

Purz.ai

16,711 views • 9 months ago

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇
0:43

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇

fabdream.ai

272,557 views • 2 years ago

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.
0:37

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,547 views • 2 years ago

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.
0:30

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.

Rory Flynn

47,822 views • 1 month ago

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>
1:00

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>

LTX

545,190 views • 11 months ago

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:
1:12

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

Yuan Liu

56,547 views • 1 year ago

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.
0:20

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 views • 11 months ago

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:
4:29

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:

Ceyuan Yang

46,813 views • 1 year ago

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.
0:36

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.

Alexandre Morgand

11,804 views • 2 months ago

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖
0:36

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖

Xiaochuang Han

28,925 views • 6 months ago

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.
2:40

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.

Dreaming Tulpa 🥓👑

18,045 views • 3 years ago

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation
0:21

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation

Samuel Spitz

11,544 views • 3 months ago

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation
2:07

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation

AK

50,115 views • 1 year ago

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image
0:28

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image

AK

59,173 views • 1 year ago

Learning Temporally Consistent Video Depth from Video Diffusion Priors This work addresses the challenge of video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. Instead of directly
1:25

Learning Temporally Consistent Video Depth from Video Diffusion Priors This work addresses the challenge of video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. Instead of directly

AK

81,634 views • 2 years ago