Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Controllability matters for video generation. We thus propose SparseCtrl that enables keyframe (sketch, depth, img) animation/transition/prediction/interpolation by feeding TEMPORALLY SPARSE condition maps. It’s also compatible with #AnimateDiff. Page:

CeyuanY's profile picture

Ceyuan Yang

1,915 subscribers

39,542 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & TechnologieBildung#AnimateDiff
LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von 歸藏(guizang.ai)
歸藏(guizang.ai)vor 2 Jahren

@camenduru I'm not entirely sure if I got this right, but it sounds like this research turns the Controlnet, which used to be active in every frame, into something that's only active in a few key frames. This enhances video control while reducing resource consumption, right?

Profilbild von k_nearest
k_nearestvor 2 Jahren

When do you think this will be up on Arxiv? I’d like to read it in my study group when it’s available.

Profilbild von Yinghao Xu
Yinghao Xuvor 2 Jahren

Amazing!

Profilbild von Jay Guthrie
Jay Guthrievor 2 Jahren

@camenduru I WISH THE CODE WAS OUT 😭😭😭😭

Profilbild von k_nearest
k_nearestvor 2 Jahren

Looks cool!

Ähnliche Videos

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.
0:36

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.

ComfyUI

108,130 Aufrufe • vor 6 Monaten

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
1:03

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation

AK

36,271 Aufrufe • vor 1 Jahr

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
1:01

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

Purz.ai

16,711 Aufrufe • vor 10 Monaten

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇
0:43

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇

fabdream.ai

272,570 Aufrufe • vor 2 Jahren

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.
0:37

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,572 Aufrufe • vor 2 Jahren

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.
0:30

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.

Rory Flynn

48,319 Aufrufe • vor 2 Monaten

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>
1:00

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>

LTX

545,190 Aufrufe • vor 11 Monaten

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.
0:20

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 Aufrufe • vor 1 Jahr

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:
1:12

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

Yuan Liu

56,547 Aufrufe • vor 1 Jahr

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:
4:29

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:

Ceyuan Yang

46,813 Aufrufe • vor 1 Jahr

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.
0:36

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.

Alexandre Morgand

11,804 Aufrufe • vor 3 Monaten

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖
0:36

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖

Xiaochuang Han

31,311 Aufrufe • vor 6 Monaten

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.
2:40

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.

Dreaming Tulpa 🥓👑

18,045 Aufrufe • vor 3 Jahren

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation
0:21

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation

Samuel Spitz

11,544 Aufrufe • vor 3 Monaten

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation
2:07

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation

AK

50,115 Aufrufe • vor 1 Jahr

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:
0:20

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:

Xingang Pan

28,248 Aufrufe • vor 1 Jahr

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image
0:28

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image

AK

59,229 Aufrufe • vor 1 Jahr