Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Controllability matters for video generation. We thus propose SparseCtrl that enables keyframe (sketch, depth, img) animation/transition/prediction/interpolation by feeding TEMPORALLY SPARSE condition maps. It’s also compatible with #AnimateDiff. Page:

CeyuanY's profile picture

Ceyuan Yang

1,915 subscribers

39,542 görüntüleme • 2 yıl önce •via X (Twitter)

Bilim & TeknolojiEğitim#AnimateDiff
LIVE

Anya Rossi• Live Now

Private livecam show

5 Yorum

歸藏(guizang.ai) profil fotoğrafı
歸藏(guizang.ai)2 yıl önce

@camenduru I'm not entirely sure if I got this right, but it sounds like this research turns the Controlnet, which used to be active in every frame, into something that's only active in a few key frames. This enhances video control while reducing resource consumption, right?

k_nearest profil fotoğrafı
k_nearest2 yıl önce

When do you think this will be up on Arxiv? I’d like to read it in my study group when it’s available.

Yinghao Xu profil fotoğrafı
Yinghao Xu2 yıl önce

Amazing!

Jay Guthrie profil fotoğrafı
Jay Guthrie2 yıl önce

@camenduru I WISH THE CODE WAS OUT 😭😭😭😭

k_nearest profil fotoğrafı
k_nearest2 yıl önce

Looks cool!

Benzer Videolar

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.
0:36

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.

ComfyUI

108,130 görüntüleme • 6 ay önce

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
1:01

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

Purz.ai

16,711 görüntüleme • 10 ay önce

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
1:03

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation

AK

36,271 görüntüleme • 1 yıl önce

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇
0:43

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇

fabdream.ai

272,624 görüntüleme • 2 yıl önce

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 görüntüleme • 1 yıl önce

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.
0:37

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,575 görüntüleme • 2 yıl önce

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.
0:30

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.

Rory Flynn

48,371 görüntüleme • 2 ay önce

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>
1:00

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>

LTX

545,190 görüntüleme • 1 yıl önce

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:
1:12

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

Yuan Liu

56,547 görüntüleme • 1 yıl önce

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:
4:29

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:

Ceyuan Yang

46,813 görüntüleme • 1 yıl önce

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.
0:20

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 görüntüleme • 1 yıl önce

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.
0:36

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.

Alexandre Morgand

11,804 görüntüleme • 3 ay önce

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖
0:36

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖

Xiaochuang Han

31,311 görüntüleme • 7 ay önce

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.
2:40

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.

Dreaming Tulpa 🥓👑

18,045 görüntüleme • 3 yıl önce

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation
0:21

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation

Samuel Spitz

11,557 görüntüleme • 4 ay önce

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:
0:20

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:

Xingang Pan

28,248 görüntüleme • 1 yıl önce

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation
2:07

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation

AK

50,267 görüntüleme • 1 yıl önce

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image
0:28

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image

AK

59,229 görüntüleme • 1 yıl önce