Загрузка видео...

Не удалось загрузить видео

На главную

Controllability matters for video generation. We thus propose SparseCtrl that enables keyframe (sketch, depth, img) animation/transition/prediction/interpolation by feeding TEMPORALLY SPARSE condition maps. It’s also compatible with #AnimateDiff. Page:

CeyuanY's profile picture

Ceyuan Yang

1,915 subscribers

39,542 просмотров • 2 лет назад •via X (Twitter)

Наука и технологииОбразование#AnimateDiff
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 5

Фото профиля 歸藏(guizang.ai)
歸藏(guizang.ai)2 лет назад

@camenduru I'm not entirely sure if I got this right, but it sounds like this research turns the Controlnet, which used to be active in every frame, into something that's only active in a few key frames. This enhances video control while reducing resource consumption, right?

Фото профиля k_nearest
k_nearest2 лет назад

When do you think this will be up on Arxiv? I’d like to read it in my study group when it’s available.

Фото профиля Yinghao Xu
Yinghao Xu2 лет назад

Amazing!

Фото профиля Jay Guthrie
Jay Guthrie2 лет назад

@camenduru I WISH THE CODE WAS OUT 😭😭😭😭

Фото профиля k_nearest
k_nearest2 лет назад

Looks cool!

Похожие видео

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.
0:36

LTX-2 is natively supported in ComfyUI on Day 0 🎬🔊 The next chapter in controllability for open-source video generation. - Open-source audio-video foundation model - Generates motion, dialogue, SFX, and music together - Canny, Depth & Pose video-to-video control - Keyframe-driven generation - Native upscaling and prompt enhancement Everything in a single pass.

ComfyUI

108,130 просмотров • 6 месяцев назад

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
1:01

InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

Purz.ai

16,711 просмотров • 10 месяцев назад

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation
1:03

Radial Attention O(nlog n) Sparse Attention with Energy Decay for Long Video Generation

AK

36,271 просмотров • 1 год назад

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇
0:43

I was approached by the Founder of The Future of Money event in Toronto to create a unique 40 sec AI-video for their home page banner, illustrating the dynamic evolution of money. Made with new interpolation techniques in #AnimateDiff Check it out on their site 👇

fabdream.ai

272,624 просмотров • 2 лет назад

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 просмотров • 1 год назад

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.
0:37

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,575 просмотров • 2 лет назад

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.
0:30

GPT2 + Seedance 2.0 → Yoga Flow. Controlling pose transition. (Prompt + process in thread) PROCESS: 01. GPT2 Img: Create pose sequence in 02. GPT2 Img: Create base character 03. SD2 Omni-Ref: Animating scene 04. Codex for stitching + frame interpolation While not perfect step-to-step... Still impressive.

Rory Flynn

48,422 просмотров • 2 месяцев назад

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>
1:00

60-second AI video generation just got unlocked! LTXV is the first model to generate native long-form video, with controllability that beats every open source model. - 8× longer than typical gen video - 10–100× faster & cheaper - Runs even on consumer GPUs - Pose, depth & control LoRAs supported Available now. Go try it >>

LTX

545,190 просмотров • 1 год назад

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:
1:12

I'm excited to share our new work Align3R that estimates camera poses and consistent depth maps from a monocular video of a dynamic scene. Project page: Code: Paper:

Yuan Liu

56,547 просмотров • 1 год назад

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:
4:29

We propose Long Context Tuning (LCT) for scene-level video generation to bridge the gap between current single-shot generation and real-world narrative video productions. Homepage: Report:

Ceyuan Yang

46,842 просмотров • 1 год назад

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.
0:20

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 просмотров • 1 год назад

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.
0:36

"DVD: Dynamic Video Depth" TL;DR: Recovers temporally consistent depth from monocular videos using diffusion priors + geometric constraints, handling dynamic scenes and motion robustly.

Alexandre Morgand

11,823 просмотров • 3 месяцев назад

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖
0:36

Can we simplify video generation by decomposing it into interleaved text-video co-generation? Would explicit, repeated thinking in language improve generation in pixels? We introduce TV2TV: a unified model that jointly learns - language modeling (next-token prediction) - video flow matching (next-frame prediction) At inference, TV2TV dynamically alternates between textual thinking and video generation. Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport. 📖

Xiaochuang Han

31,311 просмотров • 7 месяцев назад

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.
2:40

VideoComposer brings ControlNet guidance to text and video-to-video. The model enables to combine multiple modalities like text, sketch, style and even motion to drive video generation. The results look extremely good.

Dreaming Tulpa 🥓👑

18,045 просмотров • 3 лет назад

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation
0:21

With Agent 4, we also vastly improved Replit Animation Here's a launch video for Agent 4, built 100% in Replit Animation

Samuel Spitz

11,557 просмотров • 4 месяцев назад

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:
0:20

Introducing 𝐒𝐀𝐑𝟑𝐃, which tokenizes 3D objects into multiscale tokens and generates 3D objects by autoregressive next-scale prediction. 𝐒𝐀𝐑𝟑𝐃 enables fast 3D generation and comprehensive 3D understanding. arXiv: Project:

Xingang Pan

28,248 просмотров • 1 год назад

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation
2:07

Any2Caption Interpreting Any Condition to Caption for Controllable Video Generation

AK

50,267 просмотров • 1 год назад

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image
0:28

Direct3D-S2 Gigascale 3D Generation Made Easy with Spatial Sparse Attention high resolution 3D generation from image

AK

59,247 просмотров • 1 год назад