Loading video...

Video Failed to Load

Go Home

Introducing Ctrl-VI, a video sampling method allowing for a flexible set of user controls—ranging from coarse but easy-to-specify text prompts to precise camera/object trajectories. (1/n)

zhang_yunzhi's profile picture

Yunzhi Zhang

2,543 subscribers

42,591 views • 7 months ago •via X (Twitter)

ArtsEducationScience & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Stable Virtual Camera: Generative View Synthesis with Diffusion Models Hint: Check the project website. It is awesome! Contributions: 1. A training strategy for jointly modeling large viewpoint changes and temporal smoothness. 2. A two-pass procedural sampling method for smooth video generation along arbitrarily long camera trajectories. 3. A comprehensive benchmark that evaluates NVS methods across different datasets and settings. 4. An open-source release of model weights to support future research.
0:24

Stable Virtual Camera: Generative View Synthesis with Diffusion Models Hint: Check the project website. It is awesome! Contributions: 1. A training strategy for jointly modeling large viewpoint changes and temporal smoothness. 2. A two-pass procedural sampling method for smooth video generation along arbitrarily long camera trajectories. 3. A comprehensive benchmark that evaluates NVS methods across different datasets and settings. 4. An open-source release of model weights to support future research.

MrNeRF

38,619 views • 1 year ago

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.
0:21

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,548 views • 2 years ago

A viral video has surfaced showing a man using a metal object to take cashew nuts from stored bags, sparking debate online. Many have questioned whether he was conducting a legitimate sampling process and if the method used was appropriate.
0:43

A viral video has surfaced showing a man using a metal object to take cashew nuts from stored bags, sparking debate online. Many have questioned whether he was conducting a legitimate sampling process and if the method used was appropriate.

THE STATE NEWS

104,017 views • 2 months ago

Introducing Poe Apps: a new, easy way to create and use visual interfaces into any combination of the 100+ text, image, video, and audio models on Poe. (1/5)
0:54

Introducing Poe Apps: a new, easy way to create and use visual interfaces into any combination of the 100+ text, image, video, and audio models on Poe. (1/5)

Poe

66,708 views • 1 year ago

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:
0:46

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

Yuan Liu

33,363 views • 1 year ago

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇
0:48

1/N Most Vision-Language-Action models need tons of data for finetuning, and still fail for new objects and instructions. Introducing OTTER, a lightweight, easy-to-train model that uses text-aware visual features to nail unseen tasks out of the box! Here's how it works 👇

Fangchen Liu

68,288 views • 1 year ago

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:
1:00

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Yu Xiang

57,228 views • 1 year ago

New Studio camera controls are now in beta, introducing a new Locking functionality for camera speed as well as new modes of camera movement.
0:26

New Studio camera controls are now in beta, introducing a new Locking functionality for camera speed as well as new modes of camera movement.

Bloxy News

11,589 views • 1 year ago

🎨 KLING AI 1.6 PROMPTS THREAD 🎨 Here for you a new thread of prompts for Kling AI 1.6 model text to video. You can use them as ideas or base prompts. All details in the thread , bookmark for later👇 1/18
1:30

🎨 KLING AI 1.6 PROMPTS THREAD 🎨 Here for you a new thread of prompts for Kling AI 1.6 model text to video. You can use them as ideas or base prompts. All details in the thread , bookmark for later👇 1/18

LudovicCreator

24,736 views • 1 year ago

enigmatic_e shows how our models fill a gap in his creative workflow, making it easy to stage characters within scenes and direct precise camera movements. 8/n
1:55

enigmatic_e shows how our models fill a gap in his creative workflow, making it easy to stage characters within scenes and direct precise camera movements. 8/n

World Labs

34,987 views • 1 year ago

"near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence [...] a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation"
0:53

"near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence [...] a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consistent 3D representation"

Fabien Benetou

222,906 views • 3 years ago

🎨 KLING AI 1.6 POV PROMPTS THREAD 🎨 Here for you a new thread of prompts for Kling AI text to video. You can use them as ideas or base prompts. Sound on : Freepik All details in the thread , bookmark for later👇 1/25
2:05

🎨 KLING AI 1.6 POV PROMPTS THREAD 🎨 Here for you a new thread of prompts for Kling AI text to video. You can use them as ideas or base prompts. Sound on : Freepik All details in the thread , bookmark for later👇 1/25

LudovicCreator

58,020 views • 1 year ago

Introducing FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes! Our system merges a library of skills with foundation models for efficient robotic feeding, tailored to user preferences. 🔗 📃 To Appear at RSS ’24 1/N
0:41

Introducing FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes! Our system merges a library of skills with foundation models for efficient robotic feeding, tailored to user preferences. 🔗 📃 To Appear at RSS ’24 1/N

Priya Sundaresan

23,080 views • 1 year ago

Meta just announced Movie Gen. Movie Gen generates high-quality videos from text prompts, lifelike audio synced to video, precise video edits, and personalized videos featuring someone of your choice from an uploaded image. Here are some examples of what it can do:
1:13

Meta just announced Movie Gen. Movie Gen generates high-quality videos from text prompts, lifelike audio synced to video, precise video edits, and personalized videos featuring someone of your choice from an uploaded image. Here are some examples of what it can do:

CHRIS FIRST

3,823,673 views • 1 year ago

New VST/AU Plugin! 🚨 Play with Lyria RealTime directly from inside your favorite DAW with “The Infinite Crate” 🎧🎶 Like other Lyria RT demos, you can mix together text prompts and other controls to steer the model in real-time. But now with a VST plugin you can feed audio directly into your DAW for sampling, live performance, or even a practice partner to jam with. 💾 Get it here:
0:20

New VST/AU Plugin! 🚨 Play with Lyria RealTime directly from inside your favorite DAW with “The Infinite Crate” 🎧🎶 Like other Lyria RT demos, you can mix together text prompts and other controls to steer the model in real-time. But now with a VST plugin you can feed audio directly into your DAW for sampling, live performance, or even a practice partner to jam with. 💾 Get it here:

Jesse Engel

59,834 views • 11 months ago

Introducing MULTI-FRAMES — a major step forward in controllable video generation. Define up to 10 keyframes and generate a coherent one-take sequence up to ~54 seconds. Transitions can be set from 1–6 seconds, with prompts to control how each scene evolves between frames. Check our showcase video below👇 #DreaminaMultiFrames
0:58

Introducing MULTI-FRAMES — a major step forward in controllable video generation. Define up to 10 keyframes and generate a coherent one-take sequence up to ~54 seconds. Transitions can be set from 1–6 seconds, with prompts to control how each scene evolves between frames. Check our showcase video below👇 #DreaminaMultiFrames

Dreamina AI

139,742 views • 6 months ago

Now, camera work isn’t limited by physics. Introducing Higgsfield Mix - combine multiple motion controls in a single shot, including moves that aren’t possible with real cameras. Also dropping 10 new motion controls built for speed, tension, and cinematic impact. 🧩 1/n
0:53

Now, camera work isn’t limited by physics. Introducing Higgsfield Mix - combine multiple motion controls in a single shot, including moves that aren’t possible with real cameras. Also dropping 10 new motion controls built for speed, tension, and cinematic impact. 🧩 1/n

Higgsfield AI 🧩

650,559 views • 1 year ago