Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

byebye expensive motion tracking equipment 👋 ai makes motion capturing so easy now! nvidia presented GENMO last week, a new model that can generate and estimate human motion from text, audio, video, and 3D keyframes

Dreaming Tulpa 🥓👑

54,918 subscribers

73,693 views • 1 year ago •via X (Twitter)

Health & Wellness Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

PowerBeatsVR3 years ago

Get ready for a full-body VR workout that’s fun, fast, and intuitive — Play PowerBeatsVR (Now on Meta Quest) 🔥

Paliesk Debesį1 year ago

Is it real time? Would love to use something like this for VR chat.

Adrian Werner1 year ago

It's not going to replace expensive motion tracking for high end production because the fidelity is too low. But it is a cool stuff to have for indie studios. It's not anything new, plenty of such systems are already in use, for example inZOI has inhouse one.

NΞXUS STUDIO ⒶI1 year ago

Awesome, is it possible to generate a tracking shot of a car too?

Atiko 💎1 year ago

Wow

BLENDER SUSHI 🫶 X - 24/7 Blenderian1 year ago

Fingers typing behind clothes :)

JSFILMZ1 year ago

ai mocap been out for like 5 years

WaveSpeedAI1 year ago

Cool!

Jorge1 year ago

some people might complain about "le ai is taking le work" but actually you still gotta know dem moves I've seen people doing mocaps at home and moving really weirdly, with like 3000€ equipment

rey1 year ago

@grok buddy, wt the hell is going on, I thought this wasn't suppose to come till 2030 ,r we in the singularity already 😂

Related Videos

AI can synthesize dynamic textures on 3D meshes without UV maps! MeshNCA is a model that can generate dynamic textures from images, text prompts, and motion vector fields.

AI can synthesize dynamic textures on 3D meshes without UV maps! MeshNCA is a model that can generate dynamic textures from images, text prompts, and motion vector fields.

Dreaming Tulpa 🥓👑

79,593 views • 2 years ago

The free UE5.5 AI motion capture plugin is compatible with Unity, Blender, and VMC. It supports both real-time motion capture via webcam and video uploads to generate 3D motion data. Powered by a 1-billion-parameter motion model running locally on your computer, it requires 8GB of VRAM for real-time processing. Compatible with Nvidia, AMD, and Intel GPUs, it can leverage NVIDIA CUDA to significantly boost graphics computation efficiency.

The free UE5.5 AI motion capture plugin is compatible with Unity, Blender, and VMC. It supports both real-time motion capture via webcam and video uploads to generate 3D motion data. Powered by a 1-billion-parameter motion model running locally on your computer, it requires 8GB of VRAM for real-time processing. Compatible with Nvidia, AMD, and Intel GPUs, it can leverage NVIDIA CUDA to significantly boost graphics computation efficiency.

CYANPUPPETS

17,807 views • 1 year ago

Next-level motion from @HeyGen_Official I got to play with the new AI Motion from HeyGen, and it blew my mind! You generate an image and describe the motion; then, you can generate long videos from that avatar. Sound effects included! 1. Tales from The Heartland

Alex Patrascu

57,728 views • 1 year ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,311 views • 3 years ago

new Loopy model from ByteDance can generate whole videos of realistic face motion from just ONE IMAGE and a SOUND getting that feeling again...

new Loopy model from ByteDance can generate whole videos of realistic face motion from just ONE IMAGE and a SOUND getting that feeling again...

the real deepfates

305,897 views • 1 year ago

Introducing 3D Motion. You can now turn any image into a 3D motion video on Morphic.

Introducing 3D Motion. You can now turn any image into a 3D motion video on Morphic.

Morphic

221,760 views • 10 months ago

animators are not needed anymore this 3D AI motion capture plugin can convert character movement from real video to 3D data and.. you can apply the motion to any 3D character.. link in comments

animators are not needed anymore this 3D AI motion capture plugin can convert character movement from real video to 3D data and.. you can apply the motion to any 3D character.. link in comments

el.cine

64,486 views • 6 months ago

Want to experience humanoid behavior foundation model in your browser? SONIC Web demo now live at Featuring: - Motion tracking - Text to motion using Kimodo Try it out!

Want to experience humanoid behavior foundation model in your browser? SONIC Web demo now live at Featuring: - Motion tracking - Text to motion using Kimodo Try it out!

Zhengyi “Zen” Luo

38,270 views • 2 months ago

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Splatter a Video can turn a video into a 3D Gaussian representation! This allows for enhanced video tracking, depth prediction, motion and appearance editing, and stereoscopic video generation.

Dreaming Tulpa 🥓👑

11,572 views • 2 years ago

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,603 views • 11 months ago

SpatialTrackerV2: unified, end-to-end 3D point tracking model which simultaneously estimates Camera Motion, Consistent Geometry and Pixel-wise 3D Trajectories.

SpatialTrackerV2: unified, end-to-end 3D point tracking model which simultaneously estimates Camera Motion, Consistent Geometry and Pixel-wise 3D Trajectories.

Bilawal Sidhu

20,346 views • 11 months ago

3D artists should stop hating AI this AI Motion Plugin for Blender, unity and UE5 can converts character motion in video into 3D data.. in real time free to try, link in comment

3D artists should stop hating AI this AI Motion Plugin for Blender, unity and UE5 can converts character motion in video into 3D data.. in real time free to try, link in comment

el.cine

4,762,443 views • 7 months ago

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model abs: project page:

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model abs: project page:

AK

90,072 views • 3 years ago

AI motion capture is levelling up AGI Mocap can generate character rigging for your 3D models in real time, and you can export to 3D softwares directly

AI motion capture is levelling up AGI Mocap can generate character rigging for your 3D models in real time, and you can export to 3D softwares directly

el.cine

88,519 views • 1 year ago

Full body motion, audio driven timing, camera direction, and world control in one model. Omnia moves past stiff avatar clips and into real performance. It understands motion, sound, text, and the scene together, so what you create feels directed, not generated. Available now.

Hedra

353,191 views • 4 months ago

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

Jiaman Li

15,788 views • 1 year ago

Adobe just launched AI video generator. key features: - generate motion graphics (new in AI world) - text/image to video - Shot size control (close up, wide shot.. ) - Camera angle control (low angle, aerial shot ..) - camera motion control - keyframe link in comment

Adobe just launched AI video generator. key features: - generate motion graphics (new in AI world) - text/image to video - Shot size control (close up, wide shot.. ) - Camera angle control (low angle, aerial shot ..) - camera motion control - keyframe link in comment

el.cine

269,769 views • 1 year ago

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,548 views • 2 years ago