Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Story-to-Motion: Synthesizing Infinite and Controllable Character Animation from Long Text Outperforms previous SotA motion synthesis methods across the board proj: abs:

Aran Komatsuzaki

177,913 subscribers

38,401 views • 2 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Comments

Pseudonym 🦅2 years ago

Looks like we have all the pieces now to have a LLM assemble a 3d universe. Motion, physics, models, textures, voice, and endless content all from natural language. Surely someone is working on this? If not someone mail me large crates of cash 😂

kache2 years ago

weits

Chef Wang2 years ago

Astonishing

Emma Catnip2 years ago

✨👀✨

Sourav Kumar Bose2 years ago

🤯 AI-generated animated films are almost here!

Adrian Perdjon1 year ago

RigPlay dataset might be perfect for your development. Learn more at [email protected]

Related Videos

Loopy Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency paper page: With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

AK

128,852 views • 1 year ago

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

SceneScape: Text-Driven Consistent Scene Generation abs: project page: text-driven perpetual view generation -- synthesizing long videos of arbitrary scenes solely from an input text describing the scene and camera poses

AK

73,258 views • 3 years ago

$Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.$

Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.

AK

93,740 views • 3 years ago

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

📢📢📢 Excited to share our new work Autonomous Character-Scene Interaction Synthesis from Text Instruction (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Siyuan Huang

11,451 views • 1 year ago

Got five papers accepted by #ECCV2024 European Conference on Computer Vision #ECCV2026 ! Huge thanks to all my collaborators! 😃 See you in Milan 🇮🇹 Summary of Selected Works (I made a fast-forward for them 😄) - [Shape Generation] Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models, ECCV 2024. - [Efficient Motion Generation] EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, ECCV 2024. - [Controllable Motion Generation] TLControl: Trajectory and Language Control for Human Motion Synthesis, ECCV 2024. - [Avatar Generation] Disentangled Clothed Avatar Generation from Text Descriptions, ECCV 2024. Project Page: Surf-D: EMDM: TLControl: SOSMPL:

Zhiyang (Frank) Dou

18,187 views • 2 years ago

Introducing Motion Tokens. Data-driven animation. Every property controlled at runtime. Update text, colors, gradients and transforms — all from code.

Introducing Motion Tokens. Data-driven animation. Every property controlled at runtime. Update text, colors, gradients and transforms — all from code.

LottieFiles

60,459 views • 5 months ago

animators are not needed anymore this 3D AI motion capture plugin can convert character movement from real video to 3D data and.. you can apply the motion to any 3D character.. link in comments

animators are not needed anymore this 3D AI motion capture plugin can convert character movement from real video to 3D data and.. you can apply the motion to any 3D character.. link in comments

el.cine

64,585 views • 8 months ago

Next is "Eunha" death motion and motion editing. Please write down the character you want to see! #ryona #リョナ

Next is "Eunha" death motion and motion editing. Please write down the character you want to see! #ryona #リョナ

Ferret

12,805 views • 1 year ago

Next-level motion from @HeyGen_Official I got to play with the new AI Motion from HeyGen, and it blew my mind! You generate an image and describe the motion; then, you can generate long videos from that avatar. Sound effects included! 1. Tales from The Heartland

Alex Patrascu

57,728 views • 1 year ago

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

Tencent Hy

328,874 views • 7 months ago

Interested in upgrading your Motion Matching system? Check out our #SIGGRAPH2024 work where we introduce *Codebook Matching* to train neural character controllers for synthesizing lifelike embodied animations in #Games and #VR! Project Page:

Interested in upgrading your Motion Matching system? Check out our #SIGGRAPH2024 work where we introduce Codebook Matching to train neural character controllers for synthesizing lifelike embodied animations in #Games and #VR! Project Page:

Sebastian Starke

18,659 views • 2 years ago

Warudo Features - Motion Recorder & Motion Player 🌠The simple and universal motion-recording function allows you to play your own actions and dances at any time!🎇 🌟Play multiple Character Motions at the same time 💃Motion data can be applied to any character model ✂️ Freely trim Motion to a suitable timing 💖You can even export the recorded motion to Unity Animation (.anim) for various editing and use!😋 🌻For more information, watch this tutorial!🌻 白夜 Tira 夢宮ヨユ ⚡ SLiMe VTuber / Warudo 🎵BGM: ASMRZ(TANAKA, NEEDMORECASH) - 晚安小姐(prod. Gwana) & フォンコール #Warudo #VTuber #3D #LiveStream #3DLive #Unity #mocap #motion

Warudo Features - Motion Recorder & Motion Player 🌠The simple and universal motion-recording function allows you to play your own actions and dances at any time!🎇 🌟Play multiple Character Motions at the same time 💃Motion data can be applied to any character model ✂️ Freely trim Motion to a suitable timing 💖You can even export the recorded motion to Unity Animation (.anim) for various editing and use!😋 🌻For more information, watch this tutorial!🌻 白夜 Tira 夢宮ヨユ ⚡ SLiMe VTuber / Warudo 🎵BGM: ASMRZ(TANAKA, NEEDMORECASH) - 晚安小姐(prod. Gwana) & フォンコール #Warudo #VTuber #3D #LiveStream #3DLive #Unity #mocap #motion

Warudo @ Hakuya Labs

48,361 views • 1 year ago

I modified "Story Framework" (from Fab) to include MetaHuman face and body animations to NPCs! The face animation is Audio Driven Anim and the body animation is from Narrative Motion. I'm going to make a short narrative game using this framework and put all of the MOCAP source animation online.

I modified "Story Framework" (from Fab) to include MetaHuman face and body animations to NPCs! The face animation is Audio Driven Anim and the body animation is from Narrative Motion. I'm going to make a short narrative game using this framework and put all of the MOCAP source animation online.

Matt Workman

21,142 views • 11 months ago

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

149,079 views • 1 year ago

Gemini Omni Flash is live on Arcads. - Infinite world knowledge - top-tier editing - real motion design and text that stays crisp across every frame. This is the Nano Banana moment for AI video. Also available on Claude via the Arcads MCP.

Gemini Omni Flash is live on Arcads. - Infinite world knowledge - top-tier editing - real motion design and text that stays crisp across every frame. This is the Nano Banana moment for AI video. Also available on Claude via the Arcads MCP.

arcads AI

197,065 views • 1 month ago

🚨 New awesome feature announcement! 🚨 MOTION BLENDING IS HERE! 🚀✨ Blend any motion from our platform - whether it's from video or text - with seamless, AI-driven transitions. From bold moves to subtle shifts, your animations have never looked smoother! #SMPL #3D #Animation #genAI

🚨 New awesome feature announcement! 🚨 MOTION BLENDING IS HERE! 🚀✨ Blend any motion from our platform - whether it's from video or text - with seamless, AI-driven transitions. From bold moves to subtle shifts, your animations have never looked smoother! #SMPL #3D #Animation #genAI

Meshcapade

51,705 views • 1 year ago

Champ Controllable and Consistent Human Image Animation with 3D Parametric Guidance In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion

Champ Controllable and Consistent Human Image Animation with 3D Parametric Guidance In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion

AK

194,356 views • 2 years ago

If you can explain it, you can animate it. Turn your ideas into custom motion graphics in minutes with HeyGen’s newest feature, Motion Designer. Keep every animation on brand with your own style and colors. No templates. No After Effects. No endless tutorials. Create professional, controllable motion graphics for explainers, training videos, and product launches, all inside HeyGen. Every scene, every transition, and every detail designed to help your ideas move fast. Because great communication deserves more than static slides. It deserves motion. Don’t just tell your story. Free it.

HeyGen

1,017,849 views • 9 months ago