Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We present Neural Riemannian Motion Fields (NRMF) as a promising paradigm for generative modeling of articulated motion. Unlike various diffusion based models, NRMF is useful both in generation and in deployment as a prior across tasks: #CVPR #AI #CVML👇

Tolga Birdal

4,042 subscribers

16,795 просмотров • 11 месяцев назад •via X (Twitter)

Наука и технологии #CVPR #AI #CVML

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,353 просмотров • 3 лет назад

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

Xueyan Zou

39,094 просмотров • 2 месяцев назад

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

Yuan Liu

33,467 просмотров • 1 год назад

$Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.$

Example-based Motion Synthesis via Generative Motion Matching paper page: present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.

AK

93,740 просмотров • 3 лет назад

Nvidia presents Align Your Steps Optimizing Sampling Schedules in Diffusion Models Diffusion models (DMs) have established themselves as the state-of-the-art generative modeling approach in the visual domain and beyond. A crucial drawback of DMs is their slow sampling speed,

Nvidia presents Align Your Steps Optimizing Sampling Schedules in Diffusion Models Diffusion models (DMs) have established themselves as the state-of-the-art generative modeling approach in the visual domain and beyond. A crucial drawback of DMs is their slow sampling speed,

AK

32,888 просмотров • 2 лет назад

Motion planning in complex tasks is hard and still done via slow, explicit, traditional planners. We present a generalist Neural Motion Planner -- a single neural network that plans complex dynamic motions quickly and accurately at test time. Building upon our lab's sim2real efforts, the key idea is to create many complex scenes in simulation and then distill classical motion planner trajectories into a single reactive neural network policy. More details in the thread below! 👇 Open-sourced:

Motion planning in complex tasks is hard and still done via slow, explicit, traditional planners. We present a generalist Neural Motion Planner -- a single neural network that plans complex dynamic motions quickly and accurately at test time. Building upon our lab's sim2real efforts, the key idea is to create many complex scenes in simulation and then distill classical motion planner trajectories into a single reactive neural network policy. More details in the thread below! 👇 Open-sourced:

Deepak Pathak

28,642 просмотров • 2 лет назад

Improving the motion-tracked panorama generation in Sitrec. Creating a wider view of a video is often useful for geolocation. So I've tried to automate it as much as possible.

Improving the motion-tracked panorama generation in Sitrec. Creating a wider view of a video is often useful for geolocation. So I've tried to automate it as much as possible.

Mick West

12,223 просмотров • 3 месяцев назад

Loopy Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency paper page: With the introduction of diffusion-based video generation techniques, audio-conditioned human video generation has recently achieved significant breakthroughs in both the naturalness of motion and the synthesis of portrait details. Due to the limited control of audio signals in driving human motion, existing methods often add auxiliary spatial signals to stabilize movements, which may compromise the naturalness and freedom of motion. In this paper, we propose an end-to-end audio-only conditioned video diffusion model named Loopy. Specifically, we designed an inter- and intra-clip temporal module and an audio-to-latents module, enabling the model to leverage long-term motion information from the data to learn natural motion patterns and improving audio-portrait movement correlation. This method removes the need for manually specified spatial motion templates used in existing methods to constrain motion during inference. Extensive experiments show that Loopy outperforms recent audio-driven portrait diffusion models, delivering more lifelike and high-quality results across various scenarios.

AK

128,852 просмотров • 2 лет назад

Gaussian Shell Maps are a new neural scene representation that connects fields and 3D Gaussians. This representation unlocks the full potential of 3D Gaussian splatting for generative AI applications, such as 3D avatar generation. 1/2

Gordon Wetzstein

52,480 просмотров • 2 лет назад

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

The Humanoid Hub

28,004 просмотров • 7 месяцев назад

Collaborative Score Distillation for Consistent Visual Synthesis paper page: Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as "particles" in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

Collaborative Score Distillation for Consistent Visual Synthesis paper page: Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as "particles" in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

AK

33,513 просмотров • 3 лет назад

Got five papers accepted by #ECCV2024 European Conference on Computer Vision #ECCV2026 ! Huge thanks to all my collaborators! 😃 See you in Milan 🇮🇹 Summary of Selected Works (I made a fast-forward for them 😄) - [Shape Generation] Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models, ECCV 2024. - [Efficient Motion Generation] EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, ECCV 2024. - [Controllable Motion Generation] TLControl: Trajectory and Language Control for Human Motion Synthesis, ECCV 2024. - [Avatar Generation] Disentangled Clothed Avatar Generation from Text Descriptions, ECCV 2024. Project Page: Surf-D: EMDM: TLControl: SOSMPL:

Zhiyang (Frank) Dou

18,192 просмотров • 2 лет назад

Google isn’t betting on a single AI architecture. Sundar Pichai, CEO of Google: “We’re going to push the diffusion paradigm as hard as possible.” “All of today’s mainline Gemini models are autoregressive. Diffusion is a different paradigm.” “For the same capability, diffusion can be much faster.” “It’s behind the mainline models today, but there will be areas where it’s the right tool.” “We’re pushing multiple directions in parallel, and bringing them together where it makes sense.”

Google isn’t betting on a single AI architecture. Sundar Pichai, CEO of Google: “We’re going to push the diffusion paradigm as hard as possible.” “All of today’s mainline Gemini models are autoregressive. Diffusion is a different paradigm.” “For the same capability, diffusion can be much faster.” “It’s behind the mainline models today, but there will be areas where it’s the right tool.” “We’re pushing multiple directions in parallel, and bringing them together where it makes sense.”

Forward Future

152,933 просмотров • 8 месяцев назад

Generative video is dope but AI motion capture is just as cool. Pretty much James Cameron ish in your living room — and that too with just a 2D camera

Generative video is dope but AI motion capture is just as cool. Pretty much James Cameron ish in your living room — and that too with just a 2D camera

Bilawal Sidhu

89,178 просмотров • 1 год назад

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

AK

32,849 просмотров • 2 лет назад

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

RAI Institute

13,556 просмотров • 1 год назад

GenCast is a diffusion model, similar to the machine learning models which also power generative AI. 🎨 We trained it on 40 years of historical data from ECMWF, which included variables such as temperature, wind speed, and pressure at various altitudes - enabling it to learn global weather patterns.

GenCast is a diffusion model, similar to the machine learning models which also power generative AI. 🎨 We trained it on 40 years of historical data from ECMWF, which included variables such as temperature, wind speed, and pressure at various altitudes - enabling it to learn global weather patterns.

Google DeepMind

26,360 просмотров • 1 год назад

Today we're introducing Gen-4, our new series of state-of-the-art AI models for media generation and world consistency. Gen-4 is a significant step forward for fidelity, dynamic motion and controllability in generative media. Gen-4 Image-to-Video is rolling out today to all paid plans and Enterprise customers. 1/8

Today we're introducing Gen-4, our new series of state-of-the-art AI models for media generation and world consistency. Gen-4 is a significant step forward for fidelity, dynamic motion and controllability in generative media. Gen-4 Image-to-Video is rolling out today to all paid plans and Enterprise customers. 1/8

Runway

737,166 просмотров • 1 год назад

Open models are accelerating development and deployment across agentic AI, physical AI, and a growing range of industries. ✨ “We have billions of dollars of supercomputers in operation, so we can develop our open models,” said NVIDIA CEO Jensen Huang in a recent keynote. “That work is starting to attract attention across industries worldwide, as we build frontier AI models in many different domains.” This includes leading open models from NVIDIA, including: ⚙️ NVIDIA Nemotron for agentic AI ⚙️ NVIDIA Cosmos and GR00T for physical AI ⚙️ NVIDIA Alpamayo for autonomous vehicles ⚙️ NVIDIA Clara for biomedical AI Learn more about open source and AI ➡️

Open models are accelerating development and deployment across agentic AI, physical AI, and a growing range of industries. ✨ “We have billions of dollars of supercomputers in operation, so we can develop our open models,” said NVIDIA CEO Jensen Huang in a recent keynote. “That work is starting to attract attention across industries worldwide, as we build frontier AI models in many different domains.” This includes leading open models from NVIDIA, including: ⚙️ NVIDIA Nemotron for agentic AI ⚙️ NVIDIA Cosmos and GR00T for physical AI ⚙️ NVIDIA Alpamayo for autonomous vehicles ⚙️ NVIDIA Clara for biomedical AI Learn more about open source and AI ➡️

NVIDIA AI

19,296 просмотров • 7 месяцев назад

Make Pixels Dance: High-Dynamic Video Generation paper page: Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.

Make Pixels Dance: High-Dynamic Video Generation paper page: Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.

AK

102,706 просмотров • 2 лет назад