Loading video...

Video Failed to Load

Go Home

Here are more results from #RigidFormer: predicting physical dynamics with purely neural simulators — an attempt to learn physical dynamics in a scalable manner. 🤖 1) Controllable Articulated Body Simulation — More Results Additional Unitree G1 humanoid rollouts under controlled motion. Each sample uses a different initial state and...

frankzydou's profile picture

Zhiyang (Frank) Dou

2,018 subscribers

19,951 views • 1 month ago •via X (Twitter)

#RigidFormer
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

#DigitalTwins serve as the virtual training ground for robots as they learn to operate in the physical world. 🦾 At #HM25, Accenture and Schaeffler Group are showcasing how the Mega #NVIDIAOmniverse Blueprint accelerates #physicalAI development by simulating Digit, a humanoid robot from Agility. Learn more ➡️
0:12

#DigitalTwins serve as the virtual training ground for robots as they learn to operate in the physical world. 🦾 At #HM25, Accenture and Schaeffler Group are showcasing how the Mega #NVIDIAOmniverse Blueprint accelerates #physicalAI development by simulating Digit, a humanoid robot from Agility. Learn more ➡️

NVIDIA Omniverse

15,433 views • 1 year ago

🚀 Excited to share our #ICLR2025 work on planning with neural dynamics models! While our lab has developed diverse neural dynamics models for manipulating rigid, deformable, and granular objects, having the model alone doesn’t solve the problem—planning with it remains a challenge. 💡 Enter BaB-ND, led by Keyi and Jiangwei! We propose a scalable, GPU-accelerated branch-and-bound algorithm, inspired by neural network verification, to enable effective planning for diverse objects modeled with neural dynamics. 🔗 Project page (open-source + detailed docs!): 🎥 Watch the video to see T being pushed around obstacles, and check out Keyi’s thread for more details!
0:14

🚀 Excited to share our #ICLR2025 work on planning with neural dynamics models! While our lab has developed diverse neural dynamics models for manipulating rigid, deformable, and granular objects, having the model alone doesn’t solve the problem—planning with it remains a challenge. 💡 Enter BaB-ND, led by Keyi and Jiangwei! We propose a scalable, GPU-accelerated branch-and-bound algorithm, inspired by neural network verification, to enable effective planning for diverse objects modeled with neural dynamics. 🔗 Project page (open-source + detailed docs!): 🎥 Watch the video to see T being pushed around obstacles, and check out Keyi’s thread for more details!

Yunzhu Li

10,561 views • 1 year ago

Bringing new meaning to the word NeRD at #CoRL2025. 🤓 Neural Robot Dynamics (NeRD) bridges the sim-to-real gap with a hybrid of neural modeling and analytical simulation, unlocking zero-shot transfer and continuous learning loops for optimal robotics training. Read more & try it in Newton 🤖
0:10

Bringing new meaning to the word NeRD at #CoRL2025. 🤓 Neural Robot Dynamics (NeRD) bridges the sim-to-real gap with a hybrid of neural modeling and analytical simulation, unlocking zero-shot transfer and continuous learning loops for optimal robotics training. Read more & try it in Newton 🤖

NVIDIA Robotics

17,556 views • 9 months ago

Comparing AI video results: • Gen-3 • Kling AI 1.5 • Hailuo MiniMax • Luma Dream Machine I used a FLUX1.1 [pro] image as the input. Each model ran the same prompt 3 times, and I chose the best results: "Smooth side tracking shot of a woman jogging at twilight with the city skyline in the background, sharp focus on the woman as she pumps her arms with each stride as she jogs, depth of field, advertising commercial" Running, jogging, and walking are difficult for AI video models, and none of them produced a smooth result, but this is the worst they will be.
0:10

Comparing AI video results: • Gen-3 • Kling AI 1.5 • Hailuo MiniMax • Luma Dream Machine I used a FLUX1.1 [pro] image as the input. Each model ran the same prompt 3 times, and I chose the best results: "Smooth side tracking shot of a woman jogging at twilight with the city skyline in the background, sharp focus on the woman as she pumps her arms with each stride as she jogs, depth of field, advertising commercial" Running, jogging, and walking are difficult for AI video models, and none of them produced a smooth result, but this is the worst they will be.

Heather Cooper

57,663 views • 1 year ago

I've been in Florida for 10 years now. We all learn what the pineapple means at a certain point.... The swingers in this state are creepy and aggressive. They are also legion. They signal to each other using pineapples. You've been warned!!
0:12

Sensitive content

I've been in Florida for 10 years now. We all learn what the pineapple means at a certain point.... The swingers in this state are creepy and aggressive. They are also legion. They signal to each other using pineapples. You've been warned!!

Lj Priest (Formerly Trader Jill)

8,698,417 views • 1 year ago

As announced in partnership with NVIDIA at CES, we’re excited to introduce Stable Point Aware 3D (SPAR3D), setting a new standard in 3D generation. Ideal for running on NVIDIA RTX AI PCs, SPAR3D enables real-time editing and complete structure generation of 3D objects from a single image in under a second. You can download the weights on Hugging Face and code on GitHub, or access the model through the Stability AI API. Learn more here: (1/3)
0:13

As announced in partnership with NVIDIA at CES, we’re excited to introduce Stable Point Aware 3D (SPAR3D), setting a new standard in 3D generation. Ideal for running on NVIDIA RTX AI PCs, SPAR3D enables real-time editing and complete structure generation of 3D objects from a single image in under a second. You can download the weights on Hugging Face and code on GitHub, or access the model through the Stability AI API. Learn more here: (1/3)

Stability AI

181,441 views • 1 year ago

📢 Our lab has been exploring 3D world models for years — and we’re thrilled to share **PhysTwin**: a milestone that reconstructs object appearance, geometry, and dynamics from just a few seconds of interaction! Led by the amazing Hanxiao Jiang 👉 PhysTwin combines **Gaussian splatting** with **inverse dynamics optimization** based on simple **spring-mass** systems. ⚙️ The result? Real-time, action-conditioned 3D video prediction under novel interactions (i.e., 3D world models). 🔑 A few key takeaways: 1. Having the right structure (e.g., particles/masses) helps navigate the trade-off between sample efficiency, generalization, and broad applicability. 2. Visual foundation models (VFMs) have matured to the point where they can provide rich supervision for world modeling (e.g., tracking, shape completion). 3. Beyond VFMs, many crucial components have come together in recent years: Gaussian splats for rendering, NVIDIA Warp for high-performance simulation, and scene/asset generation from a wide range of labs and companies. The future of 3D world models is looking bright! ✨ 4. The resulting digital twin supports a wide range of downstream applications—especially in data generation and policy evaluation, thanks to its realistic rendering and simulation capabilities. 🎥 All code and data to reproduce the results, along with interactive demos, are available on the website. Check the following visualizations of: (1) observations, (2) reconstructed state/actions, (3) interactive digital twins, and (4) the overlays between real-world robot teleoperation and our model’s open-loop predictions.
0:05

📢 Our lab has been exploring 3D world models for years — and we’re thrilled to share **PhysTwin**: a milestone that reconstructs object appearance, geometry, and dynamics from just a few seconds of interaction! Led by the amazing Hanxiao Jiang 👉 PhysTwin combines **Gaussian splatting** with **inverse dynamics optimization** based on simple **spring-mass** systems. ⚙️ The result? Real-time, action-conditioned 3D video prediction under novel interactions (i.e., 3D world models). 🔑 A few key takeaways: 1. Having the right structure (e.g., particles/masses) helps navigate the trade-off between sample efficiency, generalization, and broad applicability. 2. Visual foundation models (VFMs) have matured to the point where they can provide rich supervision for world modeling (e.g., tracking, shape completion). 3. Beyond VFMs, many crucial components have come together in recent years: Gaussian splats for rendering, NVIDIA Warp for high-performance simulation, and scene/asset generation from a wide range of labs and companies. The future of 3D world models is looking bright! ✨ 4. The resulting digital twin supports a wide range of downstream applications—especially in data generation and policy evaluation, thanks to its realistic rendering and simulation capabilities. 🎥 All code and data to reproduce the results, along with interactive demos, are available on the website. Check the following visualizations of: (1) observations, (2) reconstructed state/actions, (3) interactive digital twins, and (4) the overlays between real-world robot teleoperation and our model’s open-loop predictions.

Yunzhu Li

25,279 views • 1 year ago

The theory of higher order topological dynamics, which combines multilevel interactions between discrete topology and nonlinear dynamics, has the potential to enhance our understanding of complex systems such as the functions of the nervous system, the development of next-generation machine learning and the creation of advanced nodal processing algorithms. An important and unexpected collective behavior of signal processing in multilevel nodal networks has been observed to lead to a synchronization and diffusion of the irrotational and the solenoidal components of the systems revealing a deep relation of these mechanisms with the complexity of discrete topology. The perspective of the preliminary study linked here offers insights into how topology morphs dynamics, how dynamics stem from topology and how topology evolves dynamically. 🔗
0:20

The theory of higher order topological dynamics, which combines multilevel interactions between discrete topology and nonlinear dynamics, has the potential to enhance our understanding of complex systems such as the functions of the nervous system, the development of next-generation machine learning and the creation of advanced nodal processing algorithms. An important and unexpected collective behavior of signal processing in multilevel nodal networks has been observed to lead to a synchronization and diffusion of the irrotational and the solenoidal components of the systems revealing a deep relation of these mechanisms with the complexity of discrete topology. The perspective of the preliminary study linked here offers insights into how topology morphs dynamics, how dynamics stem from topology and how topology evolves dynamically. 🔗

Maurizio Iβλἄ

40,749 views • 1 year ago

An uncommon sighting of Russian OFBCh-2F fragmentation drone bombs. Three separate fuze wells accommodate different types of fuze choices, such as the UZRG-M in the video, and a UDZ grenade fuze for the rare OFBCh-3 on the right. 1/2
0:14

An uncommon sighting of Russian OFBCh-2F fragmentation drone bombs. Three separate fuze wells accommodate different types of fuze choices, such as the UZRG-M in the video, and a UDZ grenade fuze for the rare OFBCh-3 on the right. 1/2

Roy🇨🇦

77,606 views • 1 year ago

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.
0:13

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 views • 5 months ago

📢MoRight: Motion Control Done Right "What if your video model actually understood cause and effect?" Existing motion-controlled video models entangle camera and object motion, and treat everything as kinematic displacement. MoRight changes both. 🔥 Motion Causality — MoRight decomposes motion into actions & consequences. Give an action → MoRight predicts consequences (aka motion simulation) . Give a desired outcome → MoRight recovers the driving action (aka motion planning). Not merely displacing pixels. 🎬 Disentangled Control — MoRight separates camera and object motion, allowing users to independently control each of them. No entanglement. Project Page: Paper:
0:16

📢MoRight: Motion Control Done Right "What if your video model actually understood cause and effect?" Existing motion-controlled video models entangle camera and object motion, and treat everything as kinematic displacement. MoRight changes both. 🔥 Motion Causality — MoRight decomposes motion into actions & consequences. Give an action → MoRight predicts consequences (aka motion simulation) . Give a desired outcome → MoRight recovers the driving action (aka motion planning). Not merely displacing pixels. 🎬 Disentangled Control — MoRight separates camera and object motion, allowing users to independently control each of them. No entanglement. Project Page: Paper:

Shaowei Liu

31,805 views • 3 months ago

🔹Comparing AI video results: • Gen-3 • Kling AI 1.5 • Hailuo MiniMax • Luma Dream Machine I used a FLUX LoRA trained on my selfies as the input. Each model ran the same prompt 3 times, and I chose the best results: "Tracking shot of a curator standing and speaking to the camera as she welcomes museum guests. Warm, inviting atmosphere." Each model interpreted the prompt differently, and the backgrounds are interesting:
0:10

🔹Comparing AI video results: • Gen-3 • Kling AI 1.5 • Hailuo MiniMax • Luma Dream Machine I used a FLUX LoRA trained on my selfies as the input. Each model ran the same prompt 3 times, and I chose the best results: "Tracking shot of a curator standing and speaking to the camera as she welcomes museum guests. Warm, inviting atmosphere." Each model interpreted the prompt differently, and the backgrounds are interesting:

Heather Cooper

41,285 views • 1 year ago

Video diffusion models have strong implicit representations of 3D shape, material, and lighting, but controlling them with language is cumbersome, and control is critical for artists and animators. GenLit connects these implicit representations with a continuous 5D control signal describing the direction and intensity of a point light source. This enables single-image near-field relighting of an image using a video diffusion model. We use a ControlNet-like approach and show that, with a small amount of synthetic data, GenLit generalizes to complex real-world images. Given a single image and the 5D lighting signal, GenLit creates a video of a moving light source that is inside the scene. It moves around and behind scene objects, producing effects such as shading, cast shadows, secularities, and interreflections with a realism that is hard to obtain with traditional inverse rendering methods. GenLit shows that it is possible to get continuous control over implicit physical processes within a video model. I think this is just the beginning and promises to make such models much more practical for creators. Shrisha Bharadwaj will present today at SIGGRAPH Asia Room: S423/S424, Level 4 @ 13:50 on 15 of Dec.
0:13

Video diffusion models have strong implicit representations of 3D shape, material, and lighting, but controlling them with language is cumbersome, and control is critical for artists and animators. GenLit connects these implicit representations with a continuous 5D control signal describing the direction and intensity of a point light source. This enables single-image near-field relighting of an image using a video diffusion model. We use a ControlNet-like approach and show that, with a small amount of synthetic data, GenLit generalizes to complex real-world images. Given a single image and the 5D lighting signal, GenLit creates a video of a moving light source that is inside the scene. It moves around and behind scene objects, producing effects such as shading, cast shadows, secularities, and interreflections with a realism that is hard to obtain with traditional inverse rendering methods. GenLit shows that it is possible to get continuous control over implicit physical processes within a video model. I think this is just the beginning and promises to make such models much more practical for creators. Shrisha Bharadwaj will present today at SIGGRAPH Asia Room: S423/S424, Level 4 @ 13:50 on 15 of Dec.

Michael Black

22,144 views • 6 months ago

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.
0:17

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.

Bilawal Sidhu

25,802 views • 5 months ago

(1/2) Check out "𝐏𝐨𝐥𝐲𝐃𝐢𝐟𝐟: Generating 3D Polygonal Meshes with Diffusion Models"! Our model operates directly on the polygons of 3D meshes and generates novel shapes as output through an iterative diffusion process.
0:11

(1/2) Check out "𝐏𝐨𝐥𝐲𝐃𝐢𝐟𝐟: Generating 3D Polygonal Meshes with Diffusion Models"! Our model operates directly on the polygons of 3D meshes and generates novel shapes as output through an iterative diffusion process.

Matthias Niessner

57,941 views • 2 years ago

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.
0:13

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

MrNeRF

10,607 views • 1 year ago

Under the Geneva Conventions, the recruitment of children under the age of fifteen for combat and warfare is prohibited. What is shown in this video constitutes a war crime, as the Druze National Guard militia affiliated with Hikmat al-Hijri in As-Suwayda are shown arming children from As-Suwayda and using them in combat and warfare on the frontlines, as documented in this video. #Syria #Suwayda
0:15

Under the Geneva Conventions, the recruitment of children under the age of fifteen for combat and warfare is prohibited. What is shown in this video constitutes a war crime, as the Druze National Guard militia affiliated with Hikmat al-Hijri in As-Suwayda are shown arming children from As-Suwayda and using them in combat and warfare on the frontlines, as documented in this video. #Syria #Suwayda

Omar Alhariri

45,330 views • 6 months ago

In Better on Bedrock v1.0.0-stable, xKingDark updated our UI to be more flexible and compatible with other packs! One of our favorite is the toasts, as they can now stack instead of overriding each other!! Yep! The download is live here for the release! Thanks for participating in the public test!
0:12

In Better on Bedrock v1.0.0-stable, xKingDark updated our UI to be more flexible and compatible with other packs! One of our favorite is the toasts, as they can now stack instead of overriding each other!! Yep! The download is live here for the release! Thanks for participating in the public test!

XxPoggyisLitxX

13,658 views • 1 year ago

We’ve only had access to GAIA-1 for a few weeks and are discovering new capabilities every day. The results are phenomenal! GAIA isn't just a generative video model, it is a world model, ie. controllable by video, text & action prompts Why is this huge for self driving? Thread:
0:16

We’ve only had access to GAIA-1 for a few weeks and are discovering new capabilities every day. The results are phenomenal! GAIA isn't just a generative video model, it is a world model, ie. controllable by video, text & action prompts Why is this huge for self driving? Thread:

Alex Kendall

185,456 views • 3 years ago

As a Product Designer with an educational and professional background in stop motion and frame-to-frame animation, creating these loaders was seamless. I carefully considered the shape of each frame for the desired animation. Here's my process: 👇👇 ◉ ○ ○ ○ - 1/4
0:13

As a Product Designer with an educational and professional background in stop motion and frame-to-frame animation, creating these loaders was seamless. I carefully considered the shape of each frame for the desired animation. Here's my process: 👇👇 ◉ ○ ○ ○ - 1/4

Muhammed Adepoju - Design Yoda

41,672 views • 2 years ago