Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Today, we're introducing SimFoundry, our real2sim2real framework at NVIDIA GEAR that automatically turns real-world scenes into simulation-ready worlds from a single image or video. Website: Paper: This work marks a major step for our team toward leveraging simulations and synthetic data for foundation model training and systematic policy evaluation... show more

Yuke Zhu

23,849 subscribers

80,828 views • 3 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

15 Comments

Aetheris_Consulting 🇺🇸3 months ago

Similar to this

Aetheris_Consulting 🇺🇸3 months ago

Yuke awesome work here!!! Can you consider making a plug in for @UnrealEngine @unity @godotengine ? That was through a harness LLM can do physics computations paired with real2sim2real to do a number of things. This is such great work pls keep it up!!! 👍

Arthur Petron3 months ago

How well do dynamics work? Like, if I drop the tennis ball from 0.5m in real does it bounce the same in sim?

Weijie Wang2 months ago

Another antidote to hype—real frontier work with paper and open-source pledge, verifiable author; judge it by the arXiv paper, not the summary.

Sitarama Chekuri3 months ago

Looks amazing. Are the objects physics ready and interactive?

Mikhail3 months ago

thanks habibi

Ferbin3 months ago

Robot training always gets stuck building training environments. Automating that from photos changes the whole game.

Efstratios Gavves3 months ago

@yukez Great work! I think it would be fair and nice to cite our DreMa work that was the first to introduce in ICLR 2025 the paradigm:

just101012 months ago

Shameless @danfei_xu Stop stealing from students/ interviewees Shame on @gtcomputing

EB1A Experts3 months ago

Impressive work. Bridging real-world scenes and simulation-ready environments from minimal input is an exciting step toward more scalable robotics and foundation model development. Looking forward to seeing the open-source release.

Larry Panozzo3 months ago

Wow, from one video from one lens with depth estimation…amazing. Most applications would use stereo I imagine? Results could be even better then, with a different version of this framework?

Zavian Pokharkar2 months ago

Yuke

Adel Dennaoui2 months ago

When will the code be released? :)

AI Mastery Guide3 months ago

Real2sim2real from a single image is huge for robotics training data, excited to see this open-sourced.

Kavya1 month ago

@yukez, when will you be releasing the code? Would love to try this out!

Related Videos

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at

Runway

113,724 views • 7 days ago

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 views • 7 months ago

We’re open-sourcing HY-World 2.0, a multimodal world model that generates, reconstructs, and simulates interactive *3D worlds* from text, images, and videos. Outputs can be integrated into game engines and embodied simulation pipelines. Key highlights: 🔹 One-click world generation Turn text or image into interactive 3D worlds automatically. 🔹 Pipeline-ready 3D outputs Editable 3D worlds for Unity and Unreal Engine, with standard 3D exports including mesh, 3DGS, and point clouds. 🔹 Unified world model system One model family for world generation and reconstruction across synthetic and real-world scenes. 🔹 Interactive character mode Explore generated 3D worlds in real time with physics-aware movement and collision support. ✨ Apply for access: 🔗 GitHub: 🤗 Hugging Face: 📄 Technical Report:

We’re open-sourcing HY-World 2.0, a multimodal world model that generates, reconstructs, and simulates interactive 3D worlds from text, images, and videos. Outputs can be integrated into game engines and embodied simulation pipelines. Key highlights: 🔹 One-click world generation Turn text or image into interactive 3D worlds automatically. 🔹 Pipeline-ready 3D outputs Editable 3D worlds for Unity and Unreal Engine, with standard 3D exports including mesh, 3DGS, and point clouds. 🔹 Unified world model system One model family for world generation and reconstruction across synthetic and real-world scenes. 🔹 Interactive character mode Explore generated 3D worlds in real time with physics-aware movement and collision support. ✨ Apply for access: 🔗 GitHub: 🤗 Hugging Face: 📄 Technical Report:

Tencent Hy

372,797 views • 5 months ago

Woow Nvidia has just released a 2.6B open-source world model 🔥 You can turn a single image, text prompt and trajectory into controllable worlds... And on a single GPU! - Code available on GitHub - Paper as well on arxiv You can use it for many things like embodied AI and robotics research, simulations, etc. Because it can run on a single GPU (like an RTX 5090 or H100) it makes world models accessible to basically everyone!

Woow Nvidia has just released a 2.6B open-source world model 🔥 You can turn a single image, text prompt and trajectory into controllable worlds... And on a single GPU! - Code available on GitHub - Paper as well on arxiv You can use it for many things like embodied AI and robotics research, simulations, etc. Because it can run on a single GPU (like an RTX 5090 or H100) it makes world models accessible to basically everyone!

Paul Couvert

174,447 views • 4 months ago

🧵 Evaluating robot policies in the real world is slow, expensive, and hard to scale. During my internship at SceniX this summer, we had many discussions around the two key questions: how accurate must a simulator be for evaluation to be meaningful, and how do we get there? Our new framework, Real2Sim-Eval, takes a step toward that answer. By combining Gaussian Splatting for photorealistic rendering and soft-body digital twins for realistic dynamics, we make simulation predictive of real-world performance. 👉

🧵 Evaluating robot policies in the real world is slow, expensive, and hard to scale. During my internship at SceniX this summer, we had many discussions around the two key questions: how accurate must a simulator be for evaluation to be meaningful, and how do we get there? Our new framework, Real2Sim-Eval, takes a step toward that answer. By combining Gaussian Splatting for photorealistic rendering and soft-body digital twins for realistic dynamics, we make simulation predictive of real-world performance. 👉

Kaifeng Zhang

67,703 views • 11 months ago

Open-ended coding training data may no longer be the bottleneck: AI can scale open-ended tasks—and even outperform human-expert curation. FrontierCS team is releasing FrontierSmith: a system for synthesizing open-ended coding problems at scale. Starting from closed-ended coding tasks, FrontierSmith mutates, filters, and builds runnable optimization environments for long-horizon coding agents. In our experiments, FrontierSmith data trains stronger models than human-curated open-ended data on FrontierCS and ALE-bench. Blog: Paper: Code: Model:

Open-ended coding training data may no longer be the bottleneck: AI can scale open-ended tasks—and even outperform human-expert curation. FrontierCS team is releasing FrontierSmith: a system for synthesizing open-ended coding problems at scale. Starting from closed-ended coding tasks, FrontierSmith mutates, filters, and builds runnable optimization environments for long-horizon coding agents. In our experiments, FrontierSmith data trains stronger models than human-curated open-ended data on FrontierCS and ALE-bench. Blog: Paper: Code: Model:

Qiuyang Mang

121,283 views • 4 months ago

Big thanks to AK for highlighting our work! LEO marks our pioneering step towards building an embodied generalist agent that can really comprehend the 3D world! 🚀Leveraging LLMs, we train LEO with real and synthetic 3D data across a diverse spectrum of tasks. It's thrilling to see LEO surpass current state-of-the-art SOTA methods in most benchmarked tasks, all under a single, unified model. 🔥 #Generalist_Agent

Big thanks to AK for highlighting our work! LEO marks our pioneering step towards building an embodied generalist agent that can really comprehend the 3D world! 🚀Leveraging LLMs, we train LEO with real and synthetic 3D data across a diverse spectrum of tasks. It's thrilling to see LEO surpass current state-of-the-art SOTA methods in most benchmarked tasks, all under a single, unified model. 🔥 #Generalist_Agent

Siyuan Huang

22,710 views • 2 years ago

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 views • 7 months ago

We’ve raised $50 million in Series A funding from DST Global Partners, NVIDIA (NVIDIA’s venture capital arm), redalpine , prosusventures , and Moonfire Ventures , following our $7.35M seed round from Frst , Moonfire Ventures , and redalpine just a few months earlier, to build the autonomy stack that makes humanoid robots adaptive, intelligent, and ready for real-world deployment at scale. In less than a year, our team has shown that long-horizon whole-body humanoid control can scale across hardware and tasks by leveraging the power of simulation and reinforcement learning. This funding will help us grow our team, scale our compute and robot fleets, and accelerate the commercialization of our autonomy stack with OEM partners globally. You can find more details in the links shared in the comments.

We’ve raised $50 million in Series A funding from DST Global Partners, NVIDIA (NVIDIA’s venture capital arm), redalpine , prosusventures , and Moonfire Ventures , following our $7.35M seed round from Frst , Moonfire Ventures , and redalpine just a few months earlier, to build the autonomy stack that makes humanoid robots adaptive, intelligent, and ready for real-world deployment at scale. In less than a year, our team has shown that long-horizon whole-body humanoid control can scale across hardware and tasks by leveraging the power of simulation and reinforcement learning. This funding will help us grow our team, scale our compute and robot fleets, and accelerate the commercialization of our autonomy stack with OEM partners globally. You can find more details in the links shared in the comments.

Flexion

74,067 views • 10 months ago

[1/2] We’ve released the code for #pix2pixturbo and #CycleGANTurbo. These conditional GANs are able to adapt a text-to-image model such as SD-Turbo for both paired and unpaired image translation with a single step (0.11 sec on A100 and 0.29 sec on A6000). Try our code and the Gradio demo. Paper: Code: Demo: This is a joint work with Gaurav Parmar (the leading author), Taesung Park, and Srinivasa Narasimhan. This work shows that a pre-trained one-step model can be easily adapted to conditional GANs frameworks for downstream image editing and synthesis tasks. #Edges2Cats

[1/2] We’ve released the code for #pix2pixturbo and #CycleGANTurbo. These conditional GANs are able to adapt a text-to-image model such as SD-Turbo for both paired and unpaired image translation with a single step (0.11 sec on A100 and 0.29 sec on A6000). Try our code and the Gradio demo. Paper: Code: Demo: This is a joint work with Gaurav Parmar (the leading author), Taesung Park, and Srinivasa Narasimhan. This work shows that a pre-trained one-step model can be easily adapted to conditional GANs frameworks for downstream image editing and synthesis tasks. #Edges2Cats

Jun-Yan Zhu

36,509 views • 2 years ago

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

Yukang Chen

11,943 views • 1 year ago

Today, we're sharing new research on GWM Worlds 2. Our latest General World Model that generates interactive real-time simulations in continuous 720p video at 24 fps with 48000 Hz audio. Built on top of our foundational audio-visual generation model, GWM Worlds 2 allows you to define the environment, subjects, visual style, physical rules and ambience of any world. Once inside, you steer it with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. The world continues from each new input instead of following a fixed clip or script so sessions have no preset length. GWM Worlds 2 creates a new foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation. Learn more about GWM Worlds 2 at the link below.

Today, we're sharing new research on GWM Worlds 2. Our latest General World Model that generates interactive real-time simulations in continuous 720p video at 24 fps with 48000 Hz audio. Built on top of our foundational audio-visual generation model, GWM Worlds 2 allows you to define the environment, subjects, visual style, physical rules and ambience of any world. Once inside, you steer it with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. The world continues from each new input instead of following a fixed clip or script so sessions have no preset length. GWM Worlds 2 creates a new foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation. Learn more about GWM Worlds 2 at the link below.

Runway

306,008 views • 1 month ago

Your next robotics project could start with these steps. Today, we're open-sourcing Asimov 1's locomotion policy and the training code behind it, giving developers a foundation to build on. Alongside the trained policy checkpoint, we're sharing our Isaac Lab-based training code, including reward definitions, hardware-tuned actuator configurations, and domain randomization settings. That foundation gives you a starting point for your own experiments. Adapt the training setup to hardware changes. Explore different walking styles. Build toward new capabilities. We've taken the first steps. Build what comes next. Asimov 1 locomotion:

Your next robotics project could start with these steps. Today, we're open-sourcing Asimov 1's locomotion policy and the training code behind it, giving developers a foundation to build on. Alongside the trained policy checkpoint, we're sharing our Isaac Lab-based training code, including reward definitions, hardware-tuned actuator configurations, and domain randomization settings. That foundation gives you a starting point for your own experiments. Adapt the training setup to hardware changes. Explore different walking styles. Build toward new capabilities. We've taken the first steps. Build what comes next. Asimov 1 locomotion:

Asimov

57,341 views • 12 days ago

Most AI world models can generate beautiful scenes. Keeping those scenes alive for an hour without falling apart is the real challenge. That's what caught my attention about LingBot-World 2.0 (LingBot-World-Infinity) from Robbyant Instead of chasing longer videos, it focuses on something much harder: persistent, interactive worlds that stay coherent while you explore. A few highlights: • Generates worlds from a single frame and continuously responds to live user actions through a causal world model. • Streams stable 720p at 60 FPS in real time. The team reports a continuous 60 minute stress test across 20 different scenarios with no noticeable visual degradation. • Uses a Brain-Cerebellum co-simulation framework where a VLM plans events while the video model turns them into consistent world evolution. • Pilot and Director Agents help drive character behavior and introduce new objects and events. • Open sourced with a 14B flagship model, while the paper also describes a lightweight 1.3B version for a single consumer GPU. There is also an online interactive demo. The biggest takeaway? We're moving beyond AI that generates clips. We're getting closer to AI that generates living, evolving worlds you can actually interact with. And that feels like a much bigger shift than another jump in video quality. Explore more: 💻 Github: 🤗 Weights: 🌐 Website-with videos you can use : 🎮 Try it online: #Robbyant #LingBot #WorldModel #EmbodiedAI #OpenSource #Robotics #ad

Most AI world models can generate beautiful scenes. Keeping those scenes alive for an hour without falling apart is the real challenge. That's what caught my attention about LingBot-World 2.0 (LingBot-World-Infinity) from Robbyant Instead of chasing longer videos, it focuses on something much harder: persistent, interactive worlds that stay coherent while you explore. A few highlights: • Generates worlds from a single frame and continuously responds to live user actions through a causal world model. • Streams stable 720p at 60 FPS in real time. The team reports a continuous 60 minute stress test across 20 different scenarios with no noticeable visual degradation. • Uses a Brain-Cerebellum co-simulation framework where a VLM plans events while the video model turns them into consistent world evolution. • Pilot and Director Agents help drive character behavior and introduce new objects and events. • Open sourced with a 14B flagship model, while the paper also describes a lightweight 1.3B version for a single consumer GPU. There is also an online interactive demo. The biggest takeaway? We're moving beyond AI that generates clips. We're getting closer to AI that generates living, evolving worlds you can actually interact with. And that feels like a much bigger shift than another jump in video quality. Explore more: 💻 Github: 🤗 Weights: 🌐 Website-with videos you can use : 🎮 Try it online: #Robbyant #LingBot #WorldModel #EmbodiedAI #OpenSource #Robotics #ad

Alif Khan

84,553 views • 2 months ago

Introducing SimReadyGen. Agentic simulation generation for Physical AI. From a text prompt to physically accurate SimReady assets at scale. Robot training demands physically accurate environments at a scale traditional asset creation can't match. SimReadyGen closes that gap. Every generated asset carries measured physics, not estimates. Behind it is SimReady Foundry, our measured-physics pipeline: we measure how real objects behave — contact, friction, dynamics — and turn those measurements into ground-truth simulation parameters. Built on OpenUSD and integrated with NVIDIA Robotics Omniverse Libraries, SimReadyGen moves directly into Isaac Sim and Isaac Lab workflows. Try it today: Read the announcement: #PhysicalAI #Robotics #Simulation #OpenUSD

Introducing SimReadyGen. Agentic simulation generation for Physical AI. From a text prompt to physically accurate SimReady assets at scale. Robot training demands physically accurate environments at a scale traditional asset creation can't match. SimReadyGen closes that gap. Every generated asset carries measured physics, not estimates. Behind it is SimReady Foundry, our measured-physics pipeline: we measure how real objects behave — contact, friction, dynamics — and turn those measurements into ground-truth simulation parameters. Built on OpenUSD and integrated with NVIDIA Robotics Omniverse Libraries, SimReadyGen moves directly into Isaac Sim and Isaac Lab workflows. Try it today: Read the announcement: #PhysicalAI #Robotics #Simulation #OpenUSD

Lightwheel

12,221 views • 2 months ago

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

Shuo Yang

31,872 views • 6 months ago

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,960 views • 11 months ago

introducing: the open ecosystem for physical intelligence. a unified foundation where real-time, verified data from machines powers intelligent systems that sense, reason, and act in the real world. today marks a new chapter for IoTeX. visit the new website: since 2017, we've connected millions of smart devices to blockchain and helped launch the global DePIN movement. we built a high-performance L1 for machines, then modular infrastructure for DePINs. now, we're expanding into AI. this is chapter 3 for IoTeX. and the next evolution for AI.

introducing: the open ecosystem for physical intelligence. a unified foundation where real-time, verified data from machines powers intelligent systems that sense, reason, and act in the real world. today marks a new chapter for IoTeX. visit the new website: since 2017, we've connected millions of smart devices to blockchain and helped launch the global DePIN movement. we built a high-performance L1 for machines, then modular infrastructure for DePINs. now, we're expanding into AI. this is chapter 3 for IoTeX. and the next evolution for AI.

IoTeX

856,542 views • 1 year ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,897 views • 1 year ago