正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

When we run out of good training data in reality, simulation is the next gold mine. Enters Infinigen: an open-source, procedurally generated, photorealistic dataset for 3D vision. The quality is stunning! No two worlds are the same. ▸ Every little detail is randomized and customizable, even the wrinkles on... show more

Jim Fan

478,618 subscribers

363,656 次观看 • 3 年前 •via X (Twitter)

教育健康养生科学技术

Anya Rossi• Live Now

Private livecam show

10 条评论

Jim Fan 的头像

Jim Fan3 年前

High-quality automatic annotations for training visual foundation models:

Jim Fan 的头像

Jim Fan3 年前

Procedural generation down to flower petals:

Jim Fan 的头像

Jim Fan3 年前

Diversity in the simulation:

Jim Fan 的头像

Jim Fan3 年前

CVPR 2023, congrats to the authors: Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang,Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, Jia Deng.

lee 的头像

lee3 年前

Did a really high end version of this at Weta Digital, allowed James Cameron to find a spot anywhere in the world of Pandora, this was a long time ago, fidelity was unbelievable so behind closed doors we are a long way ahead of this still :) cool procedural sim in unreal now too

Kevin Patrick Murphy 的头像

Kevin Patrick Murphy3 年前

This looks super cool. Your tweet is better summary than the original authors :)

Jim Fan 的头像

Jim Fan3 年前

Thanks Kevin! 🙏

The Simulation 的头像

The Simulation3 年前

Fascinating!!!

Felix Chin MD 的头像

Felix Chin MD3 年前

It's simple. We have generative AI produce a bunch of pictures. Then we select the best pictures and use them to train new models. It’s like reinforcement learning, except we’re using the best responses to train entirely new models.

Eric Elliott 的头像

Eric Elliott3 年前

Throw in physics simulation and procedural metahumans for good measure. 🎉

相关视频

Scaling 3D scene data is a long-standing challenge in scene understanding, spatial reasoning, and robotics. Since scanning, reconstruction, and labeling are so labor-intensive, data scarcity has remained a major bottleneck. 🛑 To solve this, we propose SceneVerse++: Lifting Unlabeled Internet-level Data for 3D Scene Understanding (CVPR 2026). By reconstructing internet videos and annotating 3D scenes automatically, we’ve created a massive real-world dataset for end-to-end understanding. 🌐📐 SceneVerse++ makes it easy to scale "in-the-wild" 3D scenes toward more capable spatial reasoning systems. This significantly promotes progress in 3D VQA, visual navigation, and broader tasks in Embodied AI and Robotics. 🤖🦾 We are fully open-sourced! Check out the paper, code, and data here: 🌐 Project: 📄 Paper: 📊 Dataset: Code:

Scaling 3D scene data is a long-standing challenge in scene understanding, spatial reasoning, and robotics. Since scanning, reconstruction, and labeling are so labor-intensive, data scarcity has remained a major bottleneck. 🛑 To solve this, we propose SceneVerse++: Lifting Unlabeled Internet-level Data for 3D Scene Understanding (CVPR 2026). By reconstructing internet videos and annotating 3D scenes automatically, we’ve created a massive real-world dataset for end-to-end understanding. 🌐📐 SceneVerse++ makes it easy to scale "in-the-wild" 3D scenes toward more capable spatial reasoning systems. This significantly promotes progress in 3D VQA, visual navigation, and broader tasks in Embodied AI and Robotics. 🤖🦾 We are fully open-sourced! Check out the paper, code, and data here: 🌐 Project: 📄 Paper: 📊 Dataset: Code:

Siyuan Huang

12,779 次观看 • 4 个月前

2D is what the camera sees.👀 3D is what the world is. 🌍 Today, we’re proudly open-sourcing QuerySplat — an open-source feed-forward 3D model that turns a few unposed images into a navigable 3D scene in seconds (run on 4090🪶). Your input: a few unposed images Our output: a sharp, navigable 3D scene — generated in seconds! 🚀 🧠 organize scenes with 3D queries, not pixel-bound Gaussians 📷 no camera poses or manual calibration required ⚡ reconstruct new scenes in a single forward pass ✨ preserve sharp geometry and high-frequency appearance 🏆 achieve state-of-the-art results on DL3DV-Evaluation 🔓 code and model weights are now open-source From pixels to scenes, we are making 3D generation scalable. Try our free iOS app — it’s a lot of fun! 😆 🔗 Try it on the Web: 🔗 Paper: 🔗 Code & weights: 🔗 Project page: 🔗 #AIGC #3DGS #AI #Computervision #3D #artist #3dartist #3d

2D is what the camera sees.👀 3D is what the world is. 🌍 Today, we’re proudly open-sourcing QuerySplat — an open-source feed-forward 3D model that turns a few unposed images into a navigable 3D scene in seconds (run on 4090🪶). Your input: a few unposed images Our output: a sharp, navigable 3D scene — generated in seconds! 🚀 🧠 organize scenes with 3D queries, not pixel-bound Gaussians 📷 no camera poses or manual calibration required ⚡ reconstruct new scenes in a single forward pass ✨ preserve sharp geometry and high-frequency appearance 🏆 achieve state-of-the-art results on DL3DV-Evaluation 🔓 code and model weights are now open-source From pixels to scenes, we are making 3D generation scalable. Try our free iOS app — it’s a lot of fun! 😆 🔗 Try it on the Web: 🔗 Paper: 🔗 Code & weights: 🔗 Project page: 🔗 #AIGC #3DGS #AI #Computervision #3D #artist #3dartist #3d

InSpatio｜影溯

29,524 次观看 • 24 天前

This is pretty incredible. Opus 5 can procedurally generate 3D worlds from reference images. You can also interact with objects in the generated world and they stay in place, pretty amazing. The iterative refinement from the agent is really impressive. I asked the agent to walk the world and improve it at every step. The visual quality refinement is substantial as the agent explores the world itself. Some visual examples in the next post. I am really impressed by the visual quality. There are still some minor issues but the run is not yet complete. On the other hand it has already burn through 15% of my weekly limits on Max!

This is pretty incredible. Opus 5 can procedurally generate 3D worlds from reference images. You can also interact with objects in the generated world and they stay in place, pretty amazing. The iterative refinement from the agent is really impressive. I asked the agent to walk the world and improve it at every step. The visual quality refinement is substantial as the agent explores the world itself. Some visual examples in the next post. I am really impressed by the visual quality. There are still some minor issues but the run is not yet complete. On the other hand it has already burn through 15% of my weekly limits on Max!

Prompt

65,624 次观看 • 1 个月前

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

🚀 Announcing Echo — our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world — not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly — even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering — but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows — digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations — all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction — environments that don’t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. 🌍 #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out:

SpAItial AI

176,524 次观看 • 8 个月前

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Jim Fan

332,238 次观看 • 2 年前

GeoLibre v1.10.0 is here! GeoLibre is a free and open-source, lightweight, cloud-native GIS platform for visualizing, exploring, and analyzing geospatial data. It runs everywhere you do, in the web browser, on the desktop, on mobile, and inside Jupyter notebooks, all while keeping your data local and private. This release brings photorealistic city-scale 3D to the map, adds OGC API vector tiles, and makes vector tiles and live editing easier to work with. What's new in v1.10.0 - Google Photorealistic 3D Tiles: drape entire cities in photorealistic 3D right in the map, streamed and rendered through - ArcGIS I3S scene layers: stream Integrated Mesh and 3D Object scene layers and render them in 3D on the shared instance. - OGC API vector tiles: add standards-based OGC API - Tiles vector tile services as a remote source, right alongside XYZ and ArcGIS vector tiles. - Local HDF5 and NetCDF-4 loading: open scientific HDF5 and NetCDF-4 files straight from disk, no cloud references required. - GeoEditor from the current view: pull the vector features currently visible in the map into the GeoEditor and start editing what you are looking at, without re-importing the source data. Try it out - Launch GeoLibre Web: - GitHub: - Documentation: - Release notes: #GIS #Geospatial #OpenSource #RemoteSensing #DataVisualization #MapLibre

GeoLibre v1.10.0 is here! GeoLibre is a free and open-source, lightweight, cloud-native GIS platform for visualizing, exploring, and analyzing geospatial data. It runs everywhere you do, in the web browser, on the desktop, on mobile, and inside Jupyter notebooks, all while keeping your data local and private. This release brings photorealistic city-scale 3D to the map, adds OGC API vector tiles, and makes vector tiles and live editing easier to work with. What's new in v1.10.0 - Google Photorealistic 3D Tiles: drape entire cities in photorealistic 3D right in the map, streamed and rendered through - ArcGIS I3S scene layers: stream Integrated Mesh and 3D Object scene layers and render them in 3D on the shared instance. - OGC API vector tiles: add standards-based OGC API - Tiles vector tile services as a remote source, right alongside XYZ and ArcGIS vector tiles. - Local HDF5 and NetCDF-4 loading: open scientific HDF5 and NetCDF-4 files straight from disk, no cloud references required. - GeoEditor from the current view: pull the vector features currently visible in the map into the GeoEditor and start editing what you are looking at, without re-importing the source data. Try it out - Launch GeoLibre Web: - GitHub: - Documentation: - Release notes: #GIS #Geospatial #OpenSource #RemoteSensing #DataVisualization #MapLibre

Qiusheng Wu

24,578 次观看 • 1 个月前

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

The robot’s “eyes” just received a big upgrade. LingBot-Depth 2.0, a depth-completion model with half the depth error just dropped. 12/16 benchmarks topped. Glass, mirrors, and transparent objects are so easy for us humans, but so hard for robots, because they do not behave like ordinary surfaces in a camera pipeline. A robot that misunderstands a balcony window or a table edge, will have a completely false planning inside a false world. Huge implecation. LingBot-Depth 2.0 takes an RGB image plus a broken depth map from a sensor and then outputs a cleaner depth map and a usable 3D point cloud. Numbers on LingBot-Depth 2.0 - Excels on glass, mirrors & transparent objects — where traditional depth cameras fail - Training data: 3M → 150M (50x scale-up) - 12 out of 16 first-place rankings on depth completion benchmarks - RMSE cut in half: 0.132 → 0.062 on the hardest indoor scenes LingBot-Vision trained on boundaries, because object edges carry the geometry robots need. No human boundary labels are used, which makes this approach easier to scale. The open-sourced LingBot-Vision is the general vision backbone, and LingBot-Depth 2.0 is the depth model built on it.

Rohan Paul

25,312 次观看 • 1 个月前

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,637 次观看 • 2 年前

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.

Tracking Anything with Decoupled Video Segmentation paper page: Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation.

AK

305,683 次观看 • 3 年前

The BEDLAM2.0 dataset (B2) is here, just in time to train your 3D human pose and shape estimation methods for CVPR. B2 goes beyond BEDLAM (B1) to include widely varied and natural camera motions and fields of view, more diverse body shapes, strand-based hair, more garments, shoes, more body motions, and more 3D scenes. Compared with B1, training on B2 produces more accurate 3D human pose, resulting in SOTA accuracy, particularly for estimates in world coordinates. B2 lets you jointly train camera motion and human motion regressors, and we also provide depth maps. Check out data, code, dataset statistics, and much more. BEDLAM2.0 will appear in the 2025 NeurIPS Datasets and Benchmarks Track. Joint work with Joachim Tesch, Giorgio Becherini, Prerana Achar, Anastasios Yiannakidis, Muhammed Kocabas, Priyanka Patel.

The BEDLAM2.0 dataset (B2) is here, just in time to train your 3D human pose and shape estimation methods for CVPR. B2 goes beyond BEDLAM (B1) to include widely varied and natural camera motions and fields of view, more diverse body shapes, strand-based hair, more garments, shoes, more body motions, and more 3D scenes. Compared with B1, training on B2 produces more accurate 3D human pose, resulting in SOTA accuracy, particularly for estimates in world coordinates. B2 lets you jointly train camera motion and human motion regressors, and we also provide depth maps. Check out data, code, dataset statistics, and much more. BEDLAM2.0 will appear in the 2025 NeurIPS Datasets and Benchmarks Track. Joint work with Joachim Tesch, Giorgio Becherini, Prerana Achar, Anastasios Yiannakidis, Muhammed Kocabas, Priyanka Patel.

Michael Black

27,536 次观看 • 10 个月前

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,663 次观看 • 3 年前

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,567 次观看 • 3 年前

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike

Harry Stebbings

63,738 次观看 • 2 个月前

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

✨We are excited to open-source Tencent HY-Motion 1.0, a billion-parameter text-to-motion model built on the Diffusion Transformer (DiT) architecture and flow matching. Tencent HY-Motion 1.0 empowers developers and individual creators alike by transforming natural language into high-fidelity, fluid, and diverse 3D character animations, delivering exceptional instruction-following capabilities across a broad range of categories. The generated 3D animation assets can be seamlessly integrated into typical 3D animation pipelines.🎮🎥 Highlights: 🔹Billion-Scale DiT: Successfully scaled flow-matching DiT to 1B+ parameters, setting a new ceiling for instruction-following capability and generated motion quality. 🔹Full-Stage Training Strategy: The industry’s first motion generation model featuring a complete Pre-training → SFT → RL loop to optimize physical plausibility and semantic accuracy. 🔹Comprehensive Category Coverage: Features 200+ motion categories across 6 major classes—the most comprehensive in the industry, curated via a meticulous data pipeline. 🌐Project Page: 🔗Github: 🤗Hugging Face: 📄Technical report:

Tencent Hy

329,174 次观看 • 8 个月前

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.

evo-devo

58,452 次观看 • 1 年前

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail failure cases is not possible. To solve the evaluation challenge, Tesla has developed a World Simulator Model. It is trained on state and action pairs, so for a given current camera feed and current action, the model generates the next frame in the cameras. Once the world simulation is good enough, it can be hooked up to the policy network for a closed-loop simulation. In this setup, the model generates the next video frames, and based on those, the policy network comes up with the next action, which feeds back into the world simulation; this continuous loop creates a coherent simulation. The world simulation neural network is not just applicable to self-driving but also to humanoids, because Tesla trains it on common data. The examples show Optimus navigating or manipulating inside the world simulation, with all the pixels being generated by the world simulation model. The robotic agents can be tested in an accurately reproduced edge case and in other potential variations of the same edge case.

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail failure cases is not possible. To solve the evaluation challenge, Tesla has developed a World Simulator Model. It is trained on state and action pairs, so for a given current camera feed and current action, the model generates the next frame in the cameras. Once the world simulation is good enough, it can be hooked up to the policy network for a closed-loop simulation. In this setup, the model generates the next video frames, and based on those, the policy network comes up with the next action, which feeds back into the world simulation; this continuous loop creates a coherent simulation. The world simulation neural network is not just applicable to self-driving but also to humanoids, because Tesla trains it on common data. The examples show Optimus navigating or manipulating inside the world simulation, with all the pixels being generated by the world simulation model. The robotic agents can be tested in an accurately reproduced edge case and in other potential variations of the same edge case.

The Humanoid Hub

17,065 次观看 • 6 个月前

Trained a humanoid entirely in a 3D scan of the office. Zero real-world fine-tuning. It just walked in and worked. RL needs hundreds of thousands of attempts, and real robots can't afford to crash. A misjudged gap or a glass door collision breaks hardware and costs hours resetting. So you train in a sim. But sim policies usually train on randomized, untextured geometry; depth is easy to fake. The robot learns structure, not the real world: no materials, no lighting, no idea what anything actually is. RGB cameras carry all of that but training RGB policies in generic fake worlds won’t generalize to the real world. Niantic Spatial 🌎 Scaniverse reconstructs your scan of the real deployment site. One 360° camera walkthrough → photorealistic 3D Gaussian splat at metric scale → collision mesh pulled from the same reconstruction, so vision and physics match exactly. Drops straight into NVIDIA Isaac Sim/Lab, no manual conversion. Flexion simulation-first approach then seamlessly enables the training of RGB-only nav policies inside that reconstruction. With added domain randomization + large image encoders for robustness, this deploys straight to hardware. No real-world fine-tuning. Deployment: months of on-site adaptation → days. Tune into the NVIDIA livestream on 12 August to hear how these companies are closing the sim2real gap: NVIDIA Robotics ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Trained a humanoid entirely in a 3D scan of the office. Zero real-world fine-tuning. It just walked in and worked. RL needs hundreds of thousands of attempts, and real robots can't afford to crash. A misjudged gap or a glass door collision breaks hardware and costs hours resetting. So you train in a sim. But sim policies usually train on randomized, untextured geometry; depth is easy to fake. The robot learns structure, not the real world: no materials, no lighting, no idea what anything actually is. RGB cameras carry all of that but training RGB policies in generic fake worlds won’t generalize to the real world. Niantic Spatial 🌎 Scaniverse reconstructs your scan of the real deployment site. One 360° camera walkthrough → photorealistic 3D Gaussian splat at metric scale → collision mesh pulled from the same reconstruction, so vision and physics match exactly. Drops straight into NVIDIA Isaac Sim/Lab, no manual conversion. Flexion simulation-first approach then seamlessly enables the training of RGB-only nav policies inside that reconstruction. With added domain randomization + large image encoders for robustness, this deploys straight to hardware. No real-world fine-tuning. Deployment: months of on-site adaptation → days. Tune into the NVIDIA livestream on 12 August to hear how these companies are closing the sim2real gap: NVIDIA Robotics ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

119,024 次观看 • 18 天前

This BlenderFusion paper basically says "screw trying to describe 3D edits through text" and just... use Blender :-) The idea is pretty straightforward -- instead of trying to cram 3D understanding into a diffusion model, use depth estimation & segmentation to project 2D images into 2.5D meshes, edit them in actual 3D software, then use a fine-tuned diffusion model to make the results photorealistic again. The clever bit is their "dual-stream architecture" -- the model sees both the original scene AND the edited Blender render in parallel, learning to preserve what matters while fixing the inevitable artifacts from transforming imperfect 2.5D/3D reconstructions. They train it with smart masking strategies so it learns when to ignore the original scene (for removals/replacements) and can manipulate objects independently of camera motion. What you get is pretty impressive control -- not just moving objects around, but changing materials, deforming shapes, swapping backgrounds, all while maintaining visual coherence. Neural Assets (one of my favorite papers last year) tried to crack this with learned object tokens, but it struggled with overlapping objects and loses fine details (due to low res DINO encodings). BlenderFusion just sidesteps the whole problem -- want to rotate something 173.5 degrees? Just rotate it in Blender. Want to duplicate an object 8 times? Copy paste away. The diffusion model's only job is making it look photorealistic, not figuring out the 3D underpinnings. The catch? Lacks temporal consistency for animation. Each viewpoint is generated independently, so while a single edit looks great, smoothly animating a car or camera down the street won't work -- you'd get flickering and inconsistencies between frames. That said, this approach is so much more intuitive for finer grain image editing than trying to describe your changes in text prompts. It's the kind of thing that makes you wonder why we're trying to do everything inside neural networks when perfectly good 3D tools already exist -- giving you the best of both worlds.

This BlenderFusion paper basically says "screw trying to describe 3D edits through text" and just... use Blender :-) The idea is pretty straightforward -- instead of trying to cram 3D understanding into a diffusion model, use depth estimation & segmentation to project 2D images into 2.5D meshes, edit them in actual 3D software, then use a fine-tuned diffusion model to make the results photorealistic again. The clever bit is their "dual-stream architecture" -- the model sees both the original scene AND the edited Blender render in parallel, learning to preserve what matters while fixing the inevitable artifacts from transforming imperfect 2.5D/3D reconstructions. They train it with smart masking strategies so it learns when to ignore the original scene (for removals/replacements) and can manipulate objects independently of camera motion. What you get is pretty impressive control -- not just moving objects around, but changing materials, deforming shapes, swapping backgrounds, all while maintaining visual coherence. Neural Assets (one of my favorite papers last year) tried to crack this with learned object tokens, but it struggled with overlapping objects and loses fine details (due to low res DINO encodings). BlenderFusion just sidesteps the whole problem -- want to rotate something 173.5 degrees? Just rotate it in Blender. Want to duplicate an object 8 times? Copy paste away. The diffusion model's only job is making it look photorealistic, not figuring out the 3D underpinnings. The catch? Lacks temporal consistency for animation. Each viewpoint is generated independently, so while a single edit looks great, smoothly animating a car or camera down the street won't work -- you'd get flickering and inconsistencies between frames. That said, this approach is so much more intuitive for finer grain image editing than trying to describe your changes in text prompts. It's the kind of thing that makes you wonder why we're trying to do everything inside neural networks when perfectly good 3D tools already exist -- giving you the best of both worlds.

Bilawal Sidhu

34,440 次观看 • 1 年前

World Model is trending— let's revisit our HunyuanWorld journey. We’ve been pioneering open-source 3D world generation in the past two months, and this ride’s only getting started. 🌍 📅 July: HunyuanWorld 1.0 📌 First open-source 3D world model compatible with CG pipelines (Unity/Unreal/Blender) 📌 Hit 2K+ GitHub stars in just two months ⭐—thank you for the love! 📅 August: 1.0-Lite 📌Same top-tier quality, running on consumer GPUs! 📅 September: 1.0-Voyager 📌 Direct 3D output + world memory—taking exploration further! Seamlessly integrated into CG pipelines with layered 3D modeling (assets, terrain, skybox) and fully open-sourced.. we’re fully committed to building open-source spatial intelligence for all! 🚀 💡 Why it matters? ✅ Seamless CG Pipeline Integration: Export generated 3D scenes as standard mesh formats, effortlessly integrating into industry-standard tools like Blender, Unity, and Unreal Engine for direct editing, animation, and physical simulation. ✅ Hierarchical Scene Editing: Deconstruct scenes into semantic layers (sky, background, foreground objects) via instance recognition and layer decomposition, allowing for atomic-level control—independently modify, relocate, or replace objects without rebuilding the entire world. Project page: Github: Amazing creations by Stijn Spanhove camenduru GENEL | AIを用いた動画制作 apolinario 🌐 とりにく Directive Creator 🪥 👇 #AI #3DGeneration #OpenSource #WorldModels #Hunyuan3D #HunyuanWorld

World Model is trending— let's revisit our HunyuanWorld journey. We’ve been pioneering open-source 3D world generation in the past two months, and this ride’s only getting started. 🌍 📅 July: HunyuanWorld 1.0 📌 First open-source 3D world model compatible with CG pipelines (Unity/Unreal/Blender) 📌 Hit 2K+ GitHub stars in just two months ⭐—thank you for the love! 📅 August: 1.0-Lite 📌Same top-tier quality, running on consumer GPUs! 📅 September: 1.0-Voyager 📌 Direct 3D output + world memory—taking exploration further! Seamlessly integrated into CG pipelines with layered 3D modeling (assets, terrain, skybox) and fully open-sourced.. we’re fully committed to building open-source spatial intelligence for all! 🚀 💡 Why it matters? ✅ Seamless CG Pipeline Integration: Export generated 3D scenes as standard mesh formats, effortlessly integrating into industry-standard tools like Blender, Unity, and Unreal Engine for direct editing, animation, and physical simulation. ✅ Hierarchical Scene Editing: Deconstruct scenes into semantic layers (sky, background, foreground objects) via instance recognition and layer decomposition, allowing for atomic-level control—independently modify, relocate, or replace objects without rebuilding the entire world. Project page: Github: Amazing creations by Stijn Spanhove camenduru GENEL | AIを用いた動画制作 apolinario 🌐 とりにく Directive Creator 🪥 👇 #AI #3DGeneration #OpenSource #WorldModels #Hunyuan3D #HunyuanWorld

Tencent HY

20,178 次观看 • 11 个月前

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025

🚨 SIGGRAPH Asia 2025 Paper Alert 🚨 ➡️Paper Title: WorldExplorer: Towards Generating Fully Navigable 3D Scenes 🌟Few pointers from the paper 🎯Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce stretched-out and noisy artifacts when moving beyond central or panoramic perspectives. 🎯 To this end, authors of this paper proposed “WorldExplorer”, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. 🎯They initialize their scenes by creating multi-view consistent images corresponding to a 360 degree panorama. 🎯Then, they expanded it by leveraging video diffusion models in an iterative scene generation pipeline. 🎯Concretely, they generated multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. 🎯Their novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. 🎯Finally,they fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. 🎯Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. 🎯They believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments. 🏢Organization: TU München 🧙Paper Authors: Manuel-Andreas Schneider, Lukas Höllein , Matthias Niessner 📝 Read the Full Paper here: 🗂️ Project Page: 🧑‍💻 Code: 🎥 Be sure to watch the attached Technical Summary Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #SIGGRAPHAsia2025

naveen manwani

10,578 次观看 • 11 个月前