Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Introducing ✨Posed DROID✨, results of our efforts at automatic post-hoc calibration of a large-scale robotics manipulation dataset. We provide: 🤖 ~36k calibrated episodes with good quality extrinsic calibration 🦾 ~24k calibrated multi-view episodes with good-quality multi-view camera calibration ✅ Quality assessment metrics for all provided camera poses To achieve... show more

Zubair Irshad

1,551 subscribers

13,579 Aufrufe • vor 1 Jahr •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

8 Kommentare

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

Below we show the Camera-to-Camera transformations, post-calibration improves the alignment of obtained pointclouds! 2/n

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

Automatically calibrating a large-scale dataset is challenging. We provide quality assessment metrics with all three stages with the flexibility to narrow out bounds for downstream tasks as needed. 1️⃣ and 2️⃣ quality assessment metrics shows distribution of IOUs and Reprojection-error post-calibraton. 3/n

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

Similarly, we plot the distribution of number of matched points and cumulative curve after 3️⃣, helping to identify the top quantile of well-calibrated camera pairs within each lab. 4/n

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

Automatic calibration robotics at large-scale is challenging and while successful, our pipeline has some limitations: • CtRNet-X was trained on the Franka Panda robot. Its performance i.e. zero-shot generalizability on other robot types remains to be seen. • DUSt3R, while powerful, still struggles in scenes with heavy clutter or minimal overlap between views. • False positives are inevitably observed for Steps 2️⃣ and 3️⃣, especially in challenging lighting or geometry. There’s room for improvement. Future work could include: • Extending it to in-the-wild scenes i.e. by leveraging foundation models to perform general-purpose robot segmentation and keypoint detection. • Ensembling predictions across time to improve temporal consistency. • Fine-tuning pointmap prediction models on real robot data to better handle cluttered tabletop environments. 5/n

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

We have released our improved extrinsics. Try it out now at and read more details about it in the updated DROID paper at This was a fun collaboration with @vitorguizilini @SashaKhazatsky and @KarlPertsch!

Profilbild von Zubair Irshad

Zubair Irshadvor 1 Jahr

@SashaKhazatsky @KarlPertsch Shoutout to the authors of the wonderful papers i.e. CtRNet-X, DUSt3R, Segment Anything, CLIP and Pytorch3D and for open-sourcing their codebase to advance science and make this effort happen! Please check these works out if you haven’t already!

Profilbild von Saketh Saketh

Saketh Sakethvor 1 Jahr

Great work

Profilbild von Rainmaker

Rainmakervor 2 Jahren

In this free Substack post I share code for several machine learning models and engage in hyperparameter tuning that yields a model that delivers superior returns in the Gold market.

Ähnliche Videos

This seemingly obvious prediction didn't take long to become reality. MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors Contributions: • The first real-time SLAM system using the two-view 3D reconstruction prior MASt3R [20] as a foundation. • Efficient techniques for pointmap matching, tracking and local fusion, graph construction and loop closure, and second-order global optimization. • A state-of-the-art dense SLAM system capable of handling generic, time-varying camera models. Abstract: We present a real-time monocular dense SLAM system, designed from the ground up using MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system remains robust on in-the-wild video sequences, making no assumptions on a fixed or parametric camera model beyond a unique camera center. Key features include: - Efficient methods for pointmap matching, camera tracking, and local fusion - Graph construction and loop closure - Second-order global optimization With known calibration, a simple modification achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

This seemingly obvious prediction didn't take long to become reality. MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors Contributions: • The first real-time SLAM system using the two-view 3D reconstruction prior MASt3R [20] as a foundation. • Efficient techniques for pointmap matching, tracking and local fusion, graph construction and loop closure, and second-order global optimization. • A state-of-the-art dense SLAM system capable of handling generic, time-varying camera models. Abstract: We present a real-time monocular dense SLAM system, designed from the ground up using MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system remains robust on in-the-wild video sequences, making no assumptions on a fixed or parametric camera model beyond a unique camera center. Key features include: - Efficient methods for pointmap matching, camera tracking, and local fusion - Graph construction and loop closure - Second-order global optimization With known calibration, a simple modification achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

MrNeRF

29,974 Aufrufe • vor 1 Jahr

Wow. Recreating the Shawshank Redemption prison in 3D from a single video, in real time (!) Just read the MASt3R-SLAM paper and it's pretty neat. These folks basically built a real-time dense SLAM system on top of MASt3R, which is a transformer-based neural network that can do 3d reconstruction and localization from uncalibrated image pairs. The cool part is they don't need a fixed camera model -- it just works with arbitrary cameras -- think different focal lengths, sensor sizes, even handling zooming in video (FMV drone video anyone?!). If you've done photogrammetry or played with NeRFs you know that is a HUGE deal. They've solved some tricky problems like efficient point matching and tracking, plus they've figured out how to fuse point clouds and handle loop closures in real-time. Their system runs at about 15 FPS on a 4090 and produces both camera poses and dense geometry. When they know the camera calibration, they get SOTA results across several benchmarks, but even without calibration, they still perform well. What's interesting is the approach -- most recent SLAM work has built on DROID-SLAM's architecture, but these folks went a different direction by leveraging a strong 3D reconstruction prior. Seems to give them more coherent geometry, which makes sense since that's what MASt3R was designed for. For anyone who cares about monocular SLAM and 3D reconstruction, this feels like a significant step toward plug-and-play dense SLAM without calibration headaches -- perfect for drones, robots, AR/VR -- the works!

Wow. Recreating the Shawshank Redemption prison in 3D from a single video, in real time (!) Just read the MASt3R-SLAM paper and it's pretty neat. These folks basically built a real-time dense SLAM system on top of MASt3R, which is a transformer-based neural network that can do 3d reconstruction and localization from uncalibrated image pairs. The cool part is they don't need a fixed camera model -- it just works with arbitrary cameras -- think different focal lengths, sensor sizes, even handling zooming in video (FMV drone video anyone?!). If you've done photogrammetry or played with NeRFs you know that is a HUGE deal. They've solved some tricky problems like efficient point matching and tracking, plus they've figured out how to fuse point clouds and handle loop closures in real-time. Their system runs at about 15 FPS on a 4090 and produces both camera poses and dense geometry. When they know the camera calibration, they get SOTA results across several benchmarks, but even without calibration, they still perform well. What's interesting is the approach -- most recent SLAM work has built on DROID-SLAM's architecture, but these folks went a different direction by leveraging a strong 3D reconstruction prior. Seems to give them more coherent geometry, which makes sense since that's what MASt3R was designed for. For anyone who cares about monocular SLAM and 3D reconstruction, this feels like a significant step toward plug-and-play dense SLAM without calibration headaches -- perfect for drones, robots, AR/VR -- the works!

Bilawal Sidhu

704,124 Aufrufe • vor 1 Jahr

Volumetric Capture is Hard There is no magic solution. Anyone interested in spatial R&D can get access to some awesome datasets online to test with, which can be especially interesting when processed with 3DGS and played back as 4DGS. However you need a ton of compute power. (multiple 3080, 3090+, RTX5000+ etc) The Necessity of Multiple Cameras for Volumetric Capture Consider, for instance, this very useful fixed rig dataset comprising 20 camera streams provided by Facebook Research Or this datasets comprising 40 camera streams provided by Google The attached video demonstrates FB's RAW 3DGS data played back in real-time in our custom Sibr viewer. It's noticable how quickly the quality can breakdown out of the camera group frustrum. Even at this quality it takes a lot of time and energy to process and playback. It's not simple. Let alone the expertise required by the original research team who captured the data. Doing this with 20 cameras is hard. 150+ cameras even harder. Now imagine trying to achieve it with a single camera phone to playback on a Meta Quest or Spatial headset. #volumetriccaptureishard #camerasdontseearoundcorners #spatial

Volumetric Capture is Hard There is no magic solution. Anyone interested in spatial R&D can get access to some awesome datasets online to test with, which can be especially interesting when processed with 3DGS and played back as 4DGS. However you need a ton of compute power. (multiple 3080, 3090+, RTX5000+ etc) The Necessity of Multiple Cameras for Volumetric Capture Consider, for instance, this very useful fixed rig dataset comprising 20 camera streams provided by Facebook Research Or this datasets comprising 40 camera streams provided by Google The attached video demonstrates FB's RAW 3DGS data played back in real-time in our custom Sibr viewer. It's noticable how quickly the quality can breakdown out of the camera group frustrum. Even at this quality it takes a lot of time and energy to process and playback. It's not simple. Let alone the expertise required by the original research team who captured the data. Doing this with 20 cameras is hard. 150+ cameras even harder. Now imagine trying to achieve it with a single camera phone to playback on a Meta Quest or Spatial headset. #volumetriccaptureishard #camerasdontseearoundcorners #spatial

Infinite-Realities

129,521 Aufrufe • vor 2 Jahren

[SIGGRAPH 2025] Photoreal Scene Reconstruction from an Egocentric Device Contributions: 1. We address the importance of employing visual-inertial bundle adjustment (VIBA) that accounts for the rolling-shutter behavior of the RGB camera. This provides a continuous camera trajectory to model pixel movement in neural reconstruction. Our experiments demonstrate that using VIBA consistently improves the novel view quality in Gaussian Splatting by +1 dB in PSNR. 2. We introduce a rasterization-based image formulation pipeline that addresses common artifacts in physical image formation, including rolling shutter, lens shading, exposure, and gain compensation. Our approach is distinct in that we represent image poses as posed pixel arrays sampled from a continuous trajectory, rather than assigning a single camera pose per image, and preserve the merit of Gaussian rasterization. Unlike existing methods that require ray-tracing Gaussians, e.g., [Moenne-Loccoz et al. 2024], our formulation is applicable to general-purpose rasterization-based Gaussian splatting. When applied to 3D Gaussian Splatting (3DGS) [Kerbl et al. 2023], our approach can further enhance reconstruction quality by +1 dB. We outperform existing baselines and demonstrate a substantial quality improvement in handling complex scenes observed by egocentric devices. 3. To reduce the effect of blur from rapid head motion in darker indoor scenes, we propose a strategy of deliberately underexposing input videos during capture, inspired by HDR+ [Hasinoff et al. 2016]. We demonstrate that we can reconstruct high-quality, noise-free scene radiance from noisy, dim input videos, and further render sharp, blur-free videos at a higher dynamic range.

[SIGGRAPH 2025] Photoreal Scene Reconstruction from an Egocentric Device Contributions: 1. We address the importance of employing visual-inertial bundle adjustment (VIBA) that accounts for the rolling-shutter behavior of the RGB camera. This provides a continuous camera trajectory to model pixel movement in neural reconstruction. Our experiments demonstrate that using VIBA consistently improves the novel view quality in Gaussian Splatting by +1 dB in PSNR. 2. We introduce a rasterization-based image formulation pipeline that addresses common artifacts in physical image formation, including rolling shutter, lens shading, exposure, and gain compensation. Our approach is distinct in that we represent image poses as posed pixel arrays sampled from a continuous trajectory, rather than assigning a single camera pose per image, and preserve the merit of Gaussian rasterization. Unlike existing methods that require ray-tracing Gaussians, e.g., [Moenne-Loccoz et al. 2024], our formulation is applicable to general-purpose rasterization-based Gaussian splatting. When applied to 3D Gaussian Splatting (3DGS) [Kerbl et al. 2023], our approach can further enhance reconstruction quality by +1 dB. We outperform existing baselines and demonstrate a substantial quality improvement in handling complex scenes observed by egocentric devices. 3. To reduce the effect of blur from rapid head motion in darker indoor scenes, we propose a strategy of deliberately underexposing input videos during capture, inspired by HDR+ [Hasinoff et al. 2016]. We demonstrate that we can reconstruct high-quality, noise-free scene radiance from noisy, dim input videos, and further render sharp, blur-free videos at a higher dynamic range.

MrNeRF

15,244 Aufrufe • vor 1 Jahr

2D is what the camera sees.👀 3D is what the world is. 🌍 Today, we’re proudly open-sourcing QuerySplat — an open-source feed-forward 3D model that turns a few unposed images into a navigable 3D scene in seconds (run on 4090🪶). Your input: a few unposed images Our output: a sharp, navigable 3D scene — generated in seconds! 🚀 🧠 organize scenes with 3D queries, not pixel-bound Gaussians 📷 no camera poses or manual calibration required ⚡ reconstruct new scenes in a single forward pass ✨ preserve sharp geometry and high-frequency appearance 🏆 achieve state-of-the-art results on DL3DV-Evaluation 🔓 code and model weights are now open-source From pixels to scenes, we are making 3D generation scalable. Try our free iOS app — it’s a lot of fun! 😆 🔗 Try it on the Web: 🔗 Paper: 🔗 Code & weights: 🔗 Project page: 🔗 #AIGC #3DGS #AI #Computervision #3D #artist #3dartist #3d

2D is what the camera sees.👀 3D is what the world is. 🌍 Today, we’re proudly open-sourcing QuerySplat — an open-source feed-forward 3D model that turns a few unposed images into a navigable 3D scene in seconds (run on 4090🪶). Your input: a few unposed images Our output: a sharp, navigable 3D scene — generated in seconds! 🚀 🧠 organize scenes with 3D queries, not pixel-bound Gaussians 📷 no camera poses or manual calibration required ⚡ reconstruct new scenes in a single forward pass ✨ preserve sharp geometry and high-frequency appearance 🏆 achieve state-of-the-art results on DL3DV-Evaluation 🔓 code and model weights are now open-source From pixels to scenes, we are making 3D generation scalable. Try our free iOS app — it’s a lot of fun! 😆 🔗 Try it on the Web: 🔗 Paper: 🔗 Code & weights: 🔗 Project page: 🔗 #AIGC #3DGS #AI #Computervision #3D #artist #3dartist #3d

InSpatio｜影溯

29,682 Aufrufe • vor 1 Monat

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

GS^3: Efficient Relighting with Triple Gaussian Splatting Abstract: We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex ap pearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance.

MrNeRF

17,811 Aufrufe • vor 2 Jahren

This is #GoProMISSION1 PRO 🎥 The only 8K60 camera with a 1-inch sensor. Our compact, cinema-grade camera features a proprietary GP3 processor and 50MP sensor that enable intelligent low-light capture, industry-leading frame rates and resolutions, and groundbreaking thermal performance. ✔️ 1-inch Quad-Bayer sensor with up to 14-stops of dynamic range at the sensor for low-light capture ✔️ Longest continuous runtimes + most dependable thermal performance of any GoPro ever—over 5 hours in 1080p + over 3 hours in 4K at 100°F ✔️ Industry-leading 8K60—300% more pixels than 4K ✔️ 4K240 + 1080p960 ultra slo-mo with real frames—not AI-interpolated ✔️ 8K30 + 4K120 Open Gate capture ✔️ Gallery-ready 50MP photos + 44MP frame grabs ✔️ Up to 240 Mbps bit rate out of the box + 300 Mbps with GoPro Labs ✔️ 10-Bit color + GP-Log2 with LUTs for Rec.709 + Rec.2020 outputs ✔️ HLG HDR with Simultaneous Dual-Gain Readout—the industry standard for pros ✔️ New intelligent capture modes: Dive, Vlog, Low-Light, Sport POV, + Subject Tracking ✔️ 13% higher capacity Enduro 2 battery in the same form factor with new fast charging ✔️ Rugged + waterproof, now to 66ft (20m) without a housing ✔️ Emmy® Award Winning #HyperSmooth in-camera video stabilization ✔️ New 4-microphone array, 32-bit float audio, multi-track recording, + manual audio controls ✔️ Timecode Sync to streamline multi-camera editing + GPS with telemetry data ✔️ New Point-and-Shoot Grip compatibility for elite handheld control ✔️ Removable Lens Hood included to reduce glare + flares ✔️ Bluetooth® 5.3 Super Wideband connectivity + USB-C port for external audio capture ✔️ A cinema-grade camera that anybody can use Enhanced by a GoPro Subscription: ✔️ Unlimited cloud backup at 100% quality ✔️ Camera replacement guarantee ✔️ Up to 50% off select accessories Order your MISSION 1 Series camera now to get a free Point-and-Shoot Grip ($100 value) + free shipping at Pro-tip: Existing GoPro Subscribers save $100 with the annual camera discount.

This is #GoProMISSION1 PRO 🎥 The only 8K60 camera with a 1-inch sensor. Our compact, cinema-grade camera features a proprietary GP3 processor and 50MP sensor that enable intelligent low-light capture, industry-leading frame rates and resolutions, and groundbreaking thermal performance. ✔️ 1-inch Quad-Bayer sensor with up to 14-stops of dynamic range at the sensor for low-light capture ✔️ Longest continuous runtimes + most dependable thermal performance of any GoPro ever—over 5 hours in 1080p + over 3 hours in 4K at 100°F ✔️ Industry-leading 8K60—300% more pixels than 4K ✔️ 4K240 + 1080p960 ultra slo-mo with real frames—not AI-interpolated ✔️ 8K30 + 4K120 Open Gate capture ✔️ Gallery-ready 50MP photos + 44MP frame grabs ✔️ Up to 240 Mbps bit rate out of the box + 300 Mbps with GoPro Labs ✔️ 10-Bit color + GP-Log2 with LUTs for Rec.709 + Rec.2020 outputs ✔️ HLG HDR with Simultaneous Dual-Gain Readout—the industry standard for pros ✔️ New intelligent capture modes: Dive, Vlog, Low-Light, Sport POV, + Subject Tracking ✔️ 13% higher capacity Enduro 2 battery in the same form factor with new fast charging ✔️ Rugged + waterproof, now to 66ft (20m) without a housing ✔️ Emmy® Award Winning #HyperSmooth in-camera video stabilization ✔️ New 4-microphone array, 32-bit float audio, multi-track recording, + manual audio controls ✔️ Timecode Sync to streamline multi-camera editing + GPS with telemetry data ✔️ New Point-and-Shoot Grip compatibility for elite handheld control ✔️ Removable Lens Hood included to reduce glare + flares ✔️ Bluetooth® 5.3 Super Wideband connectivity + USB-C port for external audio capture ✔️ A cinema-grade camera that anybody can use Enhanced by a GoPro Subscription: ✔️ Unlimited cloud backup at 100% quality ✔️ Camera replacement guarantee ✔️ Up to 50% off select accessories Order your MISSION 1 Series camera now to get a free Point-and-Shoot Grip ($100 value) + free shipping at Pro-tip: Existing GoPro Subscribers save $100 with the annual camera discount.

GoPro

29,069 Aufrufe • vor 3 Monaten

Wim Wenders on the "camera movement and blocking" in Wings of Desire (1987): Filmmaker Magazine: "What was your philosophy about camera movement and blocking in Wings of Desire?" Wenders: "As we very often had to “translate” the angels’ point of view, so to speak, we were extremely keen on moving the camera as much as possible. In the absence of Steadicam equipment we worked a lot on tracks, with dollies, cranes, jib-arms etc. But we also built ourselves devices so we could move through the air from one house to the next, for instance, and we shot the opening sequence on a helicopter, which was highly difficult in West Berlin at the time, as there were no private companies flying, just the Allies with their respective army pilots. We ended up shooting with a British pilot in an army helicopter without a proper camera mount. Today, you would do these things with gyroscopes and such. Blocking has always been my department. Henri [Alekan] kept out of it completely, and I did it with his operator, Agnès Godard. I have done shot lists for complicated sets, but usually I decide on location in the morning how we design the shots. I prefer to see the actors rehearse it, before I commit to any blocking. Camera moves weren’t the real challenge, though, for finding the angels’ points of view. It dawned on me early on that our camera had to do a more complex job. I told it to Henri. “Those angels have a very loving look at us humans. We have to find a way to teach our camera to look more lovingly.” Henri just stared at me as if I was out of my mind. “How do we do that?” Well, I didn’t know of course. But I figured we had to invest more care and love ourselves into every shot that represented what the angels saw. And that’s it, in the end. A camera can reflect on what you invest into its act of seeing. That sounds pretty lofty, I guess. But it does rub off, I tell you, if you try to imagine how angels would look at us. After all, they were some sort of metaphor for me for the better persons we carry inside ourselves, or for the children we somehow preserved in ourselves." — "“Imagine How Angels Would Look at Us”: Wim Wenders on Restoring Wings of Desire" by Jim Hemphill (Filmmaker Magazine, 2018)

Wim Wenders on the "camera movement and blocking" in Wings of Desire (1987): Filmmaker Magazine: "What was your philosophy about camera movement and blocking in Wings of Desire?" Wenders: "As we very often had to “translate” the angels’ point of view, so to speak, we were extremely keen on moving the camera as much as possible. In the absence of Steadicam equipment we worked a lot on tracks, with dollies, cranes, jib-arms etc. But we also built ourselves devices so we could move through the air from one house to the next, for instance, and we shot the opening sequence on a helicopter, which was highly difficult in West Berlin at the time, as there were no private companies flying, just the Allies with their respective army pilots. We ended up shooting with a British pilot in an army helicopter without a proper camera mount. Today, you would do these things with gyroscopes and such. Blocking has always been my department. Henri [Alekan] kept out of it completely, and I did it with his operator, Agnès Godard. I have done shot lists for complicated sets, but usually I decide on location in the morning how we design the shots. I prefer to see the actors rehearse it, before I commit to any blocking. Camera moves weren’t the real challenge, though, for finding the angels’ points of view. It dawned on me early on that our camera had to do a more complex job. I told it to Henri. “Those angels have a very loving look at us humans. We have to find a way to teach our camera to look more lovingly.” Henri just stared at me as if I was out of my mind. “How do we do that?” Well, I didn’t know of course. But I figured we had to invest more care and love ourselves into every shot that represented what the angels saw. And that’s it, in the end. A camera can reflect on what you invest into its act of seeing. That sounds pretty lofty, I guess. But it does rub off, I tell you, if you try to imagine how angels would look at us. After all, they were some sort of metaphor for me for the better persons we carry inside ourselves, or for the children we somehow preserved in ourselves." — "“Imagine How Angels Would Look at Us”: Wim Wenders on Restoring Wings of Desire" by Jim Hemphill (Filmmaker Magazine, 2018)

RadiantFilm

18,101 Aufrufe • vor 6 Monaten

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Marktechpost AI

39,955 Aufrufe • vor 1 Monat

Colmap 4.0 was very recently released, so it inspired me to do some work to better understand it and its new capabilities with Rerun. I want to really understand how Colmap, and in particular, pycolmap, works outside of just calling it via the CLI. So my goal is to use the low-level pycolmap API to log every part of the pipeline. The explicit goal is to have an alternative to the SQLite database that I can utilize. Instead of SQLite, I want to try logging everything directly to rerun and use RRD. This means I can have deep inspectability and still save the features/matches/2D view geometry, but be able to view it directly in rerun. I think this is one of the superpowers that rerun provides; data and visualizations are deeply integrated. As I'm often working with sequential data (videos), I'm going to specifically focus on four things: 1. Monocular Video Simple: Calls high-level APIs such as pycolmap.extract_features, pycolmap.match_sequential, pycolmap.incremental_mapping. These are basically identical to the CLI options and provide a good baseline. 2. Monocular Video Streamed: Take the above high-level APIs and break them down to their iterator version, logging each component in a streamed manner. This way, I can stream the intermediate features to rerun while the extraction/matching/mapping is happening. 3. Rig with unknown calibration: <- WHAT THE VIDEO SHOWS This is probably the most interesting version and the first one I've been working on. It allows one to set a rig between known sensors, such as in VR/AR devices, leading to much better reconstructions with multiple cameras. This is the case where we don't know the calibration a priori, so we have to run a reconstruction twice: once as a normal Colmap reconstruction with no rig constraints, use this to generate the constraints, and then do it again with the newly found rig. 4. Rig with known calibration: This is the RoboCap example, where we have a pre-calibrated set of sensors, so we don't need to run the two reconstructions and also gain better matching between cameras, both spatially and temporally. Again, this leads to a much better reconstruction! Along with all this, GLOMAP has become a first-class global mapper, making it super easy to use directly within pycolmap! I'm excited to do more with this and compare it to things like pycuvslam, vipe, and other alternatives.

Colmap 4.0 was very recently released, so it inspired me to do some work to better understand it and its new capabilities with Rerun. I want to really understand how Colmap, and in particular, pycolmap, works outside of just calling it via the CLI. So my goal is to use the low-level pycolmap API to log every part of the pipeline. The explicit goal is to have an alternative to the SQLite database that I can utilize. Instead of SQLite, I want to try logging everything directly to rerun and use RRD. This means I can have deep inspectability and still save the features/matches/2D view geometry, but be able to view it directly in rerun. I think this is one of the superpowers that rerun provides; data and visualizations are deeply integrated. As I'm often working with sequential data (videos), I'm going to specifically focus on four things: 1. Monocular Video Simple: Calls high-level APIs such as pycolmap.extract_features, pycolmap.match_sequential, pycolmap.incremental_mapping. These are basically identical to the CLI options and provide a good baseline. 2. Monocular Video Streamed: Take the above high-level APIs and break them down to their iterator version, logging each component in a streamed manner. This way, I can stream the intermediate features to rerun while the extraction/matching/mapping is happening. 3. Rig with unknown calibration: <- WHAT THE VIDEO SHOWS This is probably the most interesting version and the first one I've been working on. It allows one to set a rig between known sensors, such as in VR/AR devices, leading to much better reconstructions with multiple cameras. This is the case where we don't know the calibration a priori, so we have to run a reconstruction twice: once as a normal Colmap reconstruction with no rig constraints, use this to generate the constraints, and then do it again with the newly found rig. 4. Rig with known calibration: This is the RoboCap example, where we have a pre-calibrated set of sensors, so we don't need to run the two reconstructions and also gain better matching between cameras, both spatially and temporally. Again, this leads to a much better reconstruction! Along with all this, GLOMAP has become a first-class global mapper, making it super easy to use directly within pycolmap! I'm excited to do more with this and compare it to things like pycuvslam, vipe, and other alternatives.

Pablo Vela

30,070 Aufrufe • vor 5 Monaten

#GoProMAX2 is here 🚨 Industry-leading, true 8K 360 video delivers 21% more resolution than the competition, resulting in unmatched image quality that you only get from #GoPro. ✔️ Emmy® Award-Winning 360 technology ✔️ The only true 8K 360 camera. No misleading upscaling, no unusable black pixels, no AI-generated content ✔️ Twist + go replaceable lenses made from water-repelling optical glass. No tools or calibration required to swap ✔️ 5.6K60 + 4K100 for up to 4x slo-mo in 360 ✔️ 10-Bit color, GP-Log encoding, + with GoPro Labs, category-leading 300mbps bitrate ✔️ Seamless, invisible pole shots thanks to a new, back-to-back lens design + built-in 1/4-20 mount ✔️ Easy AI-powered editing tools with intuitive Reframe modes that anyone can use ✔️ Automatic POV + Selfie modes for minimal editing, while retaining full 360 flexibility ✔️ Most-in-class 6 microphones that unlock true-to-life spatial audio with innovative Audio Field of View ✔️ 23% larger, 1960mAh cold-weather Enduro battery ✔️ 360 Night Effects for creative capture in the dark ✔️ Sleek form factor for noninvasive mounting in action sports ✔️ Unbreakable Max #HyperSmooth with 360° Horizon Lock ✔️ Rugged + waterproof to 16ft (5m) ✔️ Quick-release magnetic mounting + compatibility with GoPro's entire accessory ecosystem ✔️ Single lens 4K60 video in Max HyperView at 180° FOV ✔️ 29MP 360 photos for cropping + zooming without quality loss ✔️ Bluetooth® audio connectivity for wireless microphones ✔️ Lightning-fast transfer speeds to the GoPro Quik App with Wifi 6 + BLE 5.3 🇺🇸 Designed in the USA Enhanced by a GoPro Subscription: ✔️ AI-edited highlight videos automatically sent to your phone ✔️ Unlimited cloud storage at 100% quality ✔️ No-questions-asked camera replacement 📦 Pre-order today, with free shipping and a free 1-year GoPro Subscription at Orders will ship on or before September 30th. *MAX2 delivers up to 21% more video resolution compared to competitive 360 cameras’ native maximum video resolution before they up-scale.

#GoProMAX2 is here 🚨 Industry-leading, true 8K 360 video delivers 21% more resolution than the competition, resulting in unmatched image quality that you only get from #GoPro. ✔️ Emmy® Award-Winning 360 technology ✔️ The only true 8K 360 camera. No misleading upscaling, no unusable black pixels, no AI-generated content ✔️ Twist + go replaceable lenses made from water-repelling optical glass. No tools or calibration required to swap ✔️ 5.6K60 + 4K100 for up to 4x slo-mo in 360 ✔️ 10-Bit color, GP-Log encoding, + with GoPro Labs, category-leading 300mbps bitrate ✔️ Seamless, invisible pole shots thanks to a new, back-to-back lens design + built-in 1/4-20 mount ✔️ Easy AI-powered editing tools with intuitive Reframe modes that anyone can use ✔️ Automatic POV + Selfie modes for minimal editing, while retaining full 360 flexibility ✔️ Most-in-class 6 microphones that unlock true-to-life spatial audio with innovative Audio Field of View ✔️ 23% larger, 1960mAh cold-weather Enduro battery ✔️ 360 Night Effects for creative capture in the dark ✔️ Sleek form factor for noninvasive mounting in action sports ✔️ Unbreakable Max #HyperSmooth with 360° Horizon Lock ✔️ Rugged + waterproof to 16ft (5m) ✔️ Quick-release magnetic mounting + compatibility with GoPro's entire accessory ecosystem ✔️ Single lens 4K60 video in Max HyperView at 180° FOV ✔️ 29MP 360 photos for cropping + zooming without quality loss ✔️ Bluetooth® audio connectivity for wireless microphones ✔️ Lightning-fast transfer speeds to the GoPro Quik App with Wifi 6 + BLE 5.3 🇺🇸 Designed in the USA Enhanced by a GoPro Subscription: ✔️ AI-edited highlight videos automatically sent to your phone ✔️ Unlimited cloud storage at 100% quality ✔️ No-questions-asked camera replacement 📦 Pre-order today, with free shipping and a free 1-year GoPro Subscription at Orders will ship on or before September 30th. *MAX2 delivers up to 21% more video resolution compared to competitive 360 cameras’ native maximum video resolution before they up-scale.

GoPro

139,379 Aufrufe • vor 11 Monaten

Introducing WALL-WM, our open-source World Model for embodied AI and the next piece of our open-source robotics stack. Carving World Action Modeling at the Event Joints Read the blog: Why it matters WALL-WM shifts robot world modeling from fixed-length action chunks to event-grounded video-action pretraining. It learns around events like reaching, contact, grasping, lifting, moving, and placing, so language, vision, and action align more naturally. Why you should care WALL-WM brings together: •Event-grounded VLA pretraining •Prior-aligned video-action architecture •Wan-based video tower + randomly initialized action DiT •Multi-view perception with sight-cone masking, tube patch masking, and Camera RoPE •Event Mode for variable-length execution •Unified Mode with Staircase Decoding •DMuon for large-scale training The goal: help robots learn what physically matters, not just what happens in the next fixed slice of time. Code (coming soon): #opensource #EmbodiedAI

Introducing WALL-WM, our open-source World Model for embodied AI and the next piece of our open-source robotics stack. Carving World Action Modeling at the Event Joints Read the blog: Why it matters WALL-WM shifts robot world modeling from fixed-length action chunks to event-grounded video-action pretraining. It learns around events like reaching, contact, grasping, lifting, moving, and placing, so language, vision, and action align more naturally. Why you should care WALL-WM brings together: •Event-grounded VLA pretraining •Prior-aligned video-action architecture •Wan-based video tower + randomly initialized action DiT •Multi-view perception with sight-cone masking, tube patch masking, and Camera RoPE •Event Mode for variable-length execution •Unified Mode with Staircase Decoding •DMuon for large-scale training The goal: help robots learn what physically matters, not just what happens in the next fixed slice of time. Code (coming soon): #opensource #EmbodiedAI

X Square Robot

42,165 Aufrufe • vor 3 Monaten

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,323 Aufrufe • vor 2 Jahren

TEDRA Text-based Editing of Dynamic and Photoreal Actors discuss: Over the past years, significant progress has been made in creating photorealistic and drivable 3D avatars solely from videos of real humans. However, a core remaining challenge is the fine-grained and user-friendly editing of clothing styles by means of textual descriptions. To this end, we present TEDRA, the first method allowing text-based edits of an avatar, which maintains the avatar's high fidelity, space-time coherency, as well as dynamics, and enables skeletal pose and view control. We begin by training a model to create a controllable and high-fidelity digital replica of the real actor. Next, we personalize a pretrained generative diffusion model by fine-tuning it on various frames of the real character captured from different camera angles, ensuring the digital representation faithfully captures the dynamics and movements of the real person. This two-stage process lays the foundation for our approach to dynamic human avatar editing. Utilizing this personalized diffusion model, we modify the dynamic avatar based on a provided text prompt using our Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) within a model-based guidance framework. Additionally, we propose a time step annealing strategy to ensure high-quality edits. Our results demonstrate a clear improvement over prior work in functionality and visual quality.

AK

17,302 Aufrufe • vor 2 Jahren

Excited to share this demo from Over the Reality! Watch our Unitree Go2 robodog navigating our office while reconstructing the 3D space in real-time using the VGGT-based foundation vision model. This is a prime example of machine perception in action, turning raw RGB camera feeds into rich, detailed 3D maps! The robodog's RGB cam generates a dense, textured 3D reconstruction via VGGT from a few photograms, capturing nuances like object shapes and surfaces with impressive fidelity (main view). Compare that to the standard LiDAR system (top right), it's sparser, more point-cloud focused, lacking the visual richness. Vision models are closing the gap fast! What's powering this? VGGT, a cutting-edge foundation model for 3D perception, trained on datasets orders of magnitude smaller than our massive OVER 3D maps dataset. Imagine the leap when we apply OVER's scale to VGGT-like transformer based architectures, denser reconstructions, better generalization, revolutionary for robotics, machine perception & AR! Stay tuned for more breakthroughs at the intersection of AI, robotics, and DePIN. We're building the future of Physical AI and Spatial Computing at Over the Reality 🌐 What do you think, ready for robodogs in your world? Drop your thoughts! 🤖🌐

Excited to share this demo from Over the Reality! Watch our Unitree Go2 robodog navigating our office while reconstructing the 3D space in real-time using the VGGT-based foundation vision model. This is a prime example of machine perception in action, turning raw RGB camera feeds into rich, detailed 3D maps! The robodog's RGB cam generates a dense, textured 3D reconstruction via VGGT from a few photograms, capturing nuances like object shapes and surfaces with impressive fidelity (main view). Compare that to the standard LiDAR system (top right), it's sparser, more point-cloud focused, lacking the visual richness. Vision models are closing the gap fast! What's powering this? VGGT, a cutting-edge foundation model for 3D perception, trained on datasets orders of magnitude smaller than our massive OVER 3D maps dataset. Imagine the leap when we apply OVER's scale to VGGT-like transformer based architectures, denser reconstructions, better generalization, revolutionary for robotics, machine perception & AR! Stay tuned for more breakthroughs at the intersection of AI, robotics, and DePIN. We're building the future of Physical AI and Spatial Computing at Over the Reality 🌐 What do you think, ready for robodogs in your world? Drop your thoughts! 🤖🌐

Over the Reality 🌐

359,717 Aufrufe • vor 11 Monaten

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…

Grace Zhang

54,449 Aufrufe • vor 2 Monaten

Episode 5 of How I AI with some practical AI hacks for managers from hilary gridley, Head of Core Product at WHOOP. This one is all about feedback: how to scale it, how to make it specific, how to give it to AI tools. You’ll learn how to: - build rubrics for what “good” looks like - create GPTs to “think like you” and use them to scale quality feedback to your team - use prompting techniques to improve the insights you get from AI - improve your quality of writing and strategic thinking by bringing it all together Plus we gab about getting more women in AI, girly use cases for chatGPT, and whether or not getting invited to meetings is a good thing. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for genetic workflows ☑️ Vanta automate compliance and simplify security

Episode 5 of How I AI with some practical AI hacks for managers from hilary gridley, Head of Core Product at WHOOP. This one is all about feedback: how to scale it, how to make it specific, how to give it to AI tools. You’ll learn how to: - build rubrics for what “good” looks like - create GPTs to “think like you” and use them to scale quality feedback to your team - use prompting techniques to improve the insights you get from AI - improve your quality of writing and strategic thinking by bringing it all together Plus we gab about getting more women in AI, girly use cases for chatGPT, and whether or not getting invited to meetings is a good thing. Check it out in the links below 👇🔗 As always, a big thanks to our sponsors: 🔀 Orkes enterprise platform for genetic workflows ☑️ Vanta automate compliance and simplify security

claire vo 🖤

19,127 Aufrufe • vor 1 Jahr

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,106 Aufrufe • vor 1 Jahr

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Michael Black

21,696 Aufrufe • vor 1 Jahr

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,236 Aufrufe • vor 2 Jahren