Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Live monocular 3D reconstruction from camera feed, with multi-frame stitching and relative positioning (frame buffer of 4)... running on M3 mac cpu at 24fps... visualised using Rerun next up... realtime semantic navigation in 3d space...

ud

1,524 subscribers

22,080 Aufrufe • vor 6 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

🚀Introducing LLaVA-NeXT Interleave: Now AI can understand and reason with multiple images at once - This opens up multi-image scenarios like multi-frame videos, multi-view 3D, and multiple inter-leaved images. - An all round LMM that can understand videos, images, and 3D More⬇️

🚀Introducing LLaVA-NeXT Interleave: Now AI can understand and reason with multiple images at once - This opens up multi-image scenarios like multi-frame videos, multi-view 3D, and multiple inter-leaved images. - An all round LMM that can understand videos, images, and 3D More⬇️

Gradio

27,655 Aufrufe • vor 2 Jahren

Semantically annotating 3D gaussian splats on the fly using gemini 3.1 + sparkjs 1. Load any 3D scene and hit scan 2. Get 2D detections from VLM 3. Cluster outputs & project into 3D world space 4. Save as a persistent 3D semantic layer Inspired by Alexander Chen's experiments with gemini visual intelligence. Just had to try to lift it from 2D to 3D!

Semantically annotating 3D gaussian splats on the fly using gemini 3.1 + sparkjs 1. Load any 3D scene and hit scan 2. Get 2D detections from VLM 3. Cluster outputs & project into 3D world space 4. Save as a persistent 3D semantic layer Inspired by Alexander Chen's experiments with gemini visual intelligence. Just had to try to lift it from 2D to 3D!

Bilawal Sidhu

56,754 Aufrufe • vor 2 Monaten

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering Contributions: • We reformulate video stabilization as a novel 3D grounded scheme of local reconstruction and rendering. This approach is naturally robust to diverse camera motions and scene dynamics, is temporally consistent, and is capable of full frame stabilization. • We propose a novel test-time optimization for each unstable video. It leverages multi-view dynamics-aware photometric supervision and cross-frame regularization to achieve temporally consistent reconstructions. To avoid frame cropping, we introduce a scene extrapolation module based on video completion. • We provide a 3D-grounded dataset for our task by re-purposing an existing one, and introduce new metrics on sparse and dense reconstruction to evaluate 3D scene consistency. Extensive experiments (quantitative, qualitative, user study) versus image-based and gyro-basedmethods demonstrate the merits of our method.

MrNeRF

11,638 Aufrufe • vor 1 Jahr

Over the holiday period, we took an early look at the new DLSS 4 feature set running on an RTX 5080 in Cyberpunk 2077. Super resolution, multi frame-gen, ray reconstruction - initial thoughts and capture right here:

Over the holiday period, we took an early look at the new DLSS 4 feature set running on an RTX 5080 in Cyberpunk 2077. Super resolution, multi frame-gen, ray reconstruction - initial thoughts and capture right here:

Digital Foundry

254,406 Aufrufe • vor 1 Jahr

Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes Contributions: • We propose STORM, the first feed-forward, self-supervised method for fast and accurate reconstruction of dynamic 3D scenes from sparse, multi-timestep, posed camera images. • Our bottom-up framework aggregates and transforms per-frame 3D Gaussian Splats into a cohesive scene representation, enabling self-supervised motion estimation. Furthermore, we introduce motion tokens that capture common motion primitives and regularize motion predictions, facilitating dynamic motion group segmentation without explicit motion or correspondence supervision. • We present several enhancements for in-the-wild scenarios, including sky modeling, camera exposure inconsistency handling, large novel-view extrapolation, and fine-grained human motions reconstruction, making STORM well-suited for real-world applications.

Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes Contributions: • We propose STORM, the first feed-forward, self-supervised method for fast and accurate reconstruction of dynamic 3D scenes from sparse, multi-timestep, posed camera images. • Our bottom-up framework aggregates and transforms per-frame 3D Gaussian Splats into a cohesive scene representation, enabling self-supervised motion estimation. Furthermore, we introduce motion tokens that capture common motion primitives and regularize motion predictions, facilitating dynamic motion group segmentation without explicit motion or correspondence supervision. • We present several enhancements for in-the-wild scenarios, including sky modeling, camera exposure inconsistency handling, large novel-view extrapolation, and fine-grained human motions reconstruction, making STORM well-suited for real-world applications.

MrNeRF

53,292 Aufrufe • vor 1 Jahr

💥 Think more real data is needed for scene reconstruction? Think again! Meet MegaSynth: scaling up feed-forward 3D scene reconstruction with synthesized scenes. In 3 days, it generates 700K scenes for training—70x larger than real data! ✨ The secret? Reconstruction is mostly non-semantic! No need to rely heavily on real or highly realistic synthetic data. 🌐 Project: (1/4)

💥 Think more real data is needed for scene reconstruction? Think again! Meet MegaSynth: scaling up feed-forward 3D scene reconstruction with synthesized scenes. In 3 days, it generates 700K scenes for training—70x larger than real data! ✨ The secret? Reconstruction is mostly non-semantic! No need to rely heavily on real or highly realistic synthetic data. 🌐 Project: (1/4)

Hanwen Jiang

26,969 Aufrufe • vor 1 Jahr

** Sega Genesis 3D engine update 4 ** Never can leave a good rabbit hole undug ! .. updates since last update : Changed frame buffer from a VDP tile friendly buffer to a linear buffer which needs CPU translation , which speedup drawing a LOT not having to worry about tile boundaries albiet slows buffer sending to VRAM. The interior filling of triangles now is running at 1.5c ( 68k cycles ) per pixel ! however sending the buffer is twice as slow per byte as we need the CPU to send the buffer to VRAM to write to the tile format the VDP needs, rather than DMA directly to VRAM . Overall speedup 25% + as the drawing speedup is much greater than the frame buffer slow down. Optimised Line drawing and Triangle filling -near optimal now for the linear frame buffer. Triangle drawing might get a little bit more yet. Quad drawing is next on the list as that will speedup drawing quads. Some gains will come from been smart about partial clearing of RAM buffers and partial sending to VRAM buffers, rather than send the whole buffer. Vertice rotation and projection needs a lot of optimising .. i expect good gains to come from this - this is all in C at present and all calced in real time. Line and Triangle drawing are in assembly but all vertex manipulation is in C. #SGDK #MegaDrive #SegaGenesis

Sega Genesis 3D engine update 4 Never can leave a good rabbit hole undug ! .. updates since last update : Changed frame buffer from a VDP tile friendly buffer to a linear buffer which needs CPU translation , which speedup drawing a LOT not having to worry about tile boundaries albiet slows buffer sending to VRAM. The interior filling of triangles now is running at 1.5c ( 68k cycles ) per pixel ! however sending the buffer is twice as slow per byte as we need the CPU to send the buffer to VRAM to write to the tile format the VDP needs, rather than DMA directly to VRAM . Overall speedup 25% + as the drawing speedup is much greater than the frame buffer slow down. Optimised Line drawing and Triangle filling -near optimal now for the linear frame buffer. Triangle drawing might get a little bit more yet. Quad drawing is next on the list as that will speedup drawing quads. Some gains will come from been smart about partial clearing of RAM buffers and partial sending to VRAM buffers, rather than send the whole buffer. Vertice rotation and projection needs a lot of optimising .. i expect good gains to come from this - this is all in C at present and all calced in real time. Line and Triangle drawing are in assembly but all vertex manipulation is in C. #SGDK #MegaDrive #SegaGenesis

Shannon Birt

15,134 Aufrufe • vor 3 Monaten

#Battlefield6 NVIDIA DLSS 4 Reveal Trailer 📽️ The game includes support for DLSS 4 with Multi Frame Generation, DLSS Frame Generation, DLSS Super Resolution, DLAA, and NVIDIA Reflex. Desktop GPUs Performance At 4K, Ultra settings, DLSS 4 with Multi Frame Generation and DLSS Super Resolution multiply Battlefield 6's GeForce RTX 50 Series frame rates by an average of 3.8X. ▪️ GeForce RTX 5090 performance rockets to over 470 FPS. ▪️ GeForce RTX 5080 exceeds 330 FPS. ▪️ GeForce RTX 5070 Ti is in touching distance of 300 FPS. ▪️ GeForce RTX 5070 surpasses 230 FPS. At 2560x1440, Ultra settings, DLSS 4 with Multi Frame Generation and DLSS Super Resolution increase Battlefield 6 frame rates by an average of 3X. ▪️ GeForce RTX 5090 runs at almost 600 FPS. ▪️ GeForce RTX 5060 Ti over 240 FPS. At 1920x1080, Ultra settings, the combination of DLSS 4 with Multi Frame Generation and DLSS Super Resolution boost Battlefield 6’s GeForce RTX 50 Series frame rates by an average of 2.9X. ▪️ Allowing every GPU in NVIDIA's line-up to play at over 230 FPS, maxing out at over 740 FPS on the GeForce RTX 5090. Laptop GPUs Performance GeForce RTX 50 Series Laptop GPUs benefit similarly from DLSS 4 with Multi Frame Generation and DLSS Super Resolution, with Battlefield 6 frame rates multiplied by 3X on average at 2560x1600, with Ultra settings, enabling performance of up to 360 FPS. At 1920x1080, Ultra settings, GeForce RTX 50 Series Laptop GPU performance increases by 2.8X on average, enabling Battlefield 6 frame rates to surpass 480 FPS, and the range to run at over 200 frames per second.

#Battlefield6 NVIDIA DLSS 4 Reveal Trailer 📽️ The game includes support for DLSS 4 with Multi Frame Generation, DLSS Frame Generation, DLSS Super Resolution, DLAA, and NVIDIA Reflex. Desktop GPUs Performance At 4K, Ultra settings, DLSS 4 with Multi Frame Generation and DLSS Super Resolution multiply Battlefield 6's GeForce RTX 50 Series frame rates by an average of 3.8X. ▪️ GeForce RTX 5090 performance rockets to over 470 FPS. ▪️ GeForce RTX 5080 exceeds 330 FPS. ▪️ GeForce RTX 5070 Ti is in touching distance of 300 FPS. ▪️ GeForce RTX 5070 surpasses 230 FPS. At 2560x1440, Ultra settings, DLSS 4 with Multi Frame Generation and DLSS Super Resolution increase Battlefield 6 frame rates by an average of 3X. ▪️ GeForce RTX 5090 runs at almost 600 FPS. ▪️ GeForce RTX 5060 Ti over 240 FPS. At 1920x1080, Ultra settings, the combination of DLSS 4 with Multi Frame Generation and DLSS Super Resolution boost Battlefield 6’s GeForce RTX 50 Series frame rates by an average of 2.9X. ▪️ Allowing every GPU in NVIDIA's line-up to play at over 230 FPS, maxing out at over 740 FPS on the GeForce RTX 5090. Laptop GPUs Performance GeForce RTX 50 Series Laptop GPUs benefit similarly from DLSS 4 with Multi Frame Generation and DLSS Super Resolution, with Battlefield 6 frame rates multiplied by 3X on average at 2560x1600, with Ultra settings, enabling performance of up to 360 FPS. At 1920x1080, Ultra settings, GeForce RTX 50 Series Laptop GPU performance increases by 2.8X on average, enabling Battlefield 6 frame rates to surpass 480 FPS, and the range to run at over 200 frames per second.

Battlefield Bulletin

30,559 Aufrufe • vor 9 Monaten

This seemingly obvious prediction didn't take long to become reality. MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors Contributions: • The first real-time SLAM system using the two-view 3D reconstruction prior MASt3R [20] as a foundation. • Efficient techniques for pointmap matching, tracking and local fusion, graph construction and loop closure, and second-order global optimization. • A state-of-the-art dense SLAM system capable of handling generic, time-varying camera models. Abstract: We present a real-time monocular dense SLAM system, designed from the ground up using MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system remains robust on in-the-wild video sequences, making no assumptions on a fixed or parametric camera model beyond a unique camera center. Key features include: - Efficient methods for pointmap matching, camera tracking, and local fusion - Graph construction and loop closure - Second-order global optimization With known calibration, a simple modification achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

This seemingly obvious prediction didn't take long to become reality. MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors Contributions: • The first real-time SLAM system using the two-view 3D reconstruction prior MASt3R [20] as a foundation. • Efficient techniques for pointmap matching, tracking and local fusion, graph construction and loop closure, and second-order global optimization. • A state-of-the-art dense SLAM system capable of handling generic, time-varying camera models. Abstract: We present a real-time monocular dense SLAM system, designed from the ground up using MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system remains robust on in-the-wild video sequences, making no assumptions on a fixed or parametric camera model beyond a unique camera center. Key features include: - Efficient methods for pointmap matching, camera tracking, and local fusion - Graph construction and loop closure - Second-order global optimization With known calibration, a simple modification achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

MrNeRF

29,961 Aufrufe • vor 1 Jahr

⚡Say hello to Wonder Animation⚡ - Kickstart your animated project with groundbreaking Video to 3D Scene technology. Convert live footage into 3D scenes with all camera setups, character body and face animation, and fully editable elements in one 3D space. Discover your next level of creativity today! 👉Learn more: #Animation #3D #WonderStudio #WonderDynamics

⚡Say hello to Wonder Animation⚡ - Kickstart your animated project with groundbreaking Video to 3D Scene technology. Convert live footage into 3D scenes with all camera setups, character body and face animation, and fully editable elements in one 3D space. Discover your next level of creativity today! 👉Learn more: #Animation #3D #WonderStudio #WonderDynamics

Autodesk Flow Studio

349,711 Aufrufe • vor 1 Jahr

Your iPad can function as a live camera preview using Cam Link 4K. This configuration is perfect for solo creators who need to make sure they're in focus and in frame. No more running back and forth from your camera. 😎 Get the perfect shot with Cam Link 4K:

Your iPad can function as a live camera preview using Cam Link 4K. This configuration is perfect for solo creators who need to make sure they're in focus and in frame. No more running back and forth from your camera. 😎 Get the perfect shot with Cam Link 4K:

Elgato

24,152 Aufrufe • vor 1 Jahr

Today, we are open-sourcing Hunyuan World 1.1 (WorldMirror), a universal feed-forward 3D reconstruction model. 🚀🚀🚀 While our previously released Hunyuan World 1.0 (open-sourced, lite version deployable on consumer GPUs) focused on generating 3D worlds from text or single-view images, Hunyuan World 1.1 significantly expands the input scope by unlocking video-to-3D and multi-view-to-3D world creation. Highlights: 🔹Any Input, Maximized Flexibility and Fidelity: Flexibly integrates diverse geometric priors (camera poses, intrinsics, depth maps) to resolve structural ambiguities and ensure geometrically consistent 3D outputs. 🔹Any Output, SOTA Results：This elegant architecture simultaneously generates multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussian Splattings. 🔹Single-GPU & Fast Inference: As an all-in-one, feed-forward model, Hunyuan World 1.1 runs on a single GPU and delivers all 3D attributes in a single forward pass, within seconds. 🌐Project Page: 🔗Github： 🤗Hugging Face： ✨Demo: 📄Technical Report:

Today, we are open-sourcing Hunyuan World 1.1 (WorldMirror), a universal feed-forward 3D reconstruction model. 🚀🚀🚀 While our previously released Hunyuan World 1.0 (open-sourced, lite version deployable on consumer GPUs) focused on generating 3D worlds from text or single-view images, Hunyuan World 1.1 significantly expands the input scope by unlocking video-to-3D and multi-view-to-3D world creation. Highlights: 🔹Any Input, Maximized Flexibility and Fidelity: Flexibly integrates diverse geometric priors (camera poses, intrinsics, depth maps) to resolve structural ambiguities and ensure geometrically consistent 3D outputs. 🔹Any Output, SOTA Results：This elegant architecture simultaneously generates multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussian Splattings. 🔹Single-GPU & Fast Inference: As an all-in-one, feed-forward model, Hunyuan World 1.1 runs on a single GPU and delivers all 3D attributes in a single forward pass, within seconds. 🌐Project Page: 🔗Github： 🤗Hugging Face： ✨Demo: 📄Technical Report:

Tencent Hy

168,530 Aufrufe • vor 9 Monaten

MAGS-SLAM: Monocular Multi-Agent Gaussian Splatting SLAM for Geometrically and Photometrically Consistent Reconstruction TL;DR: The first RGB-only multi-agent 3D Gaussian Splatting SLAM for collaborative photorealistic scene reconstruction. Contributions: (1) We propose the first monocular RGB-only multi-agent 3D Gaussian Splatting SLAM system. It integrates Gaussian front-ends, compact submap summaries, inter-agent verification, Sim(3) submap pose graph, and occupancy-aware fusion into a unified framework, achieving accurate tracking and photorealistic reconstruction without depth sensors. (2) We propose a Pose-Graph Bundle Adjustment (PGBA)-consistent Sim(3) loop closure mechanism for multi-agent systems, which jointly resolves intra- and inter-agent scale drift through a submap-level Sim(3) pose graph coupling geometric and photometric residuals. Robustness is ensured by a spatial-extent gate that rejects degenerate loops and an adaptive edge invalidation scheme consistent with evolving PGBA corrections. (3) We propose an occupancy-aware fusion framework for coherent multi-agent Gaussian maps. It combines occupancy-grid deduplication, decoupled coordinator, and joint pose-Gaussian photometric refinement to eliminate duplicated Gaussians, residual misalignment, and photometric seams across agents. (4) We introduce ReplicaMultiagent Plus dataset. While existing multi-agent datasets are typically limited to 2-3 agents with short trajectories, our dataset scales to 4 agents with long-horizon trajectories. In addition, we provide ground-truth geometry and semantic annotations, supporting the evaluation of monocular, RGB-D, and semantic multi-agent SLAM for collaborative dense reconstruction.

MAGS-SLAM: Monocular Multi-Agent Gaussian Splatting SLAM for Geometrically and Photometrically Consistent Reconstruction TL;DR: The first RGB-only multi-agent 3D Gaussian Splatting SLAM for collaborative photorealistic scene reconstruction. Contributions: (1) We propose the first monocular RGB-only multi-agent 3D Gaussian Splatting SLAM system. It integrates Gaussian front-ends, compact submap summaries, inter-agent verification, Sim(3) submap pose graph, and occupancy-aware fusion into a unified framework, achieving accurate tracking and photorealistic reconstruction without depth sensors. (2) We propose a Pose-Graph Bundle Adjustment (PGBA)-consistent Sim(3) loop closure mechanism for multi-agent systems, which jointly resolves intra- and inter-agent scale drift through a submap-level Sim(3) pose graph coupling geometric and photometric residuals. Robustness is ensured by a spatial-extent gate that rejects degenerate loops and an adaptive edge invalidation scheme consistent with evolving PGBA corrections. (3) We propose an occupancy-aware fusion framework for coherent multi-agent Gaussian maps. It combines occupancy-grid deduplication, decoupled coordinator, and joint pose-Gaussian photometric refinement to eliminate duplicated Gaussians, residual misalignment, and photometric seams across agents. (4) We introduce ReplicaMultiagent Plus dataset. While existing multi-agent datasets are typically limited to 2-3 agents with short trajectories, our dataset scales to 4 agents with long-horizon trajectories. In addition, we provide ground-truth geometry and semantic annotations, supporting the evaluation of monocular, RGB-D, and semantic multi-agent SLAM for collaborative dense reconstruction.

MrNeRF

19,357 Aufrufe • vor 2 Monaten

Experimenting with wrist controls on smart glasses Using our Wearables SDK, we can display a 3D wireframe A-Frame scene on compatible smartglasses (in this case the Brilliant Labs Frame running our custom firmware) We’re also using our SDK to control a virtual cursor with our Ukaton motion module strapped to our wrist, running an Edge Impulse model to detect pinches There’s a lot of third party hand control accessories we could use (TapWithUs, Doublepoint, and Mudra Band), but using our own hardware/software is great for creating custom gestures

Experimenting with wrist controls on smart glasses Using our Wearables SDK, we can display a 3D wireframe A-Frame scene on compatible smartglasses (in this case the Brilliant Labs Frame running our custom firmware) We’re also using our SDK to control a virtual cursor with our Ukaton motion module strapped to our wrist, running an Edge Impulse model to detect pinches There’s a lot of third party hand control accessories we could use (TapWithUs, Doublepoint, and Mudra Band), but using our own hardware/software is great for creating custom gestures

Brilliant Wear

29,397 Aufrufe • vor 9 Monaten

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

Bilawal Sidhu

27,278 Aufrufe • vor 8 Monaten

V3 Starship and its 33 Raptor engines ascending during their inaugural flight yesterday from up close, shot on a remote video camera! Look at the ice coming off the front of Ship 39 and steam creeping back into frame at the end!! In love with this shot! 📽️ Me for Space Scout

V3 Starship and its 33 Raptor engines ascending during their inaugural flight yesterday from up close, shot on a remote video camera! Look at the ice coming off the front of Ship 39 and steam creeping back into frame at the end!! In love with this shot! 📽️ Me for Space Scout

Cameron Schwartz

25,453 Aufrufe • vor 2 Monaten

📸 The Animations & Poses Studio is officially LIVE on Say goodbye to tedious, frame-by-frame 3D animating. Now, you can instantly give your avatars body language, movement, and a whole lot of personality in just a few clicks. Check out the demo below to see how you can generate custom animations and movements for ANY 3D avatar, yes you can even bring your custom 3D files to the studio. Monetization of animations coming soon. Jump in and start creating, and be sure to tag us with your custom animations 👇

three.ws

43,157 Aufrufe • vor 1 Monat

MVP of Multiview Video → Camera parameters + 3D keypoints. Visualized with Rerun The basic pipeline as of right now looks like this: 1. Capture 🔴 – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric view; the Insta360 Go is used for the egocentric view. 2. Sync 🕒 – Custom Gradio app using two Rerun viewers and callbacks for easily aligning frame timestamps so the ego and exo views are aligned. 3. Calibrate 🎯 – Use VGGT from Jianyuan and AI at Meta to get intrinsics/extrinsics for sparse cameras. 4. Estimate 3D 🕺 – Use RTMLib whole‑body keypoint estimator on each frame, then triangulate in 3D. What's missing? 1. No temporal coherence: I’m estimating keypoints one frame at a time and one camera at a time. This leads to a lot of jittering. For now, I plan on adding a One Euro Filter to help with jittering. Long term, I'd want to train a multiview keypoint estimator 2. Kinematic fitting is still missing; this is my next goal. The output will be joint angles, as explored in my previous posts. 3. Missing dense point cloud: VGGT seems to fail for me here. I’m looking to explore using MP‑SFM as a method for generating dense multiview depth maps + normals (plus it has a friendlier license compared to VGGT). 4. Eventually, creation of 4D Gaussian splatting using something akin to DN‑splatter—my long‑term goal is a data engine that provides poses/depths/splats/keypoints/etc.

MVP of Multiview Video → Camera parameters + 3D keypoints. Visualized with Rerun The basic pipeline as of right now looks like this: 1. Capture 🔴 – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric view; the Insta360 Go is used for the egocentric view. 2. Sync 🕒 – Custom Gradio app using two Rerun viewers and callbacks for easily aligning frame timestamps so the ego and exo views are aligned. 3. Calibrate 🎯 – Use VGGT from Jianyuan and AI at Meta to get intrinsics/extrinsics for sparse cameras. 4. Estimate 3D 🕺 – Use RTMLib whole‑body keypoint estimator on each frame, then triangulate in 3D. What's missing? 1. No temporal coherence: I’m estimating keypoints one frame at a time and one camera at a time. This leads to a lot of jittering. For now, I plan on adding a One Euro Filter to help with jittering. Long term, I'd want to train a multiview keypoint estimator 2. Kinematic fitting is still missing; this is my next goal. The output will be joint angles, as explored in my previous posts. 3. Missing dense point cloud: VGGT seems to fail for me here. I’m looking to explore using MP‑SFM as a method for generating dense multiview depth maps + normals (plus it has a friendlier license compared to VGGT). 4. Eventually, creation of 4D Gaussian splatting using something akin to DN‑splatter—my long‑term goal is a data engine that provides poses/depths/splats/keypoints/etc.

Pablo Vela

42,785 Aufrufe • vor 1 Jahr

Starting the new year without human labeling 🎉!! Multimodal lidar-camera data is a gold mine of dense 3D geometry hiding in plain sight. For supervised pretraining and validation at scale at Torc-Robotics, we rely on fully automated pseudo-labeling pipelines. Exploiting geometric priors from temporally accumulated LiDAR maps and an iterative update rule enforces joint geometric–semantic consistency while detecting moving objects via inconsistencies. We achieve 3D semantic labels and 3D bounding boxes with human-like quality at 200m+ range required for highway driving. Paper: Exciting work with Torc-Robotics with Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic.

Starting the new year without human labeling 🎉!! Multimodal lidar-camera data is a gold mine of dense 3D geometry hiding in plain sight. For supervised pretraining and validation at scale at Torc-Robotics, we rely on fully automated pseudo-labeling pipelines. Exploiting geometric priors from temporally accumulated LiDAR maps and an iterative update rule enforces joint geometric–semantic consistency while detecting moving objects via inconsistencies. We achieve 3D semantic labels and 3D bounding boxes with human-like quality at 200m+ range required for highway driving. Paper: Exciting work with Torc-Robotics with Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic.

Felix Heide

18,210 Aufrufe • vor 6 Monaten

3D hand tracking on smart glasses Not only can we stream camera data with our Wearables SDK, but we can run MediaPipe for hand tracking, applying it in WebXR environments like Three.js and A-Frame Works with compatible smart glasses, in this case the Omi AI Omi Glass, based on the Seeed Studio XIAO ESP32-S3 Sense (running our custom Firmware)

3D hand tracking on smart glasses Not only can we stream camera data with our Wearables SDK, but we can run MediaPipe for hand tracking, applying it in WebXR environments like Three.js and A-Frame Works with compatible smart glasses, in this case the Omi AI Omi Glass, based on the Seeed Studio XIAO ESP32-S3 Sense (running our custom Firmware)

Brilliant Wear

79,410 Aufrufe • vor 6 Monaten