Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Most imitation learning policies break when the camera moves or the robot changes. NOT THIS ONE 👇 [📍 Bookmark for later ] A new 3D scene representation encoder, tackles this by enabling zero-shot generalization to unseen embodiments and viewpoints… And it works with any IL algorithm. The trick? •Use... a 2D foundation model to extract semantic features •Lift them into 3D space for localization (not semantics) •Condition the IL policy on this spatially grounded vector Across 93 simulated and 6 real tasks, Adapt3R: ✅ Maintains IL performance on LIBERO & MimicGen benchmarks ✅ Outperforms DP3 and 3D Diffuser Actor in most settings ✅ Holds >80% success on LIBERO even with large camera rotations Thanks for sharing this, Animesh Garg & Albert Wilcox! 📍Paper: Website: Code:show more

Ilir Aliu

56,209 subscribers

12,209 Aufrufe • vor 1 Jahr •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Robotics keeps hitting the same wall. Single task RL works, but... it does not scale to hundreds of tasks or new embodiments. This new paper looks like a real step toward fixing that. The team introduces MMBench, a benchmark with 200 tasks across many domains and robots, and Newt, a language conditioned world model trained online across all 200 tasks at once. The simple idea behind Newt: The model learns from demos to get the right priors It trains across many tasks through online interaction It uses language to ground the goal It adapts fast when a new task shows up What stood out to me: ✅ One model trained on 200 tasks at the same time ✅ Language conditioned control for both states and RGB ✅ Better data efficiency than strong baselines ✅ Strong open loop control ✅ Fast adaptation to new tasks and embodiments ✅ Full release of 200 checkpoints, 4000 demos, code, and benchmark This is a good push toward general control instead of one model per task. If you want the full paper: Project page: —- Weekly robotics and AI insights. Subscribe free:

Robotics keeps hitting the same wall. Single task RL works, but... it does not scale to hundreds of tasks or new embodiments. This new paper looks like a real step toward fixing that. The team introduces MMBench, a benchmark with 200 tasks across many domains and robots, and Newt, a language conditioned world model trained online across all 200 tasks at once. The simple idea behind Newt: The model learns from demos to get the right priors It trains across many tasks through online interaction It uses language to ground the goal It adapts fast when a new task shows up What stood out to me: ✅ One model trained on 200 tasks at the same time ✅ Language conditioned control for both states and RGB ✅ Better data efficiency than strong baselines ✅ Strong open loop control ✅ Fast adaptation to new tasks and embodiments ✅ Full release of 200 checkpoints, 4000 demos, code, and benchmark This is a good push toward general control instead of one model per task. If you want the full paper: Project page: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

70,090 Aufrufe • vor 10 Monaten

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [📍 Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like π³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: • Up to 30 FPS • 10 to 15x speedup • Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes • Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperial’s Dyson Robotics Lab and many others who contributed to this! 📍 Save projects page + paper for later: Video: ——- if it matters in AI or Robotics you'll read it here first:

Robots can now reconstruct 3D scenes in real time from a single RGB camera. [📍 Projects page + paper] No depth sensor. No retraining. 30 FPS. Researchers at the Imperial College London introduced KV-Tracker, a training-free method that makes heavy models like π³ and Depth Anything 3 fast enough for real-time tracking. The idea is simple. These models use global self-attention, which is powerful but computationally expensive. KV-Tracker caches the key and value pairs from selected keyframes and reuses them for new frames. That cache becomes an implicit scene representation. Result: • Up to 30 FPS • 10 to 15x speedup • Accurate 6-DoF tracking on benchmarks like TUM RGB-D and 7-Scenes • Works with monocular RGB only It also supports object-level tracking with masks and allows saving the KV-cache for later reuse. For robotics, this reduces hardware constraints and moves real-time 3D perception closer to practical deployment. Credit to Marwan Taher (Marwan Taher) at Imperial’s Dyson Robotics Lab and many others who contributed to this! 📍 Save projects page + paper for later: Video: ——- if it matters in AI or Robotics you'll read it here first:

Ilir Aliu

53,992 Aufrufe • vor 6 Monaten

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 Aufrufe • vor 1 Jahr

🛠️ What if a robot could invent its own tools. And teach itself how to use them? That’s exactly what VLMgineer does: a new framework that lets Vision Language Models (VLMs) design physical tools and the actions to use them, entirely on their own. No templates. No human demonstrations. Just raw, AI-driven creativity. Why it matters ✅ Co-designs tools and actions together using VLMs, ensuring tight coupling between form and function ✅ Uses VLM-guided evolution (not random search) to refine designs intelligently ✅ Outperforms human-designed tools by +64.7% in task success across 12 RoboToolBench challenges ✅ Produces better-than-everyday tools for real manipulation tasks—measured in success rate and elegance It builds on the emerging trend of large-model-guided evolutionary design (like Eureka and AlphaEvolve) and brings it into physical robotics. It opens the door to general-purpose, automated hardware design, no strong priors needed. Code & paper: —- Weekly robotics and AI insights. Subscribe free:

🛠️ What if a robot could invent its own tools. And teach itself how to use them? That’s exactly what VLMgineer does: a new framework that lets Vision Language Models (VLMs) design physical tools and the actions to use them, entirely on their own. No templates. No human demonstrations. Just raw, AI-driven creativity. Why it matters ✅ Co-designs tools and actions together using VLMs, ensuring tight coupling between form and function ✅ Uses VLM-guided evolution (not random search) to refine designs intelligently ✅ Outperforms human-designed tools by +64.7% in task success across 12 RoboToolBench challenges ✅ Produces better-than-everyday tools for real manipulation tasks—measured in success rate and elegance It builds on the emerging trend of large-model-guided evolutionary design (like Eureka and AlphaEvolve) and brings it into physical robotics. It opens the door to general-purpose, automated hardware design, no strong priors needed. Code & paper: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

14,294 Aufrufe • vor 9 Monaten

𝗥𝗼𝗯𝗼𝘁𝘀 𝗱𝗼𝗻’𝘁 𝗻𝗲𝗲𝗱 𝗺𝗼𝗿𝗲 𝗱𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀. 𝗧𝗵𝗲𝘆 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗹𝗲𝗮𝗿𝗻 𝗳𝗿𝗼𝗺 𝗳𝗮𝗶𝗹𝘂𝗿𝗲 — 𝗮𝗳𝘁𝗲𝗿 𝘄𝗮𝘁𝗰𝗵𝗶𝗻𝗴 𝗵𝘂𝗺𝗮𝗻𝘀. Most robot learning systems assume failure is the end of learning. In our new work, we study whether robots can improve after deployment by learning from their own failures, without any human intervention, teleoperation, or corrective labels. The key idea is simple: human videos contain structure about how the world works. We use them to learn cross-embodiment representations of action, dynamics, and value, enabling a shared predictive space between human behavior and robot experience. This allows a new learning loop: 👉 pretrain on human videos 👉 deploy robot policy 👉 observe failures 👉 reinterpret failures using human priors 👉 improve autonomously We evaluate this across 7 real-world manipulation tasks, showing: 📈 40% → 81% success rate 🏆 Strong improvements over π0.6 RECAP and RISE ✔️ Zero human intervention during post-deployment improvement 🧬 Generalizes across robot embodiments and policy backbones A key finding is that explicit failure repair significantly outperforms failure reweighting, yielding substantially larger gains under identical data conditions (+25 pts vs +5 pts on the same π0.5 base policy). Overall, the results suggest a shift in how we think about robot learning: Human videos are not only for pretraining policies. They can provide the structure needed for continual self-improvement after deployment. 📄 Paper: 🌐 Project: I am grateful for working with the fantastic leads Hanzhi Chen and Anran Zhang, and our collaborators Simon Schaefer, Kejia Chen, Shi Chen, Daniel Cremers. Special thanks to Stefan Leutenegger for co-advising this project with me. ETH Zürich TU München Microsoft Check out Hanzhi's 🧵 for more details

𝗥𝗼𝗯𝗼𝘁𝘀 𝗱𝗼𝗻’𝘁 𝗻𝗲𝗲𝗱 𝗺𝗼𝗿𝗲 𝗱𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀. 𝗧𝗵𝗲𝘆 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗹𝗲𝗮𝗿𝗻 𝗳𝗿𝗼𝗺 𝗳𝗮𝗶𝗹𝘂𝗿𝗲 — 𝗮𝗳𝘁𝗲𝗿 𝘄𝗮𝘁𝗰𝗵𝗶𝗻𝗴 𝗵𝘂𝗺𝗮𝗻𝘀. Most robot learning systems assume failure is the end of learning. In our new work, we study whether robots can improve after deployment by learning from their own failures, without any human intervention, teleoperation, or corrective labels. The key idea is simple: human videos contain structure about how the world works. We use them to learn cross-embodiment representations of action, dynamics, and value, enabling a shared predictive space between human behavior and robot experience. This allows a new learning loop: 👉 pretrain on human videos 👉 deploy robot policy 👉 observe failures 👉 reinterpret failures using human priors 👉 improve autonomously We evaluate this across 7 real-world manipulation tasks, showing: 📈 40% → 81% success rate 🏆 Strong improvements over π0.6 RECAP and RISE ✔️ Zero human intervention during post-deployment improvement 🧬 Generalizes across robot embodiments and policy backbones A key finding is that explicit failure repair significantly outperforms failure reweighting, yielding substantially larger gains under identical data conditions (+25 pts vs +5 pts on the same π0.5 base policy). Overall, the results suggest a shift in how we think about robot learning: Human videos are not only for pretraining policies. They can provide the structure needed for continual self-improvement after deployment. 📄 Paper: 🌐 Project: I am grateful for working with the fantastic leads Hanzhi Chen and Anran Zhang, and our collaborators Simon Schaefer, Kejia Chen, Shi Chen, Daniel Cremers. Special thanks to Stefan Leutenegger for co-advising this project with me. ETH Zürich TU München Microsoft Check out Hanzhi's 🧵 for more details

Oier Mees

12,682 Aufrufe • vor 3 Monaten

For 41 years, to find shortest paths in a graph, the Dijkstra’s algorithm, was seen as the best possible way. Not any longer: Now a team from Tsinghua University has beaten it. They created the first faster algorithm for directed shortest paths since 1984. • Faster than Dijkstra on large sparse graphs • Works with real, non-negative edge weights • Proves that sorting is not the main bottleneck anymore Shortest-path algorithms power maps, GPS, logistics, networking, and robotics. This result shows that even the most “finished” algorithms can still be improved. Thanks for sharing, Md Ismail Šojal 🕷️! 📍 Paper: —— Weekly robotics and AI insights. Subscribe free:

For 41 years, to find shortest paths in a graph, the Dijkstra’s algorithm, was seen as the best possible way. Not any longer: Now a team from Tsinghua University has beaten it. They created the first faster algorithm for directed shortest paths since 1984. • Faster than Dijkstra on large sparse graphs • Works with real, non-negative edge weights • Proves that sorting is not the main bottleneck anymore Shortest-path algorithms power maps, GPS, logistics, networking, and robotics. This result shows that even the most “finished” algorithms can still be improved. Thanks for sharing, Md Ismail Šojal 🕷️! 📍 Paper: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

95,345 Aufrufe • vor 8 Monaten

I optimized my Dream Loop skill to give you Astra's 3D magic for 80% cheaper! This demo was one-shot, in 25 minutes, for 7% quota on ChatGPT Plus. Yes, the $20/mo one. 3 days of banging my head on the wall and burning Astra tokens. Sharing it for free. Y'all better build some cool stuff with this Link: Details, demo prompt, and cost breakdown are in replies. To be clear: It's not perfect. The demo is a small interactive scene, not a full game. For full games, your best bet is to build it with smaller models, then use this skill to improve the graphics. This is the best I could do, but it's not going to fix the fact that Astra is expensive and Plus quotas are tiny

I optimized my Dream Loop skill to give you Astra's 3D magic for 80% cheaper! This demo was one-shot, in 25 minutes, for 7% quota on ChatGPT Plus. Yes, the $20/mo one. 3 days of banging my head on the wall and burning Astra tokens. Sharing it for free. Y'all better build some cool stuff with this Link: Details, demo prompt, and cost breakdown are in replies. To be clear: It's not perfect. The demo is a small interactive scene, not a full game. For full games, your best bet is to build it with smaller models, then use this skill to improve the graphics. This is the best I could do, but it's not going to fix the fact that Astra is expensive and Plus quotas are tiny

Anshu

361,014 Aufrufe • vor 23 Tagen

This isn’t a 3D model file. It’s 100% code Our world-famous Mars Cat avatar has been reconstructed entirely as Three.js source code No GLB at runtime. No model download. No textures or UV atlas. Just code, rendered directly in the browser It was created with IMG2THREEJS by Nick. An incredible open-source project with over 15,000 stars on GitHub And our Mars Cat is now featured on the home page gallery See it live below ↓. Grab the code. Add it to your game New to Three.js? It’s one of the most popular open-source JavaScript libraries for building interactive 3D experiences and games on the web

Jay B 😻🚀

15,044 Aufrufe • vor 29 Tagen

I built a full 3D #lego brick builder using Claude 4.6. Took 12 iterations. Runs entirely in the browser. You pick a brick size, pick a color, and click to place. Snap-to-grid, collision detection, ghost preview, it feels like building with real bricks. But the fun part? Hit one of the template buttons and watch it auto-build a pyramid, a rocket, a house, or an Eiffel Tower. Bricks fly in from random directions, spin through the air, and snap into place with particle bursts. tech stack: → #Threejs for the full 3D scene → #Raycasting for click-to-place → Custom wedge geometry for slope bricks → Quadratic Bézier curves for the fly-in animations → #2d canvas particle system layered on top of the 3D scene → #InstancedMesh for the base plate studs The product potential: this could become a real educational toy. Kids (or adults) building in the browser, sharing creations, loading custom templates. No install, no app store. ( LEGO you may like this one! ) Want to try it? Link in the first comment 👇

I built a full 3D #lego brick builder using Claude 4.6. Took 12 iterations. Runs entirely in the browser. You pick a brick size, pick a color, and click to place. Snap-to-grid, collision detection, ghost preview, it feels like building with real bricks. But the fun part? Hit one of the template buttons and watch it auto-build a pyramid, a rocket, a house, or an Eiffel Tower. Bricks fly in from random directions, spin through the air, and snap into place with particle bursts. tech stack: → #Threejs for the full 3D scene → #Raycasting for click-to-place → Custom wedge geometry for slope bricks → Quadratic Bézier curves for the fly-in animations → #2d canvas particle system layered on top of the 3D scene → #InstancedMesh for the base plate studs The product potential: this could become a real educational toy. Kids (or adults) building in the browser, sharing creations, loading custom templates. No install, no app store. ( LEGO you may like this one! ) Want to try it? Link in the first comment 👇

Hasan Aboul Hasan

13,814 Aufrufe • vor 6 Monaten

Most robots still need markers, checkerboards, or long calibration rituals just to know where their arms are. Now it works from raw images in seconds. roboreg is a markerless multi arm localization toolkit that plugs into ROS 2 and RViz. No special hardware. No custom setup. You toggle between robot descriptions and the system figures out the rest. The idea is simple: ✅ Hand eye calibration from plain RGB or RGB D images ✅ Only three robot poses needed for millimeter accuracy ✅ Works with any ROS 2 compatible robot and camera ✅ Fully open source under Apache 2.0 It is powered by Hydra, a new marker free ICP variant that converges far more reliably than classical baselines and runs in under a second. If you want to try it: roboreg: ROS 2 roboreg: Hydra paper: pip install roboreg More details and discussion on Open Robotics Discourse:

Most robots still need markers, checkerboards, or long calibration rituals just to know where their arms are. Now it works from raw images in seconds. roboreg is a markerless multi arm localization toolkit that plugs into ROS 2 and RViz. No special hardware. No custom setup. You toggle between robot descriptions and the system figures out the rest. The idea is simple: ✅ Hand eye calibration from plain RGB or RGB D images ✅ Only three robot poses needed for millimeter accuracy ✅ Works with any ROS 2 compatible robot and camera ✅ Fully open source under Apache 2.0 It is powered by Hydra, a new marker free ICP variant that converges far more reliably than classical baselines and runs in under a second. If you want to try it: roboreg: ROS 2 roboreg: Hydra paper: pip install roboreg More details and discussion on Open Robotics Discourse:

Ilir Aliu

18,442 Aufrufe • vor 10 Monaten

The next breakthrough for robot VLMs isn’t a bigger model 🧠 it’s memory. RAL accepted ✅ We show that a robot can ground a VLM using its own real-world experience, not extra training. EXPTEACH/PRAGMABOT = self-generated memory + closed-loop reflection: Fail → diagnose → replan → succeed → store experience Next time → retrieve the right experience → succeed in one shot Results on real robot tasks: 36% → 84% with reflection (STM) 22% → 80% single-trial success with long-term memory (LTM+RAG) And yes, we saw tool use emerge without explicitly asking for it. If you’re fighting brittle “VLM plans” on hardware, this is for you. #AI #Robotics ANYbotics

The next breakthrough for robot VLMs isn’t a bigger model 🧠 it’s memory. RAL accepted ✅ We show that a robot can ground a VLM using its own real-world experience, not extra training. EXPTEACH/PRAGMABOT = self-generated memory + closed-loop reflection: Fail → diagnose → replan → succeed → store experience Next time → retrieve the right experience → succeed in one shot Results on real robot tasks: 36% → 84% with reflection (STM) 22% → 80% single-trial success with long-term memory (LTM+RAG) And yes, we saw tool use emerge without explicitly asking for it. If you’re fighting brittle “VLM plans” on hardware, this is for you. #AI #Robotics ANYbotics

Haitham Bou Ammar

12,206 Aufrufe • vor 7 Monaten

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.

Bilawal Sidhu

26,017 Aufrufe • vor 8 Monaten

Meta releases VGGSfM Visual Geometry Grounded Deep Structure From Motion Structure-from-motion (SfM) is a long-standing problem in the computer vision community, which aims to reconstruct the camera poses and 3D structure of a scene from a set of unconstrained 2D images. Classical frameworks solve this problem in an incremental manner by detecting and matching keypoints, registering images, triangulating 3D points, and conducting bundle adjustment. Recent research efforts have predominantly revolved around harnessing the power of deep learning techniques to enhance specific elements (e.g., keypoint matching), but are still based on the original, non-differentiable pipeline. Instead, we propose a new deep SfM pipeline VGGSfM, where each component is fully differentiable and thus can be trained in an end-to-end manner. To this end, we introduce new mechanisms and simplifications. First, we build on recent advances in deep 2D point tracking to extract reliable pixel-accurate tracks, which eliminates the need for chaining pairwise matches. Furthermore, we recover all cameras simultaneously based on the image and track features instead of gradually registering cameras. Finally, we optimise the cameras and triangulate 3D points via a differentiable bundle adjustment layer. We attain state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism, and ETH3D.

Meta releases VGGSfM Visual Geometry Grounded Deep Structure From Motion Structure-from-motion (SfM) is a long-standing problem in the computer vision community, which aims to reconstruct the camera poses and 3D structure of a scene from a set of unconstrained 2D images. Classical frameworks solve this problem in an incremental manner by detecting and matching keypoints, registering images, triangulating 3D points, and conducting bundle adjustment. Recent research efforts have predominantly revolved around harnessing the power of deep learning techniques to enhance specific elements (e.g., keypoint matching), but are still based on the original, non-differentiable pipeline. Instead, we propose a new deep SfM pipeline VGGSfM, where each component is fully differentiable and thus can be trained in an end-to-end manner. To this end, we introduce new mechanisms and simplifications. First, we build on recent advances in deep 2D point tracking to extract reliable pixel-accurate tracks, which eliminates the need for chaining pairwise matches. Furthermore, we recover all cameras simultaneously based on the image and track features instead of gradually registering cameras. Finally, we optimise the cameras and triangulate 3D points via a differentiable bundle adjustment layer. We attain state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism, and ETH3D.

AK

96,527 Aufrufe • vor 2 Jahren

🌀 #Live3D #Live2D #Live2DWIP 🌀 Today, I’d like to share two of the three methods I use to create joint twisting in my Live2D-based 2D Pseudo-3D concept. On the right is the Glue Joint： The basic idea is to use Glue to connect two pseudo-3D ArtMeshes. By taking advantage of Glue’s weight settings, it can create an effect similar to a 3D joint. Once the weights are set up, it can move very freely with a relatively low amount of work. However, it also shares the same weakness as 3D joints: when rotated at large angles, it can produce unpleasant twisting and distortion. On the left is the Hand-Bent Joint： The basic idea is to add bending Keyforms directly to a Warp Deformer, then manually sculpt the ideal deformation. This method can create the most detailed and refined joints, depending entirely on how the rigger designs them. The downside is that it requires much more work. Every additional degree of freedom requires a significant amount of manual adjustment. These two joint methods have almost opposite characteristics, so in actual Live3D work, I choose the most suitable method depending on the situation. For example, in this hand model, I used Hand-Bent Joints for the arm twisting and fingers, while the wrist uses a Glue Joint.✨

🌀 #Live3D #Live2D #Live2DWIP 🌀 Today, I’d like to share two of the three methods I use to create joint twisting in my Live2D-based 2D Pseudo-3D concept. On the right is the Glue Joint： The basic idea is to use Glue to connect two pseudo-3D ArtMeshes. By taking advantage of Glue’s weight settings, it can create an effect similar to a 3D joint. Once the weights are set up, it can move very freely with a relatively low amount of work. However, it also shares the same weakness as 3D joints: when rotated at large angles, it can produce unpleasant twisting and distortion. On the left is the Hand-Bent Joint： The basic idea is to add bending Keyforms directly to a Warp Deformer, then manually sculpt the ideal deformation. This method can create the most detailed and refined joints, depending entirely on how the rigger designs them. The downside is that it requires much more work. Every additional degree of freedom requires a significant amount of manual adjustment. These two joint methods have almost opposite characteristics, so in actual Live3D work, I choose the most suitable method depending on the situation. For example, in this hand model, I used Hand-Bent Joints for the arm twisting and fingers, while the wrist uses a Glue Joint.✨

📐Hephaestus📏Live2D匠人魂

18,943 Aufrufe • vor 3 Monaten

What even is time and space? Kurt Vonnegut's "Tralfamadorians don't see human beings as two-legged creatures, either. They see them as great millipedes - with babies' legs on one end and old people's legs at the other” So here's me walking through the world, leaving a 3D snapshot of every moment exactly when and where it happened. Feels a lot like onion skinning reality (to use the animation analogy). We are beings stuck in the arrow of time - who get momentary glimpses of what it means to break free from it. Perhaps this is one of those glimpses. Did not expect the last post to blow up like that. Really enjoyed the convos - thanks for all the love and the new rabbit holes to dive into.

What even is time and space? Kurt Vonnegut's "Tralfamadorians don't see human beings as two-legged creatures, either. They see them as great millipedes - with babies' legs on one end and old people's legs at the other” So here's me walking through the world, leaving a 3D snapshot of every moment exactly when and where it happened. Feels a lot like onion skinning reality (to use the animation analogy). We are beings stuck in the arrow of time - who get momentary glimpses of what it means to break free from it. Perhaps this is one of those glimpses. Did not expect the last post to blow up like that. Really enjoyed the convos - thanks for all the love and the new rabbit holes to dive into.

Bilawal Sidhu

660,506 Aufrufe • vor 9 Tagen