正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

✨ Presenting MicKey (#CVPR2024, Oral) ✨ We regress and match 3D camera coordinates rather then 2D key points, all in metric space. Gives you a scaled relative pose between two RGB images. Paper: Page:

Eric Brachmann

4,387 subscribers

70,399 次观看 • 2 年前 •via X (Twitter)

艺术科学技术教育 #CVPR2024

Anya Rossi• Live Now

Private livecam show

11 条评论

Eric Brachmann 的头像

Eric Brachmann2 年前

Beyond providing metric estimates, MicKey can also deal with extreme view point changes, up to opposing shots. Here are a few examples of MicKey correspondences.

Eric Brachmann 的头像

Eric Brachmann2 年前

What is a good key point if you want to go beyond matching co-visible areas? We do not know. MicKey is trained without ground-truth matches, and even without information about whether training image pairs have visual overlap or not.

Eric Brachmann 的头像

Eric Brachmann2 年前

MicKey is trained using only the relative pose between two images, and differentiable RANSAC (NG-DSAC, #ICCV2021). That means MicKey is trained to establish any correspondence it likes, as long as it leads to a good relative pose.

Eric Brachmann 的头像

Eric Brachmann2 年前

MicKey does extremely well on the #MapFreeReloc leaderboard, currently ranking second. It's best among the #CVPR2024 cohort, outperforming DUSt3R and FAR. (the field moves fast, and there is already a newer, unknown method that claims top spot)

Eric Brachmann 的头像

Eric Brachmann2 年前

Code available: Work by @NianticLabs, spearheaded by @axelbarrosotw, with @scriptide and @viprad.

Dan Goldman 的头像

Dan Goldman2 年前

Very cool work! IIUC, the scale ambiguity is not really gone, it's just inferring scale from image cues, with a training bias toward open scenes captured by adult human hands... What happens if you use it with images with adversarial cues, like a dollhouse, or trompe l'oeil?

Eric Brachmann 的头像

Eric Brachmann2 年前

Correct. Given nothing but two RGB images we can only rely on heuristics - and those can be misleading in some cases. Our initial motivation for the #MapFreeReloc benchmark was to see how far heuristics can get us. And after ~1 year I would say: Progress has been astounding.

Dmytro Mishkin 🇺🇦 的头像

Dmytro Mishkin 🇺🇦2 年前

Well deserved, congrats! P.S. I was faster to post :P

Eric Brachmann 的头像

Eric Brachmann2 年前

Thanks for sharing, and so fast too! I was trying to post early this morning, but Twitter did not like the video - without telling me why (apsect ratio...)

Dominik Kloepfer 的头像

Dominik Kloepfer2 年前

Pretty cool! If I understand correctly, the 3D keypoints are not actually constrained to lie on any surfaces — how well does the network nonetheless learn to reconstruct pointmaps? Does e.g. depth estimation emerge “for free”?

Eric Brachmann 的头像

Eric Brachmann2 年前

You are correct. There are no constraints, but still we see reasonable depth maps emerging. One additional twist here is that we only need accurate depth where key points are found. Standard depth estimators try to be accurate everywhere.

相关视频

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,663 次观看 • 3 年前

I imported 7 of Jonas’ photos into Agisoft Metashape, where I used its photogrammetric triangulation feature to match key points across the photos. By calculating corresponding points in multiple images, the software determined the camera’s orientation and position in three-dimensional space for each photo. The resulting positions, angles, and distances match what would be expected from a camera on a plane and match the specific details of Jonas’ photos, e.g., zoom, angle, and time between photos. This further proves that the Jonas photos are real, naturally taken photos that have not been manipulated. Their use in the MH370 “satellite” video is extremely strong evidence that the “satellite” video is fake. #MH370x

I imported 7 of Jonas’ photos into Agisoft Metashape, where I used its photogrammetric triangulation feature to match key points across the photos. By calculating corresponding points in multiple images, the software determined the camera’s orientation and position in three-dimensional space for each photo. The resulting positions, angles, and distances match what would be expected from a camera on a plane and match the specific details of Jonas’ photos, e.g., zoom, angle, and time between photos. This further proves that the Jonas photos are real, naturally taken photos that have not been manipulated. Their use in the MH370 “satellite” video is extremely strong evidence that the “satellite” video is fake. #MH370x

Tony Adams

75,268 次观看 • 1 年前

We are introducing Hunyuan3D-Part: an open-source part-level 3D shape generation model that outperforms all existing open and close-source models. Highlights: 🔹P3-SAM: The industry's first native 3D part segmentation model. 🔹X-Part: A part generation model that achieves state-of-the-art results in controllability and shape quality. Key-features: 1️⃣Eliminates the use of 2D SAM during training, relying solely on a large-scale dataset with 3.7 million shapes and clean part annotations. 2️⃣Introduces a new automated segmentation pipeline in 3D without user intervention. 3️⃣Implements a diffusion-based part decomposition pipeline utilizing both geometry and semantic clues. Code: Weights: Tech reports: 🔸P3-SAM： → Paper: → Project page: 🔸X-Part： → Paper: → Project page: Try it now： → (Light version) Hugging Face demo: → (Full version) Hunyuan3D Studio:

We are introducing Hunyuan3D-Part: an open-source part-level 3D shape generation model that outperforms all existing open and close-source models. Highlights: 🔹P3-SAM: The industry's first native 3D part segmentation model. 🔹X-Part: A part generation model that achieves state-of-the-art results in controllability and shape quality. Key-features: 1️⃣Eliminates the use of 2D SAM during training, relying solely on a large-scale dataset with 3.7 million shapes and clean part annotations. 2️⃣Introduces a new automated segmentation pipeline in 3D without user intervention. 3️⃣Implements a diffusion-based part decomposition pipeline utilizing both geometry and semantic clues. Code: Weights: Tech reports: 🔸P3-SAM： → Paper: → Project page: 🔸X-Part： → Paper: → Project page: Try it now： → (Light version) Hugging Face demo: → (Full version) Hunyuan3D Studio:

Tencent Hy

72,522 次观看 • 10 个月前

ELON MUSK: “The way to expand compute is really in space. There’s a lot of room in space. And if you look at the size of Earth relative to the Sun or relative to the Solar System, you realize just how tiny it is. We’re very, very tiny. We only receive about half a billionth of the Sun’s energy. So you can really think of Earth as being like a tiny dust mote in a vast darkness. So the way to expand compute without, you know, using up all the land on Earth is to do so in space. And then you can do it without using space for power and water on Earth; you can just do it in space. So I think we’ll probably be launching our first AI satellites next year, and then we’ll be able to do that, I think, at large scale in about two years.”

ELON MUSK: “The way to expand compute is really in space. There’s a lot of room in space. And if you look at the size of Earth relative to the Sun or relative to the Solar System, you realize just how tiny it is. We’re very, very tiny. We only receive about half a billionth of the Sun’s energy. So you can really think of Earth as being like a tiny dust mote in a vast darkness. So the way to expand compute without, you know, using up all the land on Earth is to do so in space. And then you can do it without using space for power and water on Earth; you can just do it in space. So I think we’ll probably be launching our first AI satellites next year, and then we’ll be able to do that, I think, at large scale in about two years.”

DogeDesigner

65,218 次观看 • 16 天前

Today, we are open-sourcing Hunyuan World 1.1 (WorldMirror), a universal feed-forward 3D reconstruction model. 🚀🚀🚀 While our previously released Hunyuan World 1.0 (open-sourced, lite version deployable on consumer GPUs) focused on generating 3D worlds from text or single-view images, Hunyuan World 1.1 significantly expands the input scope by unlocking video-to-3D and multi-view-to-3D world creation. Highlights: 🔹Any Input, Maximized Flexibility and Fidelity: Flexibly integrates diverse geometric priors (camera poses, intrinsics, depth maps) to resolve structural ambiguities and ensure geometrically consistent 3D outputs. 🔹Any Output, SOTA Results：This elegant architecture simultaneously generates multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussian Splattings. 🔹Single-GPU & Fast Inference: As an all-in-one, feed-forward model, Hunyuan World 1.1 runs on a single GPU and delivers all 3D attributes in a single forward pass, within seconds. 🌐Project Page: 🔗Github： 🤗Hugging Face： ✨Demo: 📄Technical Report:

Today, we are open-sourcing Hunyuan World 1.1 (WorldMirror), a universal feed-forward 3D reconstruction model. 🚀🚀🚀 While our previously released Hunyuan World 1.0 (open-sourced, lite version deployable on consumer GPUs) focused on generating 3D worlds from text or single-view images, Hunyuan World 1.1 significantly expands the input scope by unlocking video-to-3D and multi-view-to-3D world creation. Highlights: 🔹Any Input, Maximized Flexibility and Fidelity: Flexibly integrates diverse geometric priors (camera poses, intrinsics, depth maps) to resolve structural ambiguities and ensure geometrically consistent 3D outputs. 🔹Any Output, SOTA Results：This elegant architecture simultaneously generates multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussian Splattings. 🔹Single-GPU & Fast Inference: As an all-in-one, feed-forward model, Hunyuan World 1.1 runs on a single GPU and delivers all 3D attributes in a single forward pass, within seconds. 🌐Project Page: 🔗Github： 🤗Hugging Face： ✨Demo: 📄Technical Report:

Tencent Hy

168,530 次观看 • 9 个月前

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,637 次观看 • 2 年前

🚨 Paper Alert 🚨 ➡️Paper Title: Articulate3D: Zero-Shot Text-Driven 3D Object Posing 🌟Few pointers from the paper 🎯Authors of this paper proposed a training-free method, “Articulate3D”, to pose a 3D asset through language control. 🎯Despite advances in vision and language models, this task remains surprisingly challenging. 🎯To achieve this goal, they decomposed the problem into two steps. 🎯They modified a powerful image-generator to create target images conditioned on the input image and a text instruction. 🎯They then align the mesh to the target images through a multi-view pose optimisation step. 🎯 In detail, they introduced a self-attention rewiring mechanism (RSActrl) that decouples the source structure from pose within an image generative model, allowing it to maintain a consistent structure across varying poses. 🎯They observed that differentiable rendering is an unreliable signal for articulation optimisation; instead, they used keypoints to establish correspondences between input and target images. 🎯The effectiveness of Articulate3D is demonstrated across a diverse range of 3D objects and free-form text prompts, successfully manipulating poses while maintaining the original identity of the mesh. 🎯Quantitative evaluations and a comparative user study, in which their method was preferred over 85% of the time, confirm its superiority over existing approaches. 🏢Organization: University of Oxford , Google DeepMind 🧙Paper Authors: Oishi Deb, Anjun Hu, Ashkan Khakzar, Philip Torr, Christian Rupprecht 📝 Read the Full Paper here: 🗂️ Project Page: 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

🚨 Paper Alert 🚨 ➡️Paper Title: Articulate3D: Zero-Shot Text-Driven 3D Object Posing 🌟Few pointers from the paper 🎯Authors of this paper proposed a training-free method, “Articulate3D”, to pose a 3D asset through language control. 🎯Despite advances in vision and language models, this task remains surprisingly challenging. 🎯To achieve this goal, they decomposed the problem into two steps. 🎯They modified a powerful image-generator to create target images conditioned on the input image and a text instruction. 🎯They then align the mesh to the target images through a multi-view pose optimisation step. 🎯 In detail, they introduced a self-attention rewiring mechanism (RSActrl) that decouples the source structure from pose within an image generative model, allowing it to maintain a consistent structure across varying poses. 🎯They observed that differentiable rendering is an unreliable signal for articulation optimisation; instead, they used keypoints to establish correspondences between input and target images. 🎯The effectiveness of Articulate3D is demonstrated across a diverse range of 3D objects and free-form text prompts, successfully manipulating poses while maintaining the original identity of the mesh. 🎯Quantitative evaluations and a comparative user study, in which their method was preferred over 85% of the time, confirm its superiority over existing approaches. 🏢Organization: University of Oxford , Google DeepMind 🧙Paper Authors: Oishi Deb, Anjun Hu, Ashkan Khakzar, Philip Torr, Christian Rupprecht 📝 Read the Full Paper here: 🗂️ Project Page: 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

naveen manwani

14,334 次观看 • 11 个月前

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Bingyi Kang

67,604 次观看 • 1 年前

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,708 次观看 • 3 年前

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

@levelsio

119,210 次观看 • 1 年前

After a year of team work, we're thrilled to introduce Depth Anything 3 (DA3)! 🚀 Aiming for human-like spatial perception, DA3 extends monocular depth estimation to any-view scenarios, including single images, multi-view images, and video. In pursuit of minimal modeling, DA3 reveals two key insights: 💎 A plain transformer (e.g., vanilla DINO) is enough. No specialized architecture. ✨ A single depth-ray representation is enough. No complex 3D tasks. Three series of models have been released: the main DA3 series, a monocular metric estimation series, and a monocular depth estimation series. The core team members, aside from me: Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen. 👇(1/n) #DepthAnything3

After a year of team work, we're thrilled to introduce Depth Anything 3 (DA3)! 🚀 Aiming for human-like spatial perception, DA3 extends monocular depth estimation to any-view scenarios, including single images, multi-view images, and video. In pursuit of minimal modeling, DA3 reveals two key insights: 💎 A plain transformer (e.g., vanilla DINO) is enough. No specialized architecture. ✨ A single depth-ray representation is enough. No complex 3D tasks. Three series of models have been released: the main DA3 series, a monocular metric estimation series, and a monocular depth estimation series. The core team members, aside from me: Haotong Lin, Sili Chen, Jun Hao Liew, Donny Y. Chen. 👇(1/n) #DepthAnything3

Bingyi Kang

515,068 次观看 • 8 个月前

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Michael Black

21,696 次观看 • 1 年前

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Hitting 76.6% on R2R-CE With One RGB Camera. No LiDAR. No depth sensor. No multi-camera rig. Here's how it works. 👇 1. Pointing, not metric commands The model predicts the pixel coordinates of the next target in the camera view, plus the arrival orientation. Working in pixel space keeps it robust to camera intrinsics and world scale. When the target leaves the frame, it falls back to local displacements ("2m forward, 1.5m left, turn 25°"). 2. Grounding-first No open-source VLM base. It starts from Mistral's grounding model (pointing, counting, localization). Navigation emerges once the model knows where things are. → ~400,000 trajectories across 6,000 simulated scenes 3. Prefix-caching for training A tree-based attention mask packs a full episode into one sequence — all time steps in a single forward pass. → 22× fewer training tokens; months of training done in days 4. Online RL on top After supervised training, CISPO adds trial-and-error learning to fight distribution shift from behavior cloning. → +3.2% success rate from RL alone 5. The numbers (R2R-CE, Matterport3D) → 76.6% success on validation unseen → +9.7 pts over best single-camera approach → +4.5 pts over best depth/multi-camera system The key takeaway: state-of-the-art continuous VLN without a sensor stack — grounding-init, pixel-space actions, prefix-cached SFT, and online RL, on one RGB camera. Full analysis: Technical details: Mistral AI Mistral AI for Developers

Marktechpost AI

39,955 次观看 • 12 天前

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 次观看 • 2 年前

✨ I can now generate 3d assets for my drone sim at directly from Cursor (sponsor of #vibejam) I need buildings that you'd see in a war torn city, like warehouses in ruins, broken down abandoned houses, bombed out bridges etc. Nano Banana Pro or 2 can generate them really well and then you can put them in an image-to-3d model and you get a GLB or FBX That one you can then import into your Three.js game, the models might be big though, in my case like 16MB, so I ask it to compress it and make it more low poly so it loads fast ThreeJS then loads the individual GLBs on page load and puts them in my drone sim somewhere randomly, I think I should remove some of the grass and match the sandy color of the ruins though to make it fit in more

✨ I can now generate 3d assets for my drone sim at directly from Cursor (sponsor of #vibejam) I need buildings that you'd see in a war torn city, like warehouses in ruins, broken down abandoned houses, bombed out bridges etc. Nano Banana Pro or 2 can generate them really well and then you can put them in an image-to-3d model and you get a GLB or FBX That one you can then import into your Three.js game, the models might be big though, in my case like 16MB, so I ask it to compress it and make it more low poly so it loads fast ThreeJS then loads the individual GLBs on page load and puts them in my drone sim somewhere randomly, I think I should remove some of the grass and match the sandy color of the ruins though to make it fit in more

@levelsio

134,777 次观看 • 3 个月前

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: **Highlights:** 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

🚀 The Segment Anything Model (SAM) has been upgraded to SAM2, featuring an efficient image encoder for segmenting images and videos. But does SAM2 outperform SAM1 in medical image and video segmentation? We're thrilled to present our paper "Segment Anything in Medical Images and Videos: Benchmark and Deployment"! We comprehensively benchmark SAM2 across 11 medical image modalities and videos. 📄 Paper: 💻 Code: Highlights: 1. SAM2 doesn’t always outperform SAM1 in 2D medical images, but excels in video segmentation, making it more accurate and efficient for 3D images, such as CT and MR scans. 2. MedSAM still outperforms SAM2 on most 2D modalities, but SAM2 surpasses MedSAM for 3D image segmentation in a slice-by-slice approach. 3. Segmentation performance varies with model size; sometimes the smallest model outperforms larger ones. 4. Fine-tuning SAM2 significantly boosts its performance for medical image segmentation. While SAM2 may struggle with challenging objects that have unclear boundaries or low contrast, it excels in generating good initial segmentation masks for common medical images and videos. However, the official interface doesn’t support medical data formats and has limitations on video length. To address this, we've developed a 3D Slicer Plugin and Gradio API for efficient 3D medical image and video segmentation. We invite you to try them out and provide feedback! 🔧 Deployment: - 3D Slicer Plugin: - Gradio API: (Note: Due to GPU limitations, the online API is available for only 12 hours and may be slow. We highly recommend deploying the Gradio API with your own computing resources: A big shoutout to Jun Ma (JunMa) who recently joined our UHN AI hub (UHN AI Hub) as Machine Learning Lead, and kudos to all co-authors: Sumin Kim, Feifei Li, Mohammed Baharoon (Mohammed Baharoon), Reza Asakereh, and Hongwei Lyu! This is true teamwork! Looking forward to collaborating with the community to advance 3D medical image and video segmentation foundation models! University Health Network U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology Temerty Centre for AI in Medicine (T-CAIREM) Vector Institute #MedTech #AIinHealthcare #DeepLearning #MedicalImaging #SAM2 #MedSAM #AIResearch

Bo Wang

178,481 次观看 • 1 年前

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

AK

280,547 次观看 • 3 年前

You hated the camera, and I hated asking you to be on it. But the truth is, people wanted to see you, they wanted to know about you, they wanted to hear what you think before a match, after a match and everything in between. Because you had something that everyone wanted - you were born a star, Jayden. All I can say is, sorry that the world didn’t do better 😔🙏❤️🕊️ This is the second time that I am making a video like this, the last one was for your best friend Oshwin in 2023 whilst I was still working with the team ❤️ I hope that you two are kicking a soccer ball in heaven 🕊️🕊️ and know that we will forever miss your exceptional talent and presence. #forever23 #forever25

You hated the camera, and I hated asking you to be on it. But the truth is, people wanted to see you, they wanted to know about you, they wanted to hear what you think before a match, after a match and everything in between. Because you had something that everyone wanted - you were born a star, Jayden. All I can say is, sorry that the world didn’t do better 😔🙏❤️🕊️ This is the second time that I am making a video like this, the last one was for your best friend Oshwin in 2023 whilst I was still working with the team ❤️ I hope that you two are kicking a soccer ball in heaven 🕊️🕊️ and know that we will forever miss your exceptional talent and presence. #forever23 #forever25

Jana Swart

162,690 次观看 • 13 天前

🌀🕳️🪐 Why is the wormhole in Interstellar a sphere instead of a flat circle? ⭕️🕳️🪐* Because a real wormhole wouldn't look like a hole in space. It would look like a sphere. In movies and cartoons, wormholes are often shown as flat portals - like a doorway. But in real physics, space isn't 2D. It's three-dimensional. So if a wormhole connects two distant points in space-time, its entrance would extend in all directions. That's why in Interstellar, the wormhole appears as a floating sphere. No matter which direction you approach it from... You're still entering the same tunnel through space-time. The filmmakers actually worked with physicist Kip Thorne to make the depiction as scientifically accurate as possible. So that strange sphere in the movie? It might be the most realistic wormhole ever shown on screen. Sometimes science fiction is closer to real science than we think.*

🌀🕳️🪐 Why is the wormhole in Interstellar a sphere instead of a flat circle? ⭕️🕳️🪐* Because a real wormhole wouldn't look like a hole in space. It would look like a sphere. In movies and cartoons, wormholes are often shown as flat portals - like a doorway. But in real physics, space isn't 2D. It's three-dimensional. So if a wormhole connects two distant points in space-time, its entrance would extend in all directions. That's why in Interstellar, the wormhole appears as a floating sphere. No matter which direction you approach it from... You're still entering the same tunnel through space-time. The filmmakers actually worked with physicist Kip Thorne to make the depiction as scientifically accurate as possible. So that strange sphere in the movie? It might be the most realistic wormhole ever shown on screen. Sometimes science fiction is closer to real science than we think.*

DailyFacts 🕸️

48,927 次观看 • 1 个月前

AI Sprawl is the silent productivity killer. Here's the story of how one engineer built the antidote. We were flipping through ChatGPT, Perplexity, and Gemini, copying and pasting context as well as uploading files from our ClickUp and other apps. Then I had a flashback to before ClickUp existed and realized it's the exact same problem we solved, but for app sprawl. It's freaking happening again, but this time with AI! I had a vivid dream where our 'One app to replace them all' billboards turned into 'One AI to rule them all'. We knew we were meant to solve this problem. Today we're launching Brain MAX. A fully-native desktop app that gives you productivity superpowers... ✨ Use all AI models, from one app. ✨ Connect all your apps to AI - not surface level, but deeply. ✨ Complete tasks, by giving them to AI. ✨ Schedule meetings, send messages, and emails. ✨ Daily agenda with prepped context for each of your meetings. ✨ Talk to Text (4x faster than typing) that writes like you, and even @ mentions names perfectly. This would take 5+ separate apps before today. Brain MAX was built in our innovation labs, where a small but mighty 100x team is cookin' around the clock. Our mission with MAX is to maximize human productivity. Reply "BRAIN" if you want access.

AI Sprawl is the silent productivity killer. Here's the story of how one engineer built the antidote. We were flipping through ChatGPT, Perplexity, and Gemini, copying and pasting context as well as uploading files from our ClickUp and other apps. Then I had a flashback to before ClickUp existed and realized it's the exact same problem we solved, but for app sprawl. It's freaking happening again, but this time with AI! I had a vivid dream where our 'One app to replace them all' billboards turned into 'One AI to rule them all'. We knew we were meant to solve this problem. Today we're launching Brain MAX. A fully-native desktop app that gives you productivity superpowers... ✨ Use all AI models, from one app. ✨ Connect all your apps to AI - not surface level, but deeply. ✨ Complete tasks, by giving them to AI. ✨ Schedule meetings, send messages, and emails. ✨ Daily agenda with prepped context for each of your meetings. ✨ Talk to Text (4x faster than typing) that writes like you, and even @ mentions names perfectly. This would take 5+ separate apps before today. Brain MAX was built in our innovation labs, where a small but mighty 100x team is cookin' around the clock. Our mission with MAX is to maximize human productivity. Reply "BRAIN" if you want access.

Zeb Evans

74,313 次观看 • 1 年前