正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

How can we learn 3D visual grounding with natural supervision—only looking at QA pairs, without ground truth bounding boxes or object classification labels? We inject explicit language priors, e.g., the symmetric property that A near B ⇒ B near A, in structured vision models.

Joy Hsu

2,618 subscribers

14,843 次观看 • 2 年前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

4 条评论

Joy Hsu 的头像

Joy Hsu2 年前

Our Language-Regularized Concept Learner (LARC) extracts language constraints from LLMs, and uses them as regularization on the interpretable & intermediate representations of neuro-symbolic models. LARC's representations encode a variety of logical language properties.

Joy Hsu 的头像

Joy Hsu2 年前

LARC shows strong data efficiency and generalization to new concepts & dataset, and is a promising step towards injecting structured visual reasoning frameworks with explicit language-based priors, for learning in settings without dense visual supervision.

Joy Hsu 的头像

Joy Hsu2 年前

Will be presenting LARC at @CVPR next Thursday morning (session #3) with our fantastic intern @chunfeng3364, as well as @Weiyu_Liu_ and @jiajunwu_cs! Paper: Website: Code:

BEATRIX 李 OBSERVE 💜 ME 的头像

BEATRIX 李 OBSERVE 💜 ME2 年前

my nudes in profile

相关视频

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,736 次观看 • 1 年前

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 次观看 • 3 年前

🎉 恭喜发财🧧🐍 As we welcome the Chinese New Year, we're thrilled to announce the launch of Qwen2.5-VL , our latest flagship vision-language model! 🚀 💗 Qwen Chat: 📖 Blog: 🤗 Hugging Face: 🤖 ModelScope: 🌟 Key Highlights: * Visual Understanding : From flowers to complex charts, Qwen2.5-VL sees it all! * Agentic Capabilities : It’s a visual agent that can reason and interact with tools like computers & phones. * Long Video Comprehension : Captures events in videos over 1 hour long! ⏳🎥 * Precise Localization : Generates bounding boxes & JSON outputs for accurate object detection. * Structured Data Outputs : Perfect for finance & commerce, handling invoices, forms & more! 💼📊 Try Qwen2.5-VL now at Qwen Chat or explore models on Hugging Face & ModelScope . 🌐

🎉 恭喜发财🧧🐍 As we welcome the Chinese New Year, we're thrilled to announce the launch of Qwen2.5-VL , our latest flagship vision-language model! 🚀 💗 Qwen Chat: 📖 Blog: 🤗 Hugging Face: 🤖 ModelScope: 🌟 Key Highlights: * Visual Understanding : From flowers to complex charts, Qwen2.5-VL sees it all! * Agentic Capabilities : It’s a visual agent that can reason and interact with tools like computers & phones. * Long Video Comprehension : Captures events in videos over 1 hour long! ⏳🎥 * Precise Localization : Generates bounding boxes & JSON outputs for accurate object detection. * Structured Data Outputs : Perfect for finance & commerce, handling invoices, forms & more! 💼📊 Try Qwen2.5-VL now at Qwen Chat or explore models on Hugging Face & ModelScope . 🌐

Qwen

762,450 次观看 • 1 年前

Starting the new year without human labeling 🎉!! Multimodal lidar-camera data is a gold mine of dense 3D geometry hiding in plain sight. For supervised pretraining and validation at scale at Torc-Robotics, we rely on fully automated pseudo-labeling pipelines. Exploiting geometric priors from temporally accumulated LiDAR maps and an iterative update rule enforces joint geometric–semantic consistency while detecting moving objects via inconsistencies. We achieve 3D semantic labels and 3D bounding boxes with human-like quality at 200m+ range required for highway driving. Paper: Exciting work with Torc-Robotics with Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic.

Starting the new year without human labeling 🎉!! Multimodal lidar-camera data is a gold mine of dense 3D geometry hiding in plain sight. For supervised pretraining and validation at scale at Torc-Robotics, we rely on fully automated pseudo-labeling pipelines. Exploiting geometric priors from temporally accumulated LiDAR maps and an iterative update rule enforces joint geometric–semantic consistency while detecting moving objects via inconsistencies. We achieve 3D semantic labels and 3D bounding boxes with human-like quality at 200m+ range required for highway driving. Paper: Exciting work with Torc-Robotics with Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic.

Felix Heide

18,210 次观看 • 7 个月前

In Prompt Engineering for Vision Models, taught by Abby Jacques Verre and Caleb Kaiser of Comet , you’ll learn how to prompt and fine-tune vision models for personalized image generation, image editing, object detection and segmentation. The prompts you'll use for vision models could be text, point coordinates, or bounding boxes, depending on the model. You'll also learn to tune hyperparameters to shape the output. Models you'll use include Segment-Anything Model (SAM), OWL-ViT, and Stable Diffusion. You'll also learn to fine-tune Stable Diffusion to generate personalized images (say, an image of a specific person), using a handful of images for training. As an example of a multi-step workflow, you'll use OWL-ViT to detect an object based on a text prompt, then pass the bounding box to SAM to create a segmentation mask, and input that mask into Stable Diffusion to replace the original object with a new one based on a text prompt. Controlling vision models can be tricky; this course will teach prompting and fine-tuning techniques to get precise control over their output. Get started here:

In Prompt Engineering for Vision Models, taught by Abby Jacques Verre and Caleb Kaiser of Comet , you’ll learn how to prompt and fine-tune vision models for personalized image generation, image editing, object detection and segmentation. The prompts you'll use for vision models could be text, point coordinates, or bounding boxes, depending on the model. You'll also learn to tune hyperparameters to shape the output. Models you'll use include Segment-Anything Model (SAM), OWL-ViT, and Stable Diffusion. You'll also learn to fine-tune Stable Diffusion to generate personalized images (say, an image of a specific person), using a handful of images for training. As an example of a multi-step workflow, you'll use OWL-ViT to detect an object based on a text prompt, then pass the bounding box to SAM to create a segmentation mask, and input that mask into Stable Diffusion to replace the original object with a new one based on a text prompt. Controlling vision models can be tricky; this course will teach prompting and fine-tuning techniques to get precise control over their output. Get started here:

Andrew Ng

151,198 次观看 • 2 年前

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI Breakfast

33,725 次观看 • 3 年前

How far can a very simple eye go in solving vision tasks? Like a 1-pixel camera? Humans have one of the greatest eyes in nature, while many animals have significantly simpler eyes and visual systems yet show complex perceptual behavior. In an interesting project, we find that many computer vision tasks can be solved without a typical camera and with such simple 1-pixel sensors (photoreceptors). We also find that proper design (e.g., where to place the photoreceptors strategically) makes a big difference, so we developed a computational design method to find them. 🌐 👁️[Solving Vision Tasks with Simple Photoreceptors Instead of Cameras] 🧵1/n

How far can a very simple eye go in solving vision tasks? Like a 1-pixel camera? Humans have one of the greatest eyes in nature, while many animals have significantly simpler eyes and visual systems yet show complex perceptual behavior. In an interesting project, we find that many computer vision tasks can be solved without a typical camera and with such simple 1-pixel sensors (photoreceptors). We also find that proper design (e.g., where to place the photoreceptors strategically) makes a big difference, so we developed a computational design method to find them. 🌐 👁️[Solving Vision Tasks with Simple Photoreceptors Instead of Cameras] 🧵1/n

Amir Zamir

75,892 次观看 • 2 年前

Creator of C#, Anders Hejlsberg(Anders Hejlsberg), on the design goals that shaped the language and why standardisation was a goal from the beginning: “The overarching thing was this power and productivity of C++ with the ease of use of Visual Basic. We knew we wanted to build an object oriented language. We wanted managed code or byte code so we could target different runtime environments. We wanted garbage collection and exception handling, but also things like a unified object system where, and that's true in C#, anything can be assigned to an object. If it's a value type, we box it. And it's a self-describing object. So, reflection: you can ask an object, ‘what are you’? And you can get all of the facts about it at runtime, and you can dynamically manipulate it in ways that just don't exist in a lot of other environments. We wanted a language that made this new model of properties, methods and events first class because that was how components were built as opposed to just sort of functions and procedures and even objects. And we wanted to create a language that was standardised. We wanted to give this language to a standardisation committee and try to level the playing field there. And all of those things were sort of like what was rolled up in C#”

Creator of C#, Anders Hejlsberg(Anders Hejlsberg), on the design goals that shaped the language and why standardisation was a goal from the beginning: “The overarching thing was this power and productivity of C++ with the ease of use of Visual Basic. We knew we wanted to build an object oriented language. We wanted managed code or byte code so we could target different runtime environments. We wanted garbage collection and exception handling, but also things like a unified object system where, and that's true in C#, anything can be assigned to an object. If it's a value type, we box it. And it's a self-describing object. So, reflection: you can ask an object, ‘what are you’? And you can get all of the facts about it at runtime, and you can dynamically manipulate it in ways that just don't exist in a lot of other environments. We wanted a language that made this new model of properties, methods and events first class because that was how components were built as opposed to just sort of functions and procedures and even objects. And we wanted to create a language that was standardised. We wanted to give this language to a standardisation committee and try to level the playing field there. And all of those things were sort of like what was rolled up in C#”

The Pragmatic Engineer

18,332 次观看 • 2 个月前

🚨 New Paper Hybrid Neural World Models. We build neural world models for discontinuous environments and our method discovers when they're making error, with no ground truth. Our neural models run physics fast, leaping many simulator steps in a single pass. How we find points where not to trust it: ask the model for the same future two ways, and the disagreement between the two answers marks where it's failing without needing labels. The same recipe holds on three systems with nothing in common: a chemical wave, compressible gas with shocks, rigid bodies colliding in 3D. 👇

🚨 New Paper Hybrid Neural World Models. We build neural world models for discontinuous environments and our method discovers when they're making error, with no ground truth. Our neural models run physics fast, leaping many simulator steps in a single pass. How we find points where not to trust it: ask the model for the same future two ways, and the disagreement between the two answers marks where it's failing without needing labels. The same recipe holds on three systems with nothing in common: a chemical wave, compressible gas with shocks, rigid bodies colliding in 3D. 👇

Lossfunk

15,037 次观看 • 2 个月前

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 次观看 • 3 年前

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 次观看 • 3 年前

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

Matthias Niessner

19,005 次观看 • 4 个月前

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

🚀 New Paper: Pixel Reasoner 🧠🖼️ How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself? We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning. Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn. 🔍 So we ask: What if we could make VLMs "show their work" by reasoning directly in the pixel space? Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs. To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning ✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks: 📊 84% on InfographicsVQA 🧠 84% on V* benchmark 🧩 74% on TallyQA-Complex It also achieves strong accuracy of 68% on MVBench (a video benchmark). Website: Paper: Code: Demo: (coming soon)

Wenhu Chen

82,829 次观看 • 1 年前

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Large-scale 3D Scene Generation (all scenes are real-time rendered)!! Physically-grounded generative data without hallucinations is the missing link for robot learning and testing at scale. We introduce a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal view synthesis and generation with object permanence and explicit 3D geometry. This also allows for extreme trajectory extrapolation without failure! We also show that we can build fully data-driven simulators for end-to-end learning with this approach. Project: with the amazing team of Julian Ost, Amogh Joshi , Andrea Ramazzina, Maximilian Bömer, Mario Bijelic.

Felix Heide

27,779 次观看 • 11 个月前

Jeff Bezos on his favorite interview question “When I interview people, I ask them to give me an example of something they've invented. And I always point out, it doesn't have to be something that you actually took to the patent office. It could be a metric that you invented and followed carefully. It could be a business process that you invented. You want to select people who like to invent their way out of boxes.” A lot of people will immediately jump to what Jeff calls an “either or” solution (e.g. “We can do A or B.”). But as Jeff explains: “The right question is how can we do A and B? What invention do we need to be able to do both?”

Jeff Bezos on his favorite interview question “When I interview people, I ask them to give me an example of something they've invented. And I always point out, it doesn't have to be something that you actually took to the patent office. It could be a metric that you invented and followed carefully. It could be a business process that you invented. You want to select people who like to invent their way out of boxes.” A lot of people will immediately jump to what Jeff calls an “either or” solution (e.g. “We can do A or B.”). But as Jeff explains: “The right question is how can we do A and B? What invention do we need to be able to do both?”

Startup Archive

175,614 次观看 • 1 个月前

Jeff Bezos on his favorite interview question “When I interview people, I ask them to give me an example of something they've invented. And I always point out, it doesn't have to be something that you actually took to the patent office. It could be a metric that you invented and followed carefully. It could be a business process that you invented. You want to select people who like to invent their way out of boxes.” A lot of people will immediately jump to what Jeff calls an “either or” solution (e.g. “We can do A or B.”). But as Jeff explains: “The right question is how can we do A and B? What invention do we need to be able to do both?”

Jeff Bezos on his favorite interview question “When I interview people, I ask them to give me an example of something they've invented. And I always point out, it doesn't have to be something that you actually took to the patent office. It could be a metric that you invented and followed carefully. It could be a business process that you invented. You want to select people who like to invent their way out of boxes.” A lot of people will immediately jump to what Jeff calls an “either or” solution (e.g. “We can do A or B.”). But as Jeff explains: “The right question is how can we do A and B? What invention do we need to be able to do both?”

Startup Archive

1,846,369 次观看 • 8 个月前

"With Auston Matthews, we are one year away from having that conversation" Darren Dreger shared how the Brady Tkachuk trade will affect #LeafsForever and if they will be looking at a similar situation with Matthews in the near future, but why it's premature this offseason.

"With Auston Matthews, we are one year away from having that conversation" Darren Dreger shared how the Brady Tkachuk trade will affect #LeafsForever and if they will be looking at a similar situation with Matthews in the near future, but why it's premature this offseason.

First Up

32,504 次观看 • 1 个月前

SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians Contributions: • We propose SuperGSeg: a 3D segmentation method with neural Gaussians, designed to learn hierarchical instance segmentation features from 2D foundation models. • We introduce the concept of Super-Gaussian, a novel representation that integrates hierarchical instance segmentation features, enabling the embedding of high-dimensional language features. This approach addresses previously unfeasible challenges in representing complex scenes with rich semantic details. • Extensive experiments on the LERF-OVS and ScanNet datasets demonstrate the effectiveness of the proposed method, achieving significant improvements in open-vocabulary 3D object-level and scene-level semantic segmentation. It shows particular strength in capturing fine-grained scene details and dense pixel semantic segmentation tasks for the first time.

SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians Contributions: • We propose SuperGSeg: a 3D segmentation method with neural Gaussians, designed to learn hierarchical instance segmentation features from 2D foundation models. • We introduce the concept of Super-Gaussian, a novel representation that integrates hierarchical instance segmentation features, enabling the embedding of high-dimensional language features. This approach addresses previously unfeasible challenges in representing complex scenes with rich semantic details. • Extensive experiments on the LERF-OVS and ScanNet datasets demonstrate the effectiveness of the proposed method, achieving significant improvements in open-vocabulary 3D object-level and scene-level semantic segmentation. It shows particular strength in capturing fine-grained scene details and dense pixel semantic segmentation tasks for the first time.

MrNeRF

13,594 次观看 • 1 年前

SELF LANGUAGE AND HEALTH How we talk about ourselves matters The power of placebo, and its opposite nocebo - means that what we believe, and how we reinforce that with our language can literally make us sick or healthy For this reason, learn to see your symptoms as a healing process, rather than the sickness itself to be avoided Be HAPPY that your body is healing and cleansing you, and your recovery will be improved because of this Once you learn the truth, that you can’t catch a cold; that it’s not a bad thing to clear out garbage (although it may feel rough), then sickness becomes much easier to handle

SELF LANGUAGE AND HEALTH How we talk about ourselves matters The power of placebo, and its opposite nocebo - means that what we believe, and how we reinforce that with our language can literally make us sick or healthy For this reason, learn to see your symptoms as a healing process, rather than the sickness itself to be avoided Be HAPPY that your body is healing and cleansing you, and your recovery will be improved because of this Once you learn the truth, that you can’t catch a cold; that it’s not a bad thing to clear out garbage (although it may feel rough), then sickness becomes much easier to handle

⚡️🌞 Sol Brah 🌞🐬

14,199 次观看 • 1 年前