Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Are black-box VLAs really the endgame for generalist robots? With SAGA, we explore an alternative path to efficiently generalize across environments, behaviors, and user specifications by explicitly grounding visual affordances in 3D space. 🧵

Kuan Fang

3,391 subscribers

198,138 просмотров • 7 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

🎮 We release VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (w/ Junyi Zhang @aomaru_21490) 🌐 With 17 environments across multiple domains, we show systematically the brittleness of VLMs in visual interaction, and what training leads to. 🧵[1/8]

🎮 We release VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (w/ Junyi Zhang @aomaru_21490) 🌐 With 17 environments across multiple domains, we show systematically the brittleness of VLMs in visual interaction, and what training leads to. 🧵[1/8]

Zirui "Colin" Wang

40,493 просмотров • 6 месяцев назад

🧵The quiet revolution in robotics isn't humanoids. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

🧵The quiet revolution in robotics isn't humanoids. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

Lukas Ziegler

479,567 просмотров • 10 месяцев назад

Big thanks to AK for highlighting our work! LEO marks our pioneering step towards building an embodied generalist agent that can really comprehend the 3D world! 🚀Leveraging LLMs, we train LEO with real and synthetic 3D data across a diverse spectrum of tasks. It's thrilling to see LEO surpass current state-of-the-art SOTA methods in most benchmarked tasks, all under a single, unified model. 🔥 #Generalist_Agent

Big thanks to AK for highlighting our work! LEO marks our pioneering step towards building an embodied generalist agent that can really comprehend the 3D world! 🚀Leveraging LLMs, we train LEO with real and synthetic 3D data across a diverse spectrum of tasks. It's thrilling to see LEO surpass current state-of-the-art SOTA methods in most benchmarked tasks, all under a single, unified model. 🔥 #Generalist_Agent

Siyuan Huang

22,710 просмотров • 2 лет назад

NeRF-Insert: Local 3D editing with multimodal control signals by @benoriol et al. Project: 🖼️🚀 Objects are inserted in NeRFs using 2D inpainting models, with editing constrained to a specific region in space 🖍️ User-given coarse masks define the insert region 🎛️ Meshes or visual references are incorporated for variable control. 💾 Efficient: <12GB VRAM Method ⬇️ 1 | 2

NeRF-Insert: Local 3D editing with multimodal control signals by @benoriol et al. Project: 🖼️🚀 Objects are inserted in NeRFs using 2D inpainting models, with editing constrained to a specific region in space 🖍️ User-given coarse masks define the insert region 🎛️ Meshes or visual references are incorporated for variable control. 💾 Efficient: <12GB VRAM Method ⬇️ 1 | 2

MrNeRF

11,384 просмотров • 2 лет назад

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,736 просмотров • 1 год назад

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 просмотров • 1 год назад

Learning reliable manipulation skills for robots is probably the single most important thing for the field to accomplish. Generalist models are usually not reliable enough to achieve the ~100% reliability we really want. But Wenli Xiao has a recipe for achieving this, and for using the data to improve the base model too. Learn more here in the RoboPapers podcast with Michael Cho - Rbt/Acc ->

Learning reliable manipulation skills for robots is probably the single most important thing for the field to accomplish. Generalist models are usually not reliable enough to achieve the ~100% reliability we really want. But Wenli Xiao has a recipe for achieving this, and for using the data to improve the base model too. Learn more here in the RoboPapers podcast with Michael Cho - Rbt/Acc ->

Chris Paxton

35,602 просмотров • 7 месяцев назад

🧵The quiet revolution in robotics isn't humanoid, or world models.. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

🧵The quiet revolution in robotics isn't humanoid, or world models.. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

Lukas Ziegler

171,961 просмотров • 2 месяцев назад

Latest from Archetype AI demo land: This is Newton distinguishing between two people using the same phone. But how? Everyone has a slightly different way of typing. Newton understands the micro-differences in motion, which are mapped clearly in the embedded space (on the right). We never explicitly trained Newton to do this. Our Physical AI foundation model is capable of detecting emergent behaviors — so instead of multiple vertical AI solutions, organizations can leverage Newton to generalize across hundreds of sensor types and support a broad range of use cases. #PhysicalAI #FoundationModel

Latest from Archetype AI demo land: This is Newton distinguishing between two people using the same phone. But how? Everyone has a slightly different way of typing. Newton understands the micro-differences in motion, which are mapped clearly in the embedded space (on the right). We never explicitly trained Newton to do this. Our Physical AI foundation model is capable of detecting emergent behaviors — so instead of multiple vertical AI solutions, organizations can leverage Newton to generalize across hundreds of sensor types and support a broad range of use cases. #PhysicalAI #FoundationModel

Ivan Poupyrev

12,917 просмотров • 5 месяцев назад

🚨 🚨 We are thrilled to launch our 2024 interactive national precinct map. It’s the most ambitious project we’ve ever undertaken. Explore results, shifts from 2020 and 2016, vote density in 3D, and estimates of votes by demographic group for all 50 states+DC. 🧵

🚨 🚨 We are thrilled to launch our 2024 interactive national precinct map. It’s the most ambitious project we’ve ever undertaken. Explore results, shifts from 2020 and 2016, vote density in 3D, and estimates of votes by demographic group for all 50 states+DC. 🧵

VoteHub

964,994 просмотров • 10 месяцев назад

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

AK

684,372 просмотров • 2 лет назад

Reasoning over long horizons would allow robots to generalize better to unseen environments and settings zero-shot. One mechanism for this kind of reasoning would be world models, but traditional video world models still tend to struggle with long horizons, and are very data intensive to train. But what if instead of predicting images about the future, we predicted just the symbolic information necessary for reasoning? Nishanth Kumar tells us about Pixels to Predicates, a method for symbol grounding which allows a VLM to plan sequences of robot skills to achieve unseen goals in previously unseen settings. To find out more, watch episode #44 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

Reasoning over long horizons would allow robots to generalize better to unseen environments and settings zero-shot. One mechanism for this kind of reasoning would be world models, but traditional video world models still tend to struggle with long horizons, and are very data intensive to train. But what if instead of predicting images about the future, we predicted just the symbolic information necessary for reasoning? Nishanth Kumar tells us about Pixels to Predicates, a method for symbol grounding which allows a VLM to plan sequences of robot skills to achieve unseen goals in previously unseen settings. To find out more, watch episode #44 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

19,627 просмотров • 8 месяцев назад

Next week, we are planning to release Patch 0.3.1 with major changes and improvements to the endgame, along with many other fixes and improvements across the board. Watch this video for the broad strokes, and check out more details in today's news post.

Next week, we are planning to release Patch 0.3.1 with major changes and improvements to the endgame, along with many other fixes and improvements across the board. Watch this video for the broad strokes, and check out more details in today's news post.

Path of Exile

499,743 просмотров • 10 месяцев назад

With 𝗮𝗱𝘀 𝗶𝗻 𝗔𝗜 𝗠𝗼𝗱𝗲, your brand doesn’t just show up - 𝗶𝘁 𝗮𝗻𝘀𝘄𝗲𝗿𝘀. At #GML2026, we announced that we are testing a new generation of ad formats in AI Mode. Powered by Gemini, these ads are more visual, capable of reasoning, and bring a new level of helpfulness to both the user and the advertiser. Here are three new formats we are testing to help you reach high-intent customers in AI Mode: 🛒 𝗗𝗶𝗿𝗲𝗰𝘁 𝗢𝗳𝗳𝗲𝗿𝘀 💬 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆 𝗮𝗱𝘀 ✨ 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝗲𝗱 𝗔𝗻𝘀𝘄𝗲𝗿𝘀 Explore everything we announced by visiting Accelerate with Google:👇

Google Ads

25,561 просмотров • 2 месяцев назад

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 просмотров • 3 лет назад

Step into the map with the Street View grounding feature in Project Genie from Google DeepMind and Google Labs. Announced at I/O, this research prototype uses locations from Google Maps Street View as a foundation, letting you generate and explore interactive, 360-degree virtual environments from just a text prompt or real-world starting place. Cool, right? But… How does it actually work? 🤔 As an experimental tool, Project Genie tackles the "blank space" problem (showing both what’s in front of the camera and behind it) by utilizing Street View data to realistically generate a 360-degree view of the location you selected as the starting point for your world to generate from. Worlds generated by Genie are far more dynamic and rich because they’re created frame-by-frame based on the world description and user actions. By predicting each subsequent frame, Genie is able to simulate what it looks like to swim across an ocean, or hike to the top of a peak, marking a massive shift in interactive media and simulation pipelines. What real-world place would you want to step into and explore?

Step into the map with the Street View grounding feature in Project Genie from Google DeepMind and Google Labs. Announced at I/O, this research prototype uses locations from Google Maps Street View as a foundation, letting you generate and explore interactive, 360-degree virtual environments from just a text prompt or real-world starting place. Cool, right? But… How does it actually work? 🤔 As an experimental tool, Project Genie tackles the "blank space" problem (showing both what’s in front of the camera and behind it) by utilizing Street View data to realistically generate a 360-degree view of the location you selected as the starting point for your world to generate from. Worlds generated by Genie are far more dynamic and rich because they’re created frame-by-frame based on the world description and user actions. By predicting each subsequent frame, Genie is able to simulate what it looks like to swim across an ocean, or hike to the top of a peak, marking a massive shift in interactive media and simulation pipelines. What real-world place would you want to step into and explore?

Google AI

2,323,015 просмотров • 22 дней назад

📢 Intrinsic Image Fusion for Multi-View 3D Material Reconstruction 📢 We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! 🌍 🎥 Great work by Peter Kocsis Lukas Höllein!

📢 Intrinsic Image Fusion for Multi-View 3D Material Reconstruction 📢 We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! 🌍 🎥 Great work by Peter Kocsis Lukas Höllein!

Matthias Niessner

23,087 просмотров • 7 месяцев назад

I made a cute 3D wizard in minutes... for free The combo was simple: Unbound + Tripo. Unbound helped me shape the character idea, then Tripo turned it into a 3D model without the usual heavy workflow. The whole process felt fast, playful, and easy to experiment with If you’re a 3D artist, a visual creator, or just starting to explore 3D, try Unbound while it’s still free for a limited time. You can use it to create characters, creatures, and 3D models without feeling overwhelmed I'm sure many visual creators are going to love this Link in the first comment 👇🏻

I made a cute 3D wizard in minutes... for free The combo was simple: Unbound + Tripo. Unbound helped me shape the character idea, then Tripo turned it into a 3D model without the usual heavy workflow. The whole process felt fast, playful, and easy to experiment with If you’re a 3D artist, a visual creator, or just starting to explore 3D, try Unbound while it’s still free for a limited time. You can use it to create characters, creatures, and 3D models without feeling overwhelmed I'm sure many visual creators are going to love this Link in the first comment 👇🏻

Amira Zairi

18,116 просмотров • 3 месяцев назад

Hands are the bridge for the emergence of intelligence. DM-Hand1, developed by Shenzhen DaimonRobotics, is a humanoid hand equipped with visual and tactile sensors, enabling robots to perceive any object and its changes and respond appropriately. This means humanoid robots equipped with this hand show strong potential in precision tasks, cluttered item sorting, and especially in service roles like elderly care.

Hands are the bridge for the emergence of intelligence. DM-Hand1, developed by Shenzhen DaimonRobotics, is a humanoid hand equipped with visual and tactile sensors, enabling robots to perceive any object and its changes and respond appropriately. This means humanoid robots equipped with this hand show strong potential in precision tasks, cluttered item sorting, and especially in service roles like elderly care.

CyberRobo

97,691 просмотров • 1 год назад

Meme of the day! Robots on American streets are being controlled by operators from India In Atlanta, police officers encountered one of these machines in the evening: the robot dog spoke to them in English with an Indian accent and explained that it was “at work” and monitoring order. Such robots are used by security companies for patrols.

Meme of the day! Robots on American streets are being controlled by operators from India In Atlanta, police officers encountered one of these machines in the evening: the robot dog spoke to them in English with an Indian accent and explained that it was “at work” and monitoring order. Such robots are used by security companies for patrols.

NEXTA

68,948 просмотров • 7 месяцев назад