正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Robotic intelligence requires dexterous tool use, but generalizing across tools is hard. Our CoRL23 paper combines semantics (affordances) with low-level control (sim2real) to show functional grasping that generalizes to hammers, drills and more! 1/n

Ananye Agarwal

1,143 subscribers

27,759 次观看 • 2 年前 •via X (Twitter)

科学技术喜剧教育

Anya Rossi• Live Now

Private livecam show

6 条评论

Ananye Agarwal 的头像

Ananye Agarwal2 年前

w/ @shagunuppls @kenny__shaw, @pathak2206 Imagine grasping a hammer. You reach for the handle and not the head because of your understanding of how hammers work. In addition to low-level control (from sim) our robot needs a notion of high-level *functional affordances*. 2/

Ananye Agarwal 的头像

Ananye Agarwal2 年前

We use two complementary sources of data. First, we use internet data in the form of DINO features to get functional correspondences across objects. These generalize across object instances and categories in the wild! 3/

Ananye Agarwal 的头像

Ananye Agarwal2 年前

Second, we use large scale sim data to learn low-level control via RL. Naively training in the 16D joint space can lead to physically implausible behavior. We collect 2hrs of play data to construct an action space of realistic poses for the policy allowing easy transfer. 4/

Ananye Agarwal 的头像

Ananye Agarwal2 年前

Simple reward functions and large scale training lead to emergent behavior where grasps are adaptively adjusted on the fly. If the initial pregrasp is inaccurate, the policy automatically adjusts! 5/

Ananye Agarwal 的头像

Ananye Agarwal2 年前

The surprising part is that the low-level is trained only on hammers, yet it generalizes to many others at test time!! We use our very own custom dexterous hand LEAP hand ( More details Website: Paper: 6/6

Tony 的头像

Tony2 年前

Great job! Will it be open source?

相关视频

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without any robot teleop data collection 1/n

Homanga Bharadhwaj @ CVPR

28,408 次观看 • 1 年前

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Zhengyi “Zen” Luo

92,104 次观看 • 1 年前

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Open-source dexterous hands with fingertip sensors! 🪬 ORCA Dexterity just released three dexterous hand models. Their mission: is to democratize robotic hand dexterity. They're sharing progress on exceptional, low-cost hardware and a software layer from low-level control to robotic hand learning. This is the open-source approach to dexterous manipulation. It has 83 taxels per finger with 0.1 N force detection which pretty is impressive for an open-source design. Tactile sensing is critical for dexterous manipulation, knowing contact forces enables gentle grasping, slip detection, and force-controlled assembly. Also 700g weight for the lite version makes it practical for mounting on robot arms without exceeding payload limits. Lower weight means faster movements and lower torque requirements. Open hardware accelerates robotics by letting researchers and builders modify designs for their specific needs without starting from scratch! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

38,036 次观看 • 3 个月前

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Ilir Aliu

51,490 次观看 • 1 年前

Everyone is talking about Codex and Claude Code... AI Agent platforms that help you do any task that requires a computer... But what about tools to use ALONGSIDE Codex? Here are my top 7: 00:00 Intro 01:18 Tool 1 - Wispr Flow 04:01 Tool 2 - Raycast 06:48 Tool 3 - CleanShot X 10:07 Tool 4 - Paper 14:36 Tool 5 - Readwise 17:06 Tool 6 - Excalidraw 20:03 Tool 7 - BYOT 23:14 Summary

Everyone is talking about Codex and Claude Code... AI Agent platforms that help you do any task that requires a computer... But what about tools to use ALONGSIDE Codex? Here are my top 7: 00:00 Intro 01:18 Tool 1 - Wispr Flow 04:01 Tool 2 - Raycast 06:48 Tool 3 - CleanShot X 10:07 Tool 4 - Paper 14:36 Tool 5 - Readwise 17:06 Tool 6 - Excalidraw 20:03 Tool 7 - BYOT 23:14 Summary

Riley Brown

30,851 次观看 • 1 个月前

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n

Excited to share my final PhD project😀 We show how simple, yet elegant changes enable diffusion transformers to learn SOTA robotic policies on real robots. Our method improves performance by 20% across a wide range of highly dexterous tasks - like cutting sushi! 1/n

Sudeep Dasari

20,536 次观看 • 1 年前

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

Ilir Aliu

37,980 次观看 • 1 年前

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

39,143 次观看 • 1 个月前

San Mateo, California–based Generalist showcases their real-time, end-to-end neural network that controls robots at 100Hz for dexterous manipulation tasks. The model handles fine-motor control, bi-manual coordination, tool use, and generalization across arms and environments.

San Mateo, California–based Generalist showcases their real-time, end-to-end neural network that controls robots at 100Hz for dexterous manipulation tasks. The model handles fine-motor control, bi-manual coordination, tool use, and generalization across arms and environments.

The Humanoid Hub

42,524 次观看 • 1 年前

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

Xiaolong Wang

159,222 次观看 • 2 年前

With OpenAI, Figure 01 can now have full conversations with people -OpenAI models provide high-level visual and language intelligence -Figure neural networks deliver fast, low-level, dexterous robot actions Everything in this video is a neural network:

With OpenAI, Figure 01 can now have full conversations with people -OpenAI models provide high-level visual and language intelligence -Figure neural networks deliver fast, low-level, dexterous robot actions Everything in this video is a neural network:

Figure

5,100,721 次观看 • 2 年前

Using tools is often regarded as the hallmark of intelligence. How can robots creatively use tools beyond standard functionalities and make impossible tasks possible? 🤖🗝️ We introduce RoboTool built on LLMs, enabling multiple creative tool-use behaviors, including tool selection, sequential tool use, and tool manufacturing!🧵👇🏼

Using tools is often regarded as the hallmark of intelligence. How can robots creatively use tools beyond standard functionalities and make impossible tasks possible? 🤖🗝️ We introduce RoboTool built on LLMs, enabling multiple creative tool-use behaviors, including tool selection, sequential tool use, and tool manufacturing!🧵👇🏼

Mengdi Xu

48,204 次观看 • 2 年前

We just released TAVI -- a robotics framework that combines touch and vision to solve challenging dexterous tasks in under 1 hour. The key? Use human demonstrations to initialize a policy, followed by tactile-based online learning with vision-based rewards. Details in🧵(1/7)

We just released TAVI -- a robotics framework that combines touch and vision to solve challenging dexterous tasks in under 1 hour. The key? Use human demonstrations to initialize a policy, followed by tactile-based online learning with vision-based rewards. Details in🧵(1/7)

Lerrel Pinto

138,536 次观看 • 2 年前

New paper introduces NaVILA, a vision-language-action (VLA) model that integrates high-level visual-language understanding and low-level locomotion control. It enables humanoid or quadruped robots to navigate unseen environments with natural language instructions.

New paper introduces NaVILA, a vision-language-action (VLA) model that integrates high-level visual-language understanding and low-level locomotion control. It enables humanoid or quadruped robots to navigate unseen environments with natural language instructions.

The Humanoid Hub

20,430 次观看 • 1 年前

Traditional tokenization methods for robotic actions struggle with high-frequency, dexterous tasks due to redundancy and inefficiency. Inspired by JPEG compression, Physical Intelligence has developed a compressed action representation that accelerates VLA model training 5x.

Traditional tokenization methods for robotic actions struggle with high-frequency, dexterous tasks due to redundancy and inefficiency. Inspired by JPEG compression, Physical Intelligence has developed a compressed action representation that accelerates VLA model training 5x.

The Humanoid Hub

26,198 次观看 • 1 年前

What happens when vision🤝 robotics meet? Happy to share our new work on Pretraining Robotic Foundational Models!🔥 ARM4R is an Autoregressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better robotic model. Berkeley AI Research😊

What happens when vision🤝 robotics meet? Happy to share our new work on Pretraining Robotic Foundational Models!🔥 ARM4R is an Autoregressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better robotic model. Berkeley AI Research😊

Roei Herzig

62,711 次观看 • 1 年前

Gemini + π0 = actually useful robots! (Similar to what Physical Intelligence did with "Hi Robot") I can now verbally tell the robot that I'm building a red Lego wall or wooden tower, and it will infer the next steps by itself and pass me the necessary pieces, tools, or materials, ha! You can also just ask it to bring you things! The pipeline works as follows: - OpenAI Whisper (local) → speech to text - Gemini → makes sense of user requests, converts to robot tasks, bounding boxes, grasping points, etc. (System 2 thinking FTW!) - π0 → robotic actions The π0 was finetuned just for pick-and-place Lego bricks only, and it generalizes beautifully to all kinds of tasks. However, there's lots of room for improvement when it comes to grasping & accuracy. Things that could help: - Conditioning on grasping points - Better data collection (I'm not that great at teleop) - Lots more synthetic data and simulations

Gemini + π0 = actually useful robots! (Similar to what Physical Intelligence did with "Hi Robot") I can now verbally tell the robot that I'm building a red Lego wall or wooden tower, and it will infer the next steps by itself and pass me the necessary pieces, tools, or materials, ha! You can also just ask it to bring you things! The pipeline works as follows: - OpenAI Whisper (local) → speech to text - Gemini → makes sense of user requests, converts to robot tasks, bounding boxes, grasping points, etc. (System 2 thinking FTW!) - π0 → robotic actions The π0 was finetuned just for pick-and-place Lego bricks only, and it generalizes beautifully to all kinds of tasks. However, there's lots of room for improvement when it comes to grasping & accuracy. Things that could help: - Conditioning on grasping points - Better data collection (I'm not that great at teleop) - Lots more synthetic data and simulations

Shreyas Gite

25,485 次观看 • 1 年前

Robots need strong visuo-motor representations to manipulate objects, but it’s hard to learn these using demo data alone. Our #RSS2024 project vastly improves robotic representations, using human affordances mined from Ego4D! w/ Mohan Kumar Srirama Shikhar Bahl Abhinav Gupta

Robots need strong visuo-motor representations to manipulate objects, but it’s hard to learn these using demo data alone. Our #RSS2024 project vastly improves robotic representations, using human affordances mined from Ego4D! w/ Mohan Kumar Srirama Shikhar Bahl Abhinav Gupta

Sudeep Dasari

11,022 次观看 • 1 年前

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.

Evals now supports tool use. 🛠️ You can now use tools and Structured Outputs when completing eval runs, and evaluate tool calls based on the arguments passed and responses returned. This supports tools that are OpenAI-hosted, MCP, and non-hosted. Read more in our guides below.

OpenAI Developers

65,080 次观看 • 1 年前

Hard is good. Hard is necessary. Let’s do our job as parents to give our children the tools and mindset to deal with adversity 🧠

Hard is good. Hard is necessary. Let’s do our job as parents to give our children the tools and mindset to deal with adversity 🧠

Youth Inc.

26,022 次观看 • 1 年前