Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

How to chain multiple dexterous skills to tackle complex long-horizon manipulation tasks? Imagine retrieving a LEGO block from a pile, rotating it in-hand, and inserting it at the desired location to build a structure. Introducing our new work - Sequential Dexterity 🧵👇

Chen Wang

3,414 subscribers

159,809 görüntüleme • 3 yıl önce •via X (Twitter)

Eğitim Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

11 Yorum

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

The core of the system is a learning-based transition feasibility function that progressively finetunes the sub-policies (learned with RL) for enhancing chaining success, which can also be used during skill selection for re-planning from failures and bypassing redundant stages.

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

Despite being trained only in simulation with a few task objects, our system demonstrates generalization capability to novel object shapes and is able to zero-shot transfer to a real-world robot equipped with a dexterous hand.

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

We hope Sequential Dexterity paves the path for future research on long-horizon dexterous manipulation. Feel free to check out our code! Website & Paper: Code: Work done w/ Yuanpei Chen, @drfeifei, and Karen Liu at @StanfordAILab.

Karol Hausman profil fotoğrafı

Karol Hausman3 yıl önce

Congrats, great work!

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

Thanks Karol!

Deepak Pathak profil fotoğrafı

Deepak Pathak3 yıl önce

very cool task and results, congrats!!

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

Thanks Deepak! Very interested in LEAP hand and functional grasp

Lucy Shi profil fotoğrafı

Lucy Shi3 yıl önce

super cool work, @chenwang_j!

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

Thanks Lucy!

Kevin Zakka profil fotoğrafı

Kevin Zakka3 yıl önce

Very inspiring, congrats!

Chen Wang profil fotoğrafı

Chen Wang3 yıl önce

Thanks Kevin! huge fan of RoboPianist!

Benzer Videolar

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Mandi Zhao

121,807 görüntüleme • 1 yıl önce

Introducing 2️⃣ new AI systems for robotics: 🤖 ALOHA Unleashed to perform two-armed manipulation tasks 🦾 DemoStart to control a multi-fingered robotic hand They learned to tackle a range of actions requiring dexterity. Here's how. 🧵

Introducing 2️⃣ new AI systems for robotics: 🤖 ALOHA Unleashed to perform two-armed manipulation tasks 🦾 DemoStart to control a multi-fingered robotic hand They learned to tackle a range of actions requiring dexterity. Here's how. 🧵

Google DeepMind

297,345 görüntüleme • 2 yıl önce

Teaching bimanual robot hands to perform very complex tasks has been notoriously challenging. In our work, Bidex: Bimanual Dexterity for Complex Tasks, we’ve developed a low-cost system that completes a wide range of highly dexterous tasks in real-time.

Teaching bimanual robot hands to perform very complex tasks has been notoriously challenging. In our work, Bidex: Bimanual Dexterity for Complex Tasks, we’ve developed a low-cost system that completes a wide range of highly dexterous tasks in real-time.

Kenny Shaw

40,146 görüntüleme • 1 yıl önce

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen*, Chen Wang*, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen, Chen Wang, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

Jim Fan

69,064 görüntüleme • 3 yıl önce

Introducing /boost in Google Antigravity. The Antigravity harness can tackle a wide range of tasks right out of the box, but /boost helps if you are tackling particularly complex tasks and are willing to spend some more tokens to push the system to reason deeply about the challenge at hand. Here is how it works ⬇️

Introducing /boost in Google Antigravity. The Antigravity harness can tackle a wide range of tasks right out of the box, but /boost helps if you are tackling particularly complex tasks and are willing to spend some more tokens to push the system to reason deeply about the challenge at hand. Here is how it works ⬇️

Google Antigravity

200,684 görüntüleme • 28 gün önce

🚀🚀🚀 Ever wondered what it takes for robots to handle real-world household tasks? long-horizon execution, deformable object dexterity, and unseen object generalization — meet GR-3, ByteDance Seed’s new Vision-Language-Action (VLA) model! GR-3 is a generalizable Vision-Language-Action (VLA) model with strong capabilities in complex long-horizon tasks. It understands unseen abstract concepts, manipulates deformable objects robustly, and adapts to novel settings with minimal human data. ✨ Generalization: Generalizes well to unseen objects, environments, and even instructions with abstract concepts. ✨ Long-Horizon Manipulation: Completes long-horizon tasks with strong instruction-following capabilities. ✨ Deformable Object Manipulation: Manipulate deformable objects robustly. Project Page: Arxiv: #ByteDance #ByteDanceSeed #GR3 #VLA #Robotics #FoundationModels

🚀🚀🚀 Ever wondered what it takes for robots to handle real-world household tasks? long-horizon execution, deformable object dexterity, and unseen object generalization — meet GR-3, ByteDance Seed’s new Vision-Language-Action (VLA) model! GR-3 is a generalizable Vision-Language-Action (VLA) model with strong capabilities in complex long-horizon tasks. It understands unseen abstract concepts, manipulates deformable objects robustly, and adapts to novel settings with minimal human data. ✨ Generalization: Generalizes well to unseen objects, environments, and even instructions with abstract concepts. ✨ Long-Horizon Manipulation: Completes long-horizon tasks with strong instruction-following capabilities. ✨ Deformable Object Manipulation: Manipulate deformable objects robustly. Project Page: Arxiv: #ByteDance #ByteDanceSeed #GR3 #VLA #Robotics #FoundationModels

Xiao Ma

46,426 görüntüleme • 1 yıl önce

A few weeks ago, we shared our progress on articulated objects and long-horizon tasks. Here are two representative examples: - We've been steadily expanding our asset library to cover more articulated objects. Articulated objects have always been a challenging asset class to handle in simulation. Interacting with them requires robots to master atomic skills such as pushing, pulling, opening, and closing, and to understand part structure, interaction constraints, and how the object moves. - Long-horizon tasks can now be generated at scale. Long-horizon tasks are the other hard category: they require chaining multiple sub-goals in sequence. A failure early in the task can cascade and make the rest unrecoverable. Axis is scaling along three dimensions at once: data volume, data quality, and task difficulty.

A few weeks ago, we shared our progress on articulated objects and long-horizon tasks. Here are two representative examples: - We've been steadily expanding our asset library to cover more articulated objects. Articulated objects have always been a challenging asset class to handle in simulation. Interacting with them requires robots to master atomic skills such as pushing, pulling, opening, and closing, and to understand part structure, interaction constraints, and how the object moves. - Long-horizon tasks can now be generated at scale. Long-horizon tasks are the other hard category: they require chaining multiple sub-goals in sequence. A failure early in the task can cascade and make the rest unrecoverable. Axis is scaling along three dimensions at once: data volume, data quality, and task difficulty.

Axis Robotics

13,700 görüntüleme • 3 ay önce

🎙️ Excited to introduce one of my favorite projects from the past year: TeleDexter, from the BIGAI dexterity team. It’s a stable, human-level dexterous teleoperation system and a suite of autonomous policies trained with it. Pen spinning, complex in-hand reorientation, and long-horizon tool use—once seen as the holy grail of manipulation—are now unlocked. 🧵👇 The hardware is already here; we have some incredible high-DoF robotic hands. The bottleneck? The controller. Most current systems are stuck in "quasi-static" grasping mode. Meanwhile, dynamic in-hand dexterity has remained severely limited. 🧠 To unlock the massive capabilities of human-like hands, we need to build an excellent "cerebellum" for dexterous hands. TeleDexter solves this with a novel co-tracking approach: it simultaneously tracks both human hand kinematics and object states, beautifully bridging the gap between human intent and robotic control. In order to train a better co-tracking policy that works robustly in the real world, we designed : (1) a hybrid reward design that combines consecutive goal reaching and dense tracking, (2) an action masking strategy during training that enhances sim2real performance, (3) a dexterous curriculum for learning the long-horizon interactions. Each design is inspired by numerous trials and countless real-world experiments. We’ve synthesized all the system details, engineering challenges, and core insights into our latest post. If you're interested in the future of dexterous manipulation, grab a coffee and check it out (9-min read): If you have more time, check out the paper:

🎙️ Excited to introduce one of my favorite projects from the past year: TeleDexter, from the BIGAI dexterity team. It’s a stable, human-level dexterous teleoperation system and a suite of autonomous policies trained with it. Pen spinning, complex in-hand reorientation, and long-horizon tool use—once seen as the holy grail of manipulation—are now unlocked. 🧵👇 The hardware is already here; we have some incredible high-DoF robotic hands. The bottleneck? The controller. Most current systems are stuck in "quasi-static" grasping mode. Meanwhile, dynamic in-hand dexterity has remained severely limited. 🧠 To unlock the massive capabilities of human-like hands, we need to build an excellent "cerebellum" for dexterous hands. TeleDexter solves this with a novel co-tracking approach: it simultaneously tracks both human hand kinematics and object states, beautifully bridging the gap between human intent and robotic control. In order to train a better co-tracking policy that works robustly in the real world, we designed : (1) a hybrid reward design that combines consecutive goal reaching and dense tracking, (2) an action masking strategy during training that enhances sim2real performance, (3) a dexterous curriculum for learning the long-horizon interactions. Each design is inspired by numerous trials and countless real-world experiments. We’ve synthesized all the system details, engineering challenges, and core insights into our latest post. If you're interested in the future of dexterous manipulation, grab a coffee and check it out (9-min read): If you have more time, check out the paper:

Siyuan Huang

12,606 görüntüleme • 2 ay önce

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 görüntüleme • 2 yıl önce

Tip: Use Leo's Multi-Tab Context to draw information from several tabs at once. 📚 It's a great way to complete complex tasks with our AI assistant. Here, we're using it to build a shopping list based on multiple recipes.

Tip: Use Leo's Multi-Tab Context to draw information from several tabs at once. 📚 It's a great way to complete complex tasks with our AI assistant. Here, we're using it to build a shopping list based on multiple recipes.

Brave

65,096 görüntüleme • 1 yıl önce

At the Film Culture Carnival in Ningbo, a robot steps onto a film set, taking on tasks from camera work to visual effects support. Watch the video to see it in action. [Video/Hi New Horizon]

At the Film Culture Carnival in Ningbo, a robot steps onto a film set, taking on tasks from camera work to visual effects support. Watch the video to see it in action. [Video/Hi New Horizon]

Insight Ningbo

99,590 görüntüleme • 5 ay önce

Thrilled to share one of my favorite works this year: DexNDM! We bridge the Sim2Real gap for dexterous in-hand rotation, achieving a true "0-to-1" advancement. The key? DexNDM learns from biased, real-world data without needing any successful demonstrations. Now a general-purpose dexterous hand can stably rotate large books, long rods, & complex objects around any axis, from any wrist pose. This powerful primitive enables complex, long-horizon tasks like teleoperated screwing and furniture assembly. 📄 Paper: 🌐 Project Page:

Thrilled to share one of my favorite works this year: DexNDM! We bridge the Sim2Real gap for dexterous in-hand rotation, achieving a true "0-to-1" advancement. The key? DexNDM learns from biased, real-world data without needing any successful demonstrations. Now a general-purpose dexterous hand can stably rotate large books, long rods, & complex objects around any axis, from any wrist pose. This powerful primitive enables complex, long-horizon tasks like teleoperated screwing and furniture assembly. 📄 Paper: 🌐 Project Page:

Li Yi

38,959 görüntüleme • 10 ay önce

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: (1/4)

Ryan Hoque

114,314 görüntüleme • 1 yıl önce

Introducing GPT-5.5 A new class of intelligence for real work and powering agents, built to understand complex goals, use tools, check its work, and carry more tasks through to completion. It marks a new way of getting computer work done. Now available in ChatGPT and Codex.

Introducing GPT-5.5 A new class of intelligence for real work and powering agents, built to understand complex goals, use tools, check its work, and carry more tasks through to completion. It marks a new way of getting computer work done. Now available in ChatGPT and Codex.

OpenAI

13,229,282 görüntüleme • 5 ay önce

Introducing RoboCat, a new AI model designed to operate multiple robots. 🤖 It learns to solve new tasks on different robotic arms with as few as 100 demonstrations - and improves skills from self-generated training data. Find out more:

Introducing RoboCat, a new AI model designed to operate multiple robots. 🤖 It learns to solve new tasks on different robotic arms with as few as 100 demonstrations - and improves skills from self-generated training data. Find out more:

Google DeepMind

410,427 görüntüleme • 3 yıl önce