Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

zhengyiluo's profile picture

Zhengyi “Zen” Luo

5,847 subscribers

92,104 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & TeknolojiEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

9 Yorum

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

Learning physically realistic human-object interaction is one of the major advantage of using humanoids and physics simulation. With Omnigrasp, we learn grasping policy with no paired full-body motion. Here is the results on the Oakink ( which has no paired MoCap data. During inference, all we need is the object mesh and desired trajectory for full body grasping, no need for reference full body pose or grasping point!

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

The center piece of Omnigrasp is PULSE-X, an extension to PULSE ( ICLR 2024) that has fingers. Having this universal and dexterous motion representation enables us to learn faster and scale better. Much like human don't randomly "jitter" to learn to manipulation objects, PULSE-X provides a strong motion prior so the policy can explore much more effectively. And with PULSE-X as the action space, the state and reward designs are incredibly simple! Just object shape, object state, and desired trajectory. We use pre-grasp from PGDM ( from @SudeepDasari) in the reward design and finds it very effective in guiding the policy to learn grasps.

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

Here is the grasps Omnigrasp learned. We do not constrain the policy to certain grasps, and it can learn diverse types of grasps via trial and error / RL. It even learns different grasps for the same object (last row).

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

Big objects? No problem!

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

This work is done at Meta Reality Labs, Pittsburgh, with the team @jinkuncao @sammy_j_c @awinkler_ @kkitani @xuweipeng000. Thanks everyone!!! Many more funny object interaction videos incoming!

You Jiacheng profil fotoğrafı
You Jiacheng1 yıl önce

So reference trajectories must include contact force data?

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

The state contains contact forces, which inform the humanoid of the current contact information of its fingers. c_t is not used in reward computation, so reference trajectory does not need to include force data.

C Zhang profil fotoğrafı
C Zhang1 yıl önce

caught bug

Zhengyi “Zen” Luo profil fotoğrafı
Zhengyi “Zen” Luo1 yıl önce

Problem has been fixed m'lord.

Benzer Videolar

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n
0:49

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n

Homanga Bharadhwaj @ CVPR

28,408 görüntüleme • 1 yıl önce

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥
1:33

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥

Zhengyi “Zen” Luo

15,851 görüntüleme • 7 ay önce

An interesting work on Physical AI: PhysX-Omni. First unified sim-ready generation framework for rigid, deformable, and articulated objects, with a diverse dataset and new benchmark. 🌐 💻 📦
5:03

An interesting work on Physical AI: PhysX-Omni. First unified sim-ready generation framework for rigid, deformable, and articulated objects, with a diverse dataset and new benchmark. 🌐 💻 📦

Ziqi Huang

26,783 görüntüleme • 20 gün önce

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D
0:20

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,425 görüntüleme • 1 yıl önce

We are thrilled to announce our community's plan to Merge with Stress Capital, a community that resonates with our core vision and objective. With this collaboration, we hope to create a more valuable and diverse experience
0:43

We are thrilled to announce our community's plan to Merge with Stress Capital, a community that resonates with our core vision and objective. With this collaboration, we hope to create a more valuable and diverse experience

SEGS DAO

14,057 görüntüleme • 2 yıl önce

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:
0:25

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

Ilir Aliu

37,974 görüntüleme • 1 yıl önce

This is the diverse youth we are forced to live with. Diversity sucks.
0:58

This is the diverse youth we are forced to live with. Diversity sucks.

Libs Hate Us

461,890 görüntüleme • 7 ay önce

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]
0:25

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,392 görüntüleme • 1 yıl önce

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!
0:46

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!

Zipeng Fu

158,126 görüntüleme • 2 yıl önce

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)
1:14

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)

Wenxuan Zhou

49,846 görüntüleme • 3 yıl önce

🚨MEET THE UNITREE G1 HUMANOID Priced between $12,500 and $16,000, it's designed for sports and physical tasks, with 23 to 43 joints for dynamic movement. Its hands, equipped with 2 fingers and a thumb, handle objects with precision. Scared yet?
1:43

🚨MEET THE UNITREE G1 HUMANOID Priced between $12,500 and $16,000, it's designed for sports and physical tasks, with 23 to 43 joints for dynamic movement. Its hands, equipped with 2 fingers and a thumb, handle objects with precision. Scared yet?

Mario Nawfal

248,574 görüntüleme • 2 yıl önce

Let’s defeat the politics of hate and despair with the politics of hope. Honoured to march with the STUC against racism, and for a fair and diverse country that we must remain.
0:23

Let’s defeat the politics of hate and despair with the politics of hope. Honoured to march with the STUC against racism, and for a fair and diverse country that we must remain.

John Swinney

146,073 görüntüleme • 6 ay önce

Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing discuss: We present a novel approach to synthesize dexterous motions for physically simulated hands in tasks that require coordination between the control of two hands with high temporal precision. Instead of directly learning a joint policy to control two hands, our approach performs bimanual control through cooperative learning where each hand is treated as an individual agent. The individual policies for each hand are first trained separately, and then synchronized through latent space manipulation in a centralized environment to serve as a joint policy for two-hand control. By doing so, we avoid directly performing policy learning in the joint state-action space of two hands with higher dimensions, greatly improving the overall training efficiency. We demonstrate the effectiveness of our proposed approach in the challenging guitar-playing task. The virtual guitarist trained by our approach can synthesize motions from unstructured reference data of general guitar-playing practice motions, and accurately play diverse rhythms with complex chord pressing and string picking patterns based on the input guitar tabs that do not exist in the references. Along with this paper, we provide the motion capture data that we collected as the reference for policy training.
6:21

Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing discuss: We present a novel approach to synthesize dexterous motions for physically simulated hands in tasks that require coordination between the control of two hands with high temporal precision. Instead of directly learning a joint policy to control two hands, our approach performs bimanual control through cooperative learning where each hand is treated as an individual agent. The individual policies for each hand are first trained separately, and then synchronized through latent space manipulation in a centralized environment to serve as a joint policy for two-hand control. By doing so, we avoid directly performing policy learning in the joint state-action space of two hands with higher dimensions, greatly improving the overall training efficiency. We demonstrate the effectiveness of our proposed approach in the challenging guitar-playing task. The virtual guitarist trained by our approach can synthesize motions from unstructured reference data of general guitar-playing practice motions, and accurately play diverse rhythms with complex chord pressing and string picking patterns based on the input guitar tabs that do not exist in the references. Along with this paper, we provide the motion capture data that we collected as the reference for policy training.

AK

26,855 görüntüleme • 1 yıl önce

The ultimate test of any physics simulator is its ability to deliver real-world results. With MuJoCo Playground, we’ve combined the very best: MuJoCo’s rich and thriving ecosystem, massively parallel GPU-accelerated simulation, and real-world results across a diverse range of robot platforms: quadrupeds, humanoids, dexterous hands, and arms. Best of all? You can get started today with a single command: pip install playground
1:57

The ultimate test of any physics simulator is its ability to deliver real-world results. With MuJoCo Playground, we’ve combined the very best: MuJoCo’s rich and thriving ecosystem, massively parallel GPU-accelerated simulation, and real-world results across a diverse range of robot platforms: quadrupeds, humanoids, dexterous hands, and arms. Best of all? You can get started today with a single command: pip install playground

Kevin Zakka

153,494 görüntüleme • 1 yıl önce

Ashkenaz, Ethiopian, Sephardi, Mizrachi. A diverse tribe with different looks, languages, food and customs. But… WE ARE ALL JEWS.
1:04

Ashkenaz, Ethiopian, Sephardi, Mizrachi. A diverse tribe with different looks, languages, food and customs. But… WE ARE ALL JEWS.

Leslie Young

50,289 görüntüleme • 5 ay önce

Tired of teleoperating your robots? We built a way to scale robot datasets without teleop, dynamic simulation, or even robot hardware. Just one smartphone scan + one human hand demo video → thousands of diverse robot trajectories. Trainable by diffusion policy and VLA models as-is. Introducing: Real2Render2Real 👉
0:48

Tired of teleoperating your robots? We built a way to scale robot datasets without teleop, dynamic simulation, or even robot hardware. Just one smartphone scan + one human hand demo video → thousands of diverse robot trajectories. Trainable by diffusion policy and VLA models as-is. Introducing: Real2Render2Real 👉

Max Fu

69,224 görüntüleme • 1 yıl önce

China's Kepler Robotics shared a video of its Forerunner K2 humanoid with next-gen hands With an 11-DOF, each 'NimbleMaster' hand can handle a 15 kg payload It also has a 25-element tactile sensing matrix per finger to better grasp different objects
0:44

China's Kepler Robotics shared a video of its Forerunner K2 humanoid with next-gen hands With an 11-DOF, each 'NimbleMaster' hand can handle a 15 kg payload It also has a 25-element tactile sensing matrix per finger to better grasp different objects

Brett Adcock

16,365 görüntüleme • 1 yıl önce

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵
2:37

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵

Xiaolong Wang

309,009 görüntüleme • 2 yıl önce