正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Zhengyi “Zen” Luo

5,847 subscribers

92,104 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

9 条评论

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

Learning physically realistic human-object interaction is one of the major advantage of using humanoids and physics simulation. With Omnigrasp, we learn grasping policy with no paired full-body motion. Here is the results on the Oakink ( which has no paired MoCap data. During inference, all we need is the object mesh and desired trajectory for full body grasping, no need for reference full body pose or grasping point!

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

The center piece of Omnigrasp is PULSE-X, an extension to PULSE ( ICLR 2024) that has fingers. Having this universal and dexterous motion representation enables us to learn faster and scale better. Much like human don't randomly "jitter" to learn to manipulation objects, PULSE-X provides a strong motion prior so the policy can explore much more effectively. And with PULSE-X as the action space, the state and reward designs are incredibly simple! Just object shape, object state, and desired trajectory. We use pre-grasp from PGDM ( from @SudeepDasari) in the reward design and finds it very effective in guiding the policy to learn grasps.

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

Here is the grasps Omnigrasp learned. We do not constrain the policy to certain grasps, and it can learn diverse types of grasps via trial and error / RL. It even learns different grasps for the same object (last row).

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

Big objects? No problem!

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

This work is done at Meta Reality Labs, Pittsburgh, with the team @jinkuncao @sammy_j_c @awinkler_ @kkitani @xuweipeng000. Thanks everyone!!! Many more funny object interaction videos incoming!

You Jiacheng 的头像

You Jiacheng1 年前

So reference trajectories must include contact force data?

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

The state contains contact forces, which inform the humanoid of the current contact information of its fingers. c_t is not used in reward computation, so reference trajectory does not need to include force data.

C Zhang 的头像

C Zhang1 年前

caught bug

Zhengyi “Zen” Luo 的头像

Zhengyi “Zen” Luo1 年前

Problem has been fixed m'lord.

相关视频

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without *any* robot teleop data collection 1/n

Humans grasp objects with a purpose! Web2Grasp enables such functional grasping for dexterous robot hands via hand-object reconstruction from web images - without any robot teleop data collection 1/n

Homanga Bharadhwaj @ CVPR

28,408 次观看 • 1 年前

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥

Zhengyi “Zen” Luo

15,851 次观看 • 7 个月前

An interesting work on Physical AI: PhysX-Omni. First unified sim-ready generation framework for rigid, deformable, and articulated objects, with a diverse dataset and new benchmark. 🌐 💻 📦

An interesting work on Physical AI: PhysX-Omni. First unified sim-ready generation framework for rigid, deformable, and articulated objects, with a diverse dataset and new benchmark. 🌐 💻 📦

Ziqi Huang

26,783 次观看 • 23 天前

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,425 次观看 • 1 年前

We are thrilled to announce our community's plan to Merge with Stress Capital, a community that resonates with our core vision and objective. With this collaboration, we hope to create a more valuable and diverse experience

We are thrilled to announce our community's plan to Merge with Stress Capital, a community that resonates with our core vision and objective. With this collaboration, we hope to create a more valuable and diverse experience

SEGS DAO

14,057 次观看 • 2 年前

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

Ilir Aliu

37,974 次观看 • 1 年前

This is the diverse youth we are forced to live with. Diversity sucks.

This is the diverse youth we are forced to live with. Diversity sucks.

Libs Hate Us

462,063 次观看 • 7 个月前

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,392 次观看 • 1 年前

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)

How can robots learn generalizable manipulation skills for diverse objects? Going beyond pick-and-place, our recent work “HACMan” enables complex interactions for unseen objects, such as flipping, pushing, or tilting, using spatial action maps + RL with point clouds. (w/ @MetaAI)

Wenxuan Zhou

49,846 次观看 • 3 年前

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!

Zipeng Fu

158,140 次观看 • 2 年前

🚨MEET THE UNITREE G1 HUMANOID Priced between $12,500 and $16,000, it's designed for sports and physical tasks, with 23 to 43 joints for dynamic movement. Its hands, equipped with 2 fingers and a thumb, handle objects with precision. Scared yet?

🚨MEET THE UNITREE G1 HUMANOID Priced between $12,500 and $16,000, it's designed for sports and physical tasks, with 23 to 43 joints for dynamic movement. Its hands, equipped with 2 fingers and a thumb, handle objects with precision. Scared yet?

Mario Nawfal

248,574 次观看 • 2 年前

Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing discuss: We present a novel approach to synthesize dexterous motions for physically simulated hands in tasks that require coordination between the control of two hands with high temporal precision. Instead of directly learning a joint policy to control two hands, our approach performs bimanual control through cooperative learning where each hand is treated as an individual agent. The individual policies for each hand are first trained separately, and then synchronized through latent space manipulation in a centralized environment to serve as a joint policy for two-hand control. By doing so, we avoid directly performing policy learning in the joint state-action space of two hands with higher dimensions, greatly improving the overall training efficiency. We demonstrate the effectiveness of our proposed approach in the challenging guitar-playing task. The virtual guitarist trained by our approach can synthesize motions from unstructured reference data of general guitar-playing practice motions, and accurately play diverse rhythms with complex chord pressing and string picking patterns based on the input guitar tabs that do not exist in the references. Along with this paper, we provide the motion capture data that we collected as the reference for policy training.

Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing discuss: We present a novel approach to synthesize dexterous motions for physically simulated hands in tasks that require coordination between the control of two hands with high temporal precision. Instead of directly learning a joint policy to control two hands, our approach performs bimanual control through cooperative learning where each hand is treated as an individual agent. The individual policies for each hand are first trained separately, and then synchronized through latent space manipulation in a centralized environment to serve as a joint policy for two-hand control. By doing so, we avoid directly performing policy learning in the joint state-action space of two hands with higher dimensions, greatly improving the overall training efficiency. We demonstrate the effectiveness of our proposed approach in the challenging guitar-playing task. The virtual guitarist trained by our approach can synthesize motions from unstructured reference data of general guitar-playing practice motions, and accurately play diverse rhythms with complex chord pressing and string picking patterns based on the input guitar tabs that do not exist in the references. Along with this paper, we provide the motion capture data that we collected as the reference for policy training.

AK

26,855 次观看 • 1 年前

Let’s defeat the politics of hate and despair with the politics of hope. Honoured to march with the STUC against racism, and for a fair and diverse country that we must remain.

Let’s defeat the politics of hate and despair with the politics of hope. Honoured to march with the STUC against racism, and for a fair and diverse country that we must remain.

John Swinney

146,073 次观看 • 6 个月前

Ashkenaz, Ethiopian, Sephardi, Mizrachi. A diverse tribe with different looks, languages, food and customs. But… WE ARE ALL JEWS.

Ashkenaz, Ethiopian, Sephardi, Mizrachi. A diverse tribe with different looks, languages, food and customs. But… WE ARE ALL JEWS.

Leslie Young

50,289 次观看 • 5 个月前

The ultimate test of any physics simulator is its ability to deliver real-world results. With MuJoCo Playground, we’ve combined the very best: MuJoCo’s rich and thriving ecosystem, massively parallel GPU-accelerated simulation, and real-world results across a diverse range of robot platforms: quadrupeds, humanoids, dexterous hands, and arms. Best of all? You can get started today with a single command: pip install playground

The ultimate test of any physics simulator is its ability to deliver real-world results. With MuJoCo Playground, we’ve combined the very best: MuJoCo’s rich and thriving ecosystem, massively parallel GPU-accelerated simulation, and real-world results across a diverse range of robot platforms: quadrupeds, humanoids, dexterous hands, and arms. Best of all? You can get started today with a single command: pip install playground

Kevin Zakka

153,505 次观看 • 1 年前

Tired of teleoperating your robots? We built a way to scale robot datasets without teleop, dynamic simulation, or even robot hardware. Just one smartphone scan + one human hand demo video → thousands of diverse robot trajectories. Trainable by diffusion policy and VLA models as-is. Introducing: Real2Render2Real 👉

Tired of teleoperating your robots? We built a way to scale robot datasets without teleop, dynamic simulation, or even robot hardware. Just one smartphone scan + one human hand demo video → thousands of diverse robot trajectories. Trainable by diffusion policy and VLA models as-is. Introducing: Real2Render2Real 👉

Max Fu

69,224 次观看 • 1 年前

China's Kepler Robotics shared a video of its Forerunner K2 humanoid with next-gen hands With an 11-DOF, each 'NimbleMaster' hand can handle a 15 kg payload It also has a 25-element tactile sensing matrix per finger to better grasp different objects

China's Kepler Robotics shared a video of its Forerunner K2 humanoid with next-gen hands With an 11-DOF, each 'NimbleMaster' hand can handle a 15 kg payload It also has a 25-element tactile sensing matrix per finger to better grasp different objects

Brett Adcock

16,365 次观看 • 1 年前

How can someone with LESS THAN 200 FOLLOWERS have such a DIVERSE musical output?

How can someone with LESS THAN 200 FOLLOWERS have such a DIVERSE musical output?

Ewan Bristow

67,090 次观看 • 4 个月前