Загрузка видео...

Не удалось загрузить видео

На главную

Can robots leverage their entire body to sense and interact with their environment, rather than just relying on a centralized camera and end-effector? Introducing RoboPanoptes, a robot system that achieves whole-body dexterity through whole-body vision.

XiaomengXu11's profile picture

Xiaomeng Xu✈️IROS

1,267 subscribers

77,202 просмотров • 1 год назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 15

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

RoboPanoptes enables novel manipulation capabilities such as unboxing in narrow spaces, sweeping multiple or oversized objects, and stowing in cluttered environments, outperforming baselines in adaptability and efficiency.

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

Whole-body dexterity allows the robot to utilize its entire body surface for manipulation, e.g., leveraging multiple contact points or navigating constrained spaces. Whole-body vision provides comprehensive observations of its environment and visual feedback of its own state.

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

Whole-body dexterity and vision is enabled by 1) A modular hardware design that integrates distributed vision and actuation in a scalable and reconfigurable system.

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

2) An intuitive demonstration interface that streamlines the acquisition of complex whole-body manipulation behaviors.

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

3) A whole-body visuomotor policy that learns complex manipulation skills directly from human demonstrations, efficiently aggregating information from the camera system while maintaining resilience to sensor failures.

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

I had a great time working with Dominik Bauer and @SongShuran on this project! Check out our paper: video: and website: for more details!

Фото профиля Chongkai Gao
Chongkai Gao1 год назад

Super cool work!

Фото профиля Miguel Antonio Zapata Gonzales-Linares de la Peña
Miguel Antonio Zapata Gonzales-Linares de la Peña1 год назад

Great work Xiaomeng. Great researchers from Stanford often come from Tsinghua University. Thank you for making it open source.

Фото профиля The AI Veteran
The AI Veteran1 год назад

Great thread!

Фото профиля underscore advait patel
underscore advait patel1 год назад

super cool! how complicated/expensive is this to build?

Фото профиля Xiaomeng Xu✈️IROS
Xiaomeng Xu✈️IROS1 год назад

@adv8p The system is low-cost and built with off-the-shelf and 3D printed components! Each camera only costs $6. We will open source all the hardware design and code soon!

Фото профиля Maxime Alvarez
Maxime Alvarez1 год назад

With such coverage of the field of view you must have quite a lot of overlapping which could be very useful for 3D reconstruction of the environment.

Фото профиля ζ Pedram ζ
ζ Pedram ζ1 год назад

sweet

Фото профиля pfung
pfung1 год назад

@XiaomengXu11 brilliant! I didn't see this in the github page, but how does multi-camera perform on simple grasp task where overhead also sees the object clearly? i assume multi-camera would be slightly better but curious.

Фото профиля Ho Lab
Ho Lab1 год назад

Great work! We also addressed whole-body tactile sensing here: Do you think your policy can be applied to our robot for contact-based control?

Похожие видео

🤖Introducing TWIST: Teleoperated Whole-Body Imitation System. We develop a humanoid teleoperation system to enable coordinated, versatile, whole-body movements, using a single neural network. This is our first step toward general-purpose robots. 🌐
1:30

🤖Introducing TWIST: Teleoperated Whole-Body Imitation System. We develop a humanoid teleoperation system to enable coordinated, versatile, whole-body movements, using a single neural network. This is our first step toward general-purpose robots. 🌐

Yanjie Ze

80,217 просмотров • 1 год назад

Atlas hauling a 50 lb mini-fridge - Practiced the maneuver for millions of hours in a virtual environment. - Focused the training policy on full-body engagement rather than just hand-grasping, allowing the robot to leverage its entire frame for the lift.
0:36

Atlas hauling a 50 lb mini-fridge - Practiced the maneuver for millions of hours in a virtual environment. - Focused the training policy on full-body engagement rather than just hand-grasping, allowing the robot to leverage its entire frame for the lift.

The Humanoid Hub

112,053 просмотров • 4 месяцев назад

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.
0:30

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.

Space and Technology

31,172 просмотров • 11 дней назад

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵
2:37

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵

Xiaolong Wang

309,342 просмотров • 2 лет назад

Researchers have developed Humanoid-GPT, a new AI system that helps humanoid robots perform complex movements and tasks. Tested on the Unitree G1 robot, it enables real-time whole-body control and can perform new actions without special training. During demonstrations, the robot successfully carried out a variety of movements and tasks, showing strong potential for real-world applications.
0:28

Researchers have developed Humanoid-GPT, a new AI system that helps humanoid robots perform complex movements and tasks. Tested on the Unitree G1 robot, it enables real-time whole-body control and can perform new actions without special training. During demonstrations, the robot successfully carried out a variety of movements and tasks, showing strong potential for real-world applications.

Space and Technology

22,330 просмотров • 3 месяцев назад

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.
1:11

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

The Humanoid Hub

25,125 просмотров • 10 месяцев назад

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗
0:24

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗

Xiaomeng Xu✈️IROS

18,676 просмотров • 3 месяцев назад

Today, we’re introducing KinetIQ, our own AI framework for end-to-end orchestration of humanoid robot fleets. One system, multiple robot embodiments. Industrial, service and home environments coordinated in real time. The framework consists of 4 cognitive layers: from high-level task allocation and workflow optimisation down to VLA-based task execution and RL-trained whole-body control. Watch how KinetIQ runs both our wheeled and bipedal robots. Read more on our blog:
3:47

Today, we’re introducing KinetIQ, our own AI framework for end-to-end orchestration of humanoid robot fleets. One system, multiple robot embodiments. Industrial, service and home environments coordinated in real time. The framework consists of 4 cognitive layers: from high-level task allocation and workflow optimisation down to VLA-based task execution and RL-trained whole-body control. Watch how KinetIQ runs both our wheeled and bipedal robots. Read more on our blog:

Humanoid

23,975 просмотров • 7 месяцев назад

“Several photos showed the young girl’s dismembered body, including arms and legs in a trash bag, a head and torso in a body bag, and severed hands with missing fingers” He made a whole music video killing someone and getting rid of their body. he’s very sick omg
0:29

“Several photos showed the young girl’s dismembered body, including arms and legs in a trash bag, a head and torso in a body bag, and severed hands with missing fingers” He made a whole music video killing someone and getting rid of their body. he’s very sick omg

Pazienza✨#StopGalamseyNow

33,485 просмотров • 2 месяцев назад

“A huge robot army is loading..” Figure's Helix 02 demo shows end to end "pixels to whole body" autonomy, loading and unloading a dishwasher in a long 4 minute cycle without human intervention !!
0:33

“A huge robot army is loading..” Figure's Helix 02 demo shows end to end "pixels to whole body" autonomy, loading and unloading a dishwasher in a long 4 minute cycle without human intervention !!

SciTech Era

14,195 просмотров • 7 месяцев назад

Training a Whole-Body Control Foundation Model -- new work from my team at Agility A neural network for controlling our humanoid robots which is robust to disturbances, can handle heavy objects, and is a powerful platform for learning new whole-body skills learn more in our blog post ->
1:56

Training a Whole-Body Control Foundation Model -- new work from my team at Agility A neural network for controlling our humanoid robots which is robust to disturbances, can handle heavy objects, and is a powerful platform for learning new whole-body skills learn more in our blog post ->

Chris Paxton

46,193 просмотров • 1 год назад

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.
0:58

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 просмотров • 1 год назад

'I think putting kids on a diet isn't a great thing for their body image' Body positivity influencer, George Keywood, says that putting kids on diets would be 'bad for their body image' and their 'mental health'.
1:49

'I think putting kids on a diet isn't a great thing for their body image' Body positivity influencer, George Keywood, says that putting kids on diets would be 'bad for their body image' and their 'mental health'.

GB News

29,286 просмотров • 2 лет назад

Everyone asks if Atlas can bring them a drink, but this robot can bring you the whole fridge. Using AI-driven behaviors, Atlas is doing hard work and coordinating its whole body to manage heavy objects, balancing complex contact points with accuracy and reliability.
0:38

Everyone asks if Atlas can bring them a drink, but this robot can bring you the whole fridge. Using AI-driven behaviors, Atlas is doing hard work and coordinating its whole body to manage heavy objects, balancing complex contact points with accuracy and reliability.

Boston Dynamics

1,418,184 просмотров • 4 месяцев назад

X-Humanoid unveiled TG-VLA, which it calls the world's first whole-body VLA framework for full-size humanoid robots. The goal: move humanoids past "mobile dual-arm systems" to human-like, whole-body coordinated operation. Built on three technologies: - HEX: an open-source full-size humanoid VLA that learns across embodiments and coordinates the whole body. Claimed SOTA on real-world whole-body manipulation - HAF-VLA: hierarchical action flow that decomposes whole-body motion into subtasks. Best or joint-best on 6 benchmarks - DSRL-DCT: online RL for high-DoF humanoids in a compressed latent space. 100% success on mobile manipulation HEX is open-source.
2:48

X-Humanoid unveiled TG-VLA, which it calls the world's first whole-body VLA framework for full-size humanoid robots. The goal: move humanoids past "mobile dual-arm systems" to human-like, whole-body coordinated operation. Built on three technologies: - HEX: an open-source full-size humanoid VLA that learns across embodiments and coordinates the whole body. Claimed SOTA on real-world whole-body manipulation - HAF-VLA: hierarchical action flow that decomposes whole-body motion into subtasks. Best or joint-best on 6 benchmarks - DSRL-DCT: online RL for high-DoF humanoids in a compressed latent space. 100% success on mobile manipulation HEX is open-source.

The Humanoid Hub

19,563 просмотров • 2 месяцев назад

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."
3:25

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

Sawyer Merritt

624,910 просмотров • 8 месяцев назад

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇
3:06

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

Jianfei Yang

18,940 просмотров • 1 месяц назад