Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

My last PhD project at CMU will be presented today and tomorrow at Conference on Robot Learning as one of the Demos: SIAT (SIAT: Stretch control with Immersive AR Teleoperation) Find Vidhi Jain, Su Li and Yonatan Bisk who will be presenting this work in person. The original goal... show more

Hao Zhu

3,198 subscribers

12,458 просмотров • 1 год назад •via X (Twitter)

Здоровье и велнес Образование Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 5

Фото профиля Hao Zhu 朱昊

Hao Zhu 朱昊1 год назад

Despite the simplicity of the design the Stretch robot, it has a much different morphology than humans, e.g. the robots' arm rotates together with the base. Controlling which is unintuitive since manipulation also requires base movement. We created an intuitive interface -- SIAT.

Фото профиля Hao Zhu 朱昊

Hao Zhu 朱昊1 год назад

Fun fact on the technical side. I started this project 2 months before my thesis defense, but ROS was just too hard for me to set up and play with. Therefore, I decided to build a new system called AAct from scratch: The design of it worth another post.

Фото профиля Hao Zhu 朱昊

Hao Zhu 朱昊1 год назад

Check out our codebase at

Фото профиля Sabeer Saeed

Sabeer Saeed1 год назад

@corl_conf @viddivj @ybisk Super Congratulations!

Фото профиля Raghav Kapoor

Raghav Kapoor1 год назад

@corl_conf @viddivj @ybisk Congratulations and amazing work @_Hao_Zhu!!

Похожие видео

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 просмотров • 6 месяцев назад

The trailer for Project Ringfall is here! This rework is both a mod... and a journey I'd like you to join me on. My goal for Project Ringfall is to highlight what makes Halo Infinite unique and fun, while also making changes in areas often discussed here in the Halo community. This mod is for both Social and Competitive players! Along the way, I'll be talking about my own design philosophy, breaking down my approach to tuning weapons and player abilities, as well as sharing my design documents as I go! I appreciate every single one of you that have stuck with me as I've learned and grown as a designer. I really hope you'll enjoy this mod on it's release. I don't ever ask this, but liking and retweeting this post helps a lot. I'm putting my soul into this project, and if you like what I'm doing, the best way to support me is by sending a like and telling your friends.

The trailer for Project Ringfall is here! This rework is both a mod... and a journey I'd like you to join me on. My goal for Project Ringfall is to highlight what makes Halo Infinite unique and fun, while also making changes in areas often discussed here in the Halo community. This mod is for both Social and Competitive players! Along the way, I'll be talking about my own design philosophy, breaking down my approach to tuning weapons and player abilities, as well as sharing my design documents as I go! I appreciate every single one of you that have stuck with me as I've learned and grown as a designer. I really hope you'll enjoy this mod on it's release. I don't ever ask this, but liking and retweeting this post helps a lot. I'm putting my soul into this project, and if you like what I'm doing, the best way to support me is by sending a like and telling your friends.

ANCHr

59,382 просмотров • 2 лет назад

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,551 просмотров • 4 месяцев назад

Now, THIS is exciting. Save Windermere has partnered with Map Impact and the UK Space Agency in what we believe is the most innovative data collection project ever constructed to study the inputs into Windermere. The project will use satellite imagery to study chlorophyll-a concentrations in Windermere. Chlorophyll-a is synonymous with phosphorus loading. Phosphorus is the key nutrient that’s causing the decline of Windermere. This satellite imagery will be used in conjunction with data from one of the UK's largest mobile networks, which will provide anonymised cellular data to determine the number of people within the Windermere catchment at any given moment. This will enable us to quantify the pressure from human activity (i.e., sewage) in the area and its impact on phosphorus concentrations in the lake. This will then be compared with other inputs like land use to see how impactful sewage is in comparison to these. To find out more, please visit this link and sign up for the website to receive key updates from the project as it progresses.

Now, THIS is exciting. Save Windermere has partnered with Map Impact and the UK Space Agency in what we believe is the most innovative data collection project ever constructed to study the inputs into Windermere. The project will use satellite imagery to study chlorophyll-a concentrations in Windermere. Chlorophyll-a is synonymous with phosphorus loading. Phosphorus is the key nutrient that’s causing the decline of Windermere. This satellite imagery will be used in conjunction with data from one of the UK's largest mobile networks, which will provide anonymised cellular data to determine the number of people within the Windermere catchment at any given moment. This will enable us to quantify the pressure from human activity (i.e., sewage) in the area and its impact on phosphorus concentrations in the lake. This will then be compared with other inputs like land use to see how impactful sewage is in comparison to these. To find out more, please visit this link and sign up for the website to receive key updates from the project as it progresses.

MattStaniek

81,543 просмотров • 2 лет назад

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 просмотров • 1 год назад

A high-quality haptic control for robots as a weekend project? All for under $600. [📍 Save Github & arXiv for later] A team built DOGlove, an open-source haptic glove for robot teleoperation that you can assemble yourself for under $600. It is built for real control, not demos. • Tracks 21 degrees of freedom with a custom joint design • Provides cable-driven force feedback plus fingertip haptics • Can be used to collect data for imitation learning Affordable haptics could make teleoperation more reliable and improve the quality of data we use to train physical AI. Project link: Credit: DOGlove team, shared by Han Zhang —- Weekly robotics and AI insights. Subscribe free:

A high-quality haptic control for robots as a weekend project? All for under $600. [📍 Save Github & arXiv for later] A team built DOGlove, an open-source haptic glove for robot teleoperation that you can assemble yourself for under $600. It is built for real control, not demos. • Tracks 21 degrees of freedom with a custom joint design • Provides cable-driven force feedback plus fingertip haptics • Can be used to collect data for imitation learning Affordable haptics could make teleoperation more reliable and improve the quality of data we use to train physical AI. Project link: Credit: DOGlove team, shared by Han Zhang —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

18,290 просмотров • 6 месяцев назад

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 просмотров • 1 год назад

I’m giving an agent control over Reachy Mini from Hugging Face and letting it understand and share spatial data via SPECS for Devs AR is the human interface for robotics and physical AI imo. It feels like absolute magic to interact with this, both in voice/agent and “puppeteering” mode. I’ll probably work on AR for either an arm (manipulation tasks) or some sort of drone (locomotion in 3D space) next… Project is fully open source btw: Thank you SensAI Hackademy for sending me the robot!

I’m giving an agent control over Reachy Mini from Hugging Face and letting it understand and share spatial data via SPECS for Devs AR is the human interface for robotics and physical AI imo. It feels like absolute magic to interact with this, both in voice/agent and “puppeteering” mode. I’ll probably work on AR for either an arm (manipulation tasks) or some sort of drone (locomotion in 3D space) next… Project is fully open source btw: Thank you SensAI Hackademy for sending me the robot!

Johannes Tscharn

35,341 просмотров • 5 месяцев назад

Message from #LAKA ~ You Carry The Scar Of The One You Wounded "You May Say That You Want Peace On Earth. But Do You Realize That Peace Must Begin With You? Stop Wounding Yourself. And Do Your Best To Never Harm Another. What You Project To Another Will Remain With You. This Is An Energy That You Do Not Want. It Will Grow And So Will Your Dissatisfaction." "Be Aware That Social Media Is Designed For This Interaction And Conflict. Choose Wisely As Your Well Being Is Much More Important Than Winning An Argument."

Message from #LAKA ~ You Carry The Scar Of The One You Wounded "You May Say That You Want Peace On Earth. But Do You Realize That Peace Must Begin With You? Stop Wounding Yourself. And Do Your Best To Never Harm Another. What You Project To Another Will Remain With You. This Is An Energy That You Do Not Want. It Will Grow And So Will Your Dissatisfaction." "Be Aware That Social Media Is Designed For This Interaction And Conflict. Choose Wisely As Your Well Being Is Much More Important Than Winning An Argument."

Kab

73,592 просмотров • 1 год назад

Open-source robot arm meets hand tracking [📍GitHub below] It is designed with an industrial mindset but built as a 3D-printed desktop system. PAROL6 paired with a LEAP Motion controller is a nice example of how accessible robot teleoperation has become. • Hand motion is streamed to the robot at 100 Hz via UDP • A pneumatic gripper is controlled by simple fist open and close gestures • The entire robot stack is open source, from mechanics to control software Combine that with low-latency hand tracking and you get a very practical platform for learning manipulation, teleoperation, and human-robot interfaces. This kind of setup is great for experimentation, teleop, data collection, and teaching robots by demonstration All without proprietary hardware or locked software. Credit to SourceRobotics 📍Code: —— Weekly robotics and AI insights. Subscribe free:

Open-source robot arm meets hand tracking [📍GitHub below] It is designed with an industrial mindset but built as a 3D-printed desktop system. PAROL6 paired with a LEAP Motion controller is a nice example of how accessible robot teleoperation has become. • Hand motion is streamed to the robot at 100 Hz via UDP • A pneumatic gripper is controlled by simple fist open and close gestures • The entire robot stack is open source, from mechanics to control software Combine that with low-latency hand tracking and you get a very practical platform for learning manipulation, teleoperation, and human-robot interfaces. This kind of setup is great for experimentation, teleop, data collection, and teaching robots by demonstration All without proprietary hardware or locked software. Credit to SourceRobotics 📍Code: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

32,025 просмотров • 6 месяцев назад

This will unequivocally be the biggest curated event in the lifespan of my project and I’m beyond excited to share it with you EPROM PRESENTS — SYNTHEISM ROBOTICS — JULY 29TH — DENVER — MISSION BALLROOM Enter requested data at for pre-sale registration

This will unequivocally be the biggest curated event in the lifespan of my project and I’m beyond excited to share it with you EPROM PRESENTS — SYNTHEISM ROBOTICS — JULY 29TH — DENVER — MISSION BALLROOM Enter requested data at for pre-sale registration

EPROM

79,289 просмотров • 3 лет назад

Teleoperating a robot is hard. This means that when performing a robot task via teleoperation — say, to collect examples for training a robot policy — it’s almost unavoidably slower than you would like, below either the capabilities of the human expert on their own or the robot performing the task. Wouldn’t it be great if there was a way to fix this? Unfortunately, it’s harder than it looks. You can’t just execute faster, as this alters the distribution of environment states the policy will encounter. Nadun Ranawaka Arachchige and Zhenyang Chen propose Speed-Adaptive Imitation Learning (SAIL), which adds error-adaptive guidance, adapts execution speed according to task structure, predicts controller-invariant action targets to ensure robustness across execution speeds, and explicitly models delays from, for example, sensor latency. Watch episode #59 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

Teleoperating a robot is hard. This means that when performing a robot task via teleoperation — say, to collect examples for training a robot policy — it’s almost unavoidably slower than you would like, below either the capabilities of the human expert on their own or the robot performing the task. Wouldn’t it be great if there was a way to fix this? Unfortunately, it’s harder than it looks. You can’t just execute faster, as this alters the distribution of environment states the policy will encounter. Nadun Ranawaka Arachchige and Zhenyang Chen propose Speed-Adaptive Imitation Learning (SAIL), which adds error-adaptive guidance, adapts execution speed according to task structure, predicts controller-invariant action targets to ensure robustness across execution speeds, and explicitly models delays from, for example, sensor latency. Watch episode #59 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

27,970 просмотров • 6 месяцев назад

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the motion of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,514 просмотров • 2 лет назад

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 просмотров • 2 лет назад

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Jason Ma

98,090 просмотров • 1 год назад

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

Egocentric Videos for Robot Training 🧵 Researchers at NYU and UC Berkeley published research where they developed a system called EgoZero that trains robots using human demonstration videos recorded with smart glasses. 👓 The system converts first-person human actions into 3D point-based state-action representations. The policies executed on a gripper-equipped robot, achieved a 70% zero-shot success rate across seven manipulation tasks, with only 20 minutes of human data per task. EgoZero stands as one of the solid empirical proofs that egocentric smart-glass video collected from everyday human behavior can serve as powerful, scalable training data for real robot learning. Vincent Liu Ademi Adeniji

VaderResearch

23,055 просмотров • 11 месяцев назад

No wonder they say teleoperation is dead Peking University’s DAGroup released HumanNet, a massive 1 million hour dataset of human-centric videos that turns everyday internet footage into gold for embodied AI. It’s got first-person and third-person views, super detailed actions, object interactions, tool use, and long sequences of real behavior. Everything’s cleaned up and annotated with 3D poses, SLAM tracks, and robot-ready labels. The crazy part is,Training on just 1,000 hours of their first-person videos performs as well as (or even slightly better than) 100 hours of actual robot teleop data on downstream tasks. Basically, human videos can now stand in for expensive robot data. Scaling laws for robotics just got a whole lot more realistic. Human priors + smart curation = the future of world models.

No wonder they say teleoperation is dead Peking University’s DAGroup released HumanNet, a massive 1 million hour dataset of human-centric videos that turns everyday internet footage into gold for embodied AI. It’s got first-person and third-person views, super detailed actions, object interactions, tool use, and long sequences of real behavior. Everything’s cleaned up and annotated with 3D poses, SLAM tracks, and robot-ready labels. The crazy part is,Training on just 1,000 hours of their first-person videos performs as well as (or even slightly better than) 100 hours of actual robot teleop data on downstream tasks. Basically, human videos can now stand in for expensive robot data. Scaling laws for robotics just got a whole lot more realistic. Human priors + smart curation = the future of world models.

CyberRobo

27,048 просмотров • 2 месяцев назад

NAVIAI-I3, a more robust humanoid robot, was launched by the Zhejiang Humanoid Robotics Innovation Center. Standing 1.8 meters tall, it can run like a human and walk on various terrains. This humanoid robot appears to be primarily designed for specialized industrial tasks, such as security patrols in underground parking lots. They previously launched the lightweight bipedal NVAI-2 and a wheeled humanoid robot. The bipedal robot is used in manufacturing operations, while the wheeled humanoid robot is used for unstructured, flexible tasks such as warehouse sorting, supermarket shelving, and office cleaning. We filmed this demonstration at the World Robot Conference; you can see it in the comments section.

NAVIAI-I3, a more robust humanoid robot, was launched by the Zhejiang Humanoid Robotics Innovation Center. Standing 1.8 meters tall, it can run like a human and walk on various terrains. This humanoid robot appears to be primarily designed for specialized industrial tasks, such as security patrols in underground parking lots. They previously launched the lightweight bipedal NVAI-2 and a wheeled humanoid robot. The bipedal robot is used in manufacturing operations, while the wheeled humanoid robot is used for unstructured, flexible tasks such as warehouse sorting, supermarket shelving, and office cleaning. We filmed this demonstration at the World Robot Conference; you can see it in the comments section.

CyberRobo

15,521 просмотров • 9 месяцев назад

Watch the process of this painting by Devin for LightBox Expo! This year you can find Devin Elle Kurtz at LightBox Expo in the artist alley. She will be selling this print and lots more as well as doing a presentation on storytelling and illustration!

Watch the process of this painting by Devin for LightBox Expo! This year you can find Devin Elle Kurtz at LightBox Expo in the artist alley. She will be selling this print and lots more as well as doing a presentation on storytelling and illustration!

LightBox Expo

26,124 просмотров • 2 лет назад

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,813 просмотров • 7 месяцев назад