Using large-scale GPU parallelized reinforcement learning in Isaac Sim,... we are able to obtain a robust whole-body controller for Taku optimized for precise loco-manipulation.show more

We open-sourced all components of the HTD whole-body controller... (WBC): -Play our WBC on your browser: -Main HTD codebase entry: (data collection and HTD policy soon!) Highlighted features: -Lightweight decoupled WBC that can be trained on a single GPU -A stable control base for humanoid loco-manipulation (in a mobile-manipulation style) -Support a wide range of body poses including extreme ones -Isaac Sim/Lab training, zero-shot deploymentshow more

🤖 How can we scale up humanoid robot learning?... Introducing 🌟VLK🌟: generating large-scale synthetic data with paired egocentric observations, text, and full-body G1 kinematics for learning humanoid loco-manipulation. No teleoperation needed! Website:show more

This is a demonstration video of in-hand object rotation... with new Torobo Hand. By leveraging large-scale parallel reinforcement learning in NVIDIA Isaac Sim, the learned policy can perform object manipulation in MuJoCo and on the real robot without any additional training.show more

TWIST: a real-time teleoperation system for humanoid robots to... mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.show more

See Spot perform dynamic whole-body manipulation. Using a combination... of reinforcement learning (RL) and sampling-based control, the robot is able to autonomously drag, roll, and stack tires weighing 15 kg (33 lb), well above its maximum arm lift capacity. Learn more about coordinating locomotion and manipulation processes:show more

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We... are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵show more

Introduce OmniH2O, a learning-based system for whole-body humanoid teleop... and autonomy: 🦾Robust loco-mani policy 🦸Universal teleop interface: VR, verbal, RGB 🧠Autonomy via ShitposterAI or imitation 🔗Release the first whole-body humanoid datasetshow more

A Systematically Optimized Miniaturized Mesoscope (SOMM) for large-scale calcium... imaging in freely moving miceshow more

Looking for a fast method to render your large-scale... game components? Our lastest High-Perf Graphics paper allows to render terrains, oceans and even entire planets in realtime on any modern GPU using compute shaders. Fun fact: we use a Concurrent Binary Tree as a GPU memory manager.show more

Check out our #ICRA2025 paper ! Robust, precise, and... terrain-aware—our RL controller significantly improves over baselines for whole-body 6-DoF tracking.💥 Project website: Work led by: Tifanny Portela, Andrei Cramariuc, Mayank Mittal and Marco Huttershow more

NVIDIA's Jim Fan has verified from reliable sources that... the video by EngineAI is a real robot demo and not CGI. The controller is a neural net trained in the NVIDIA Isaac simulator using reinforcement learning. There’s also another video from a different angle.show more

Training a Whole-Body Control Foundation Model -- new work... from my team at Agility A neural network for controlling our humanoid robots which is robust to disturbances, can handle heavy objects, and is a powerful platform for learning new whole-body skills learn more in our blog post ->show more

Figure 02 robots are now walking with a smoother... gait and increased speed, thanks to sim-to-real reinforcement learning.show more

A big part of scaling robot learning to solve... real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)show more

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation... in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵show more

AI Sapiens is ROBOTIS’ upcoming open-source humanoid platform for... whole-body motion, imitation learning, reinforcement learning, and Sim2Real deployment on real hardware. Dance demonstration by SNU’s SHAPE team.show more

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

This is a neural network flying a drone at... extremely high speed, beating human champions in FPV drone racing. - Reinforcement learning as a tool is so marvelously versatile. It's able to solve both fast, reactive tasks and slow, deliberate tasks (ChatGPT RLHF). - Trained in large-scale simulation, finetuned in real world - I believe this is the paradigm that will get us to generalist robot some day. Published on Nature's cover, "Champion-Level Drone Racing using Deep Reinforcement Learning." Authors: Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun & Davide Scaramuzzashow more

We talked to Ritvik Singh about how you can... train sim-to-real dexterous manipulation policies using NVIDIA Isaac. This robot is grasping objects using pure RGB stereo: take in images from a camera pair and predict what to do, all without training in the real world.show more