So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we... show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)show more

Ok so SGFT does more than just provide a simulation initialization. Why does this matter? - Standard finetuning methods from simulation can suffer from catastrophic forgetting where performance collapses. SGFT makes consistent rapid progress during finetuning by using simulation to provide persistent guidance through training. SGFT is significantly more sample efficient and can solve difficult contact-rich tasks in the real world where prior methods can fail entirely, often within just minutes of real-world data collection! (2/6)

So what’s the key idea: while policies may not transfer directly from sim2real due to dynamics mismatch, value functions in simulation capture the approximate geometry of the problem that *does* transfer approximately from sim2real. The ordering of states defined by a sim-learned value function (V_sim) captures successful behaviors that are invariant between sim and real, even if the low-level dynamics differ somewhat. SGFT uses this insight to accelerate real-world finetuning by *using V_sim to perform potential-based reward shaping for real-world RL*. We show both theoretically and empirically that doing so effectively shortens the learning horizon, making learning far more efficient! (3/6)

SGFT is extremely simple, and easy to integrate with *any* RL-based finetuning paradigm - pick your favorite RL algorithm and add this on top. In particular, SGFT integrates nicely with Model-Based RL (MBRL). Because we shorten the learning horizon, we sidestep the core challenge facing MBRL — prediction errors that compound over long horizons. This enables us to use generative world models such as dreamer or TD-MPC to boost sample efficiency. (4/6)

We applied this on a number of contact-rich manipulation tasks in the real-world with a Franka robot. Here is a comparison of time to learn each task with our method vs existing baselines using sim2real transfer, RL finetuning, and/or model-based RL. SGFT tends to add value to *any* RL finetuning methods it’s used with, doing a lot better than finetuning methods that simply transfer policy initializations. In each case, our method outperforms baselines in sample efficiency by at least 2x, often learning in just minutes! Some tasks like hammering seem easy, but require getting dynamics precisely right. (5/6)

The key thing I took away from here is - simulation is inherently wrong, but can still be very useful! Value functions from simulation can make the job of real-world RL *much* easier, making it far more practical as a solution. This was work conceptualized and led by @patrickhyin and Tyler Westenbroek, along with a great set of collaborators - Simran Bagaria, Kevin Huang, @chinganc_rl, @Andrey__Kolobov between UW and MSR Website: Paper: We will be presenting this paper at #ICLR2025 this April 😃

Want to learn how practical AI skills and automations for your business and work? Check out our 50+ step-by-step video tutorials 100% FREE 20+ hours of Ai and Automation goodness absolutely free 🥳

Nice idea. Reminds me of "luckiness" in statistical learning theory. @bremen79

Curious to see how you tackled the cost issue in your recent work. Better priors are definitely needed for real-world RL!

Very very interesting; it might be worth checking this to see if it could also further help:

We can scale up test-time compute with verifiers or without verifiers. It's not obvious which way is better. Turns out we can do some theoretical analysis and show that the use of verifiers is more optimal, and this is supported by experiments!

Thrilled to introduce 🦏RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations! Project: RHINO is our recent attempt on human-robot interaction to bring humanoids into human's real life. To do so, we require highly responsive interaction abilities for humanoid robots. Instead of multi-stage and language-driven interaction, RHINO supports real-time interruption and automatically responds by recognizing human behaviors! In addition, we integrate a large skill set and make sure RHINO is safe for humans. It's note-worthy that besides object manipulation skills that are learned from teleoperation data, RHINO learns other interaction behaviors from only human-human interaction data, which is much more accessible and scalable! We open-source all training and real-robot deployment codes, along with the datasets for the community! As you may observe, the H1 robot currently only responds using upper-body joints. But I want to highlight that, this is a parallel project to HugWBC ( -- our recent progress on humanoid locomotion, which supports arbitrary upper-body interruption! After we extend the humanoid's ability from both aspects, making humanoids work as real human-like assistants in whole-body is not a distant dream anymore! This work is equally contributed by @ChenTimer, @lixinyao442219, and Jiahang Cao, and is co-worked by @zhu990729, Wentao Dong, me, Prof. Ying Wen, Prof. Yong Yu, Prof. Liqing Zhang, and Prof. Weinan Zhang.

My next set of lectures on foundational models for #robotics has been posted. These lectures cover learning models, using foundational models to compute rewards, and #scalingdatacollection for robotics.

crazy... G1 has made different labs compete and rush for highly similar topics