We always talk about models in robot learning, yet data quality is just as important (if not more!). How do we collect good data? It turns out RL can help you *during* data collection! If you're collecting demos, ✨Data Inoculation✨ is a simple addition to collect better data!

Good data should have high coverage: it should maximize the chance of generalizing to states visited at test time. Yet in practice, data collection is largely heuristic and ad hoc, and rarely offers good coverage. Is there a more principled way to collect high-coverage data?

Without good coverage, small errors from the learned policy compound and it can quickly go into OOD states that it cannot recover from. In a sense, lack of coverage "poisons" downstream policy performance. If only there was a "vaccine" that can help us "inoculate" our data 🤔

Enters Data Inoculation! The key idea: human and a learned residual policy collect data together in closed-loop control. The residual policy perturbs human demos, and human offers corrections constantly.

The residual policy is trained on the growing dataset with RL, so it mimics mistakes policies trained on the data make; the human's continuous correction provides recovery demonstrations. This provides higher coverage and “inoculates” the dataset against the learner's mistakes.

Data collected with Data Inoculation (DI) results in improved downstream policy success rates when trained with either BC or offline RL! Tested on three tasks that really demands *precision* (Fun fact: t-cube stacking seems simple, but is really hard to do even by human hand!)

We had two other interesting observations during our experiments: (1) DI elicits corrections on the part of the task that matters the most (e.g. the place/release part in the t-cube stacking task below)

(2) Early on during data collection, DI perturbs human commands and elicits corrections. As the residual RL policy is trained, it eventually helps the human collect more expert data! (I'm terrible at teleop, and I can clearly feel the residual policy helping me towards the end)

Much more details in the paper and videos on the website: paper: website: With the amazing lead @qiyang_li, @CharlesXu0124, and @svlevine

@Em_Nomadic Hi, where can I write a check to your pre-seed? You have been consistently dropping bangers 👑

@Em_Nomadic Haha thanks!

@vai_viswanathan @Em_Nomadic Seriously though

@vai_viswanathan @Em_Nomadic hahaha appreciate it you guys are too kind

Neat work! A few Qs 1. Are there teleoperation setups where this wouldn’t work? Eg, if a human directly controls the robot’s position? 2. Could well-tuned fixed random noise added to human actions work? Or annealed noise? 3. How does this compare to BC + RL fine-tuning?

Thanks! 1. For some setups you can _feel_ the perturbations and others you can't (i.e. you only _see_ them, our setup is in this category ), but they should all work with DI.

2. We tried fixed random noise and it didn't work super well. The reason is that they perturb in random / non meaningful directions, and a learned perturbation that makes mistakes the learned policy makes is much more meaningful

3. We study only the data collection process, which is kind of orthogonal to the learning method / algorithm (i.e. you can take the same data and do BC + RL finetuning)

Thanks for the responses, Paul! Re 2), I didn’t see any mention of tuning for the random baseline. I’m guessing you all tried but couldn’t get other hyper parameters to work? Re 3), I was thinking of online data collection from the human expert + online data collection from the (learning) policy. While this may suggest a particular algorithm, you could also BC or RL this dataset.

Nice work on this one guys! Pretty sure you were in the middle of testing the T-stack when I came to visit earlier this summer 😅 . My bad...didn't mean to be an inoculation blocker

Haha thanks Rosario! It was indeed the t-block stacking task. You must have witnessed how easy it is for the block to fall then 😂

the place/release result is really interesting! more demos isn't the same as more recovery coverage. would be interesting to compare collections with something like a fixed operator-time budget

Thank you for sharing good work. I also definitely agree that we need to develop a systematic data collection pipeline not only RL but also SFT. 😊

love the inoculation framing. does the residual policy ever push the human into states that are hard but unrealistic? curious how you keep coverage up without collecting junk

Does this carry over when there’s no robot in the loop, like someone just doing the task with their own hands?

How much does data quality improve?

Using RL during collection rather than after is a smart inversion. If the policy flags the states it's weakest in, operators spend their hours there instead of re-recording easy demos. How much does it cut total demos for the same success rate?

data quality > model hype, fr. love that the human and residual policy co-collect instead of patching coverage after. does timing jitter in the human signal mess with it? we just shipped a stereo head cam with <1 µs sync + 600 Hz IMU for exactly this reason

Using RL during collection to steer what demos you gather makes a lot of sense. Does the gain hold when the operator is less skilled, or is most of the value in fixing the bad-operator tail?

Data still lands mid collection

good data matters more than good models. i have 3 demo logs. 2 are mine and 1 is a mistake i keep. i collected them badly on purpose.

Having the residual push the human off course on purpose is a neat way to get recovery data. Did the corrections land on the place/release step from the start, or only once the residual got good at copying the policy's mistakes?

The biggest data-quality win we’ve seen in teleop is keeping the failure→recovery episodes instead of throwing them out. Does Data Inoculation pick where to perturb from the policy’s uncertainty, or is it randomized per episode?