Going back to training single task models with a focus on improving success rates - here’s a longer, unedited real-time rollout of an ACT model trained with 40 episodes. Explanation in 🧵

The robot freezing initially was due to a false positive from my safety system. If you can see a regular pattern of fast movements, this is when the action chunk border chances. I used longer action chunks which makes the robot less responsive but more “confident” in movements

My previous data was recorded in 50Hz, while data for this model (and the inference) was in 20Hz. I also controlled the robot in a slower and more careful manner. In AI, garbage in -> garbage out so data quality really matters

Why 20Hz and not 50Hz? I noticed my cheap cameras were lagging, especially the wrist camera during movements, so many frames would contain an outdated view of the scene. This did not eliminate the issues and buying better cameras is high on the priority list but 20Hz helps

The wrist camera’s shakiness is a likely reason why the robot can pick capsules in front of the main camera but struggles with one’s further away, or ones that get occluded by the arm. Perhaps masking one of the cameras (or patches of them) at times during training would help?

The data I collected is all from the same table and from 2 different coffee cups - I’ll probably add to the dataset by recording a few episodes from other places in my flat and a few other cups as well. Data diversity matters. A lot.

I also want to augment my data from human corrections during inference, dAgger style. This will help the robot align itself back on track, as the original training episodes don’t have many mistakes.

29 Years of proven track record and creating generational wealth! If you want to learn what it takes to becomes super successful then take the success test now!

Question: Why ACT why not Diffusion Policy? I had tried ACT with around 100 samples and lot of epochs (I mean a lot) but it still failed to perform good.

I might try diffusion (and VQ-BeT) as well to compare. But ACT is a good baseline and the model I have the most experience using. These experiments exist to see how different techniques can boost success rates, so you could start with any baseline model that's quick to train

So strange to watch

one day it will look indistinguishable from a human operator controlling the robot robot turing test 🤖