With an hour of training data at 5000 epochs, it's noticeably better (successfully picks up a cube most of the time), but still feels the call of the void.

Inspecting the data manually, the average spine delta is 55x larger in the original recording than it is in the rollouts. So that's awfully suspicious.

This was, in fact, busted. Fixed now!

Cool demo! Is this behavior cloning with pixel observations? Also, what's the goal if this project are you trying to create policy for a domestic robot?

Yup, it's a lightly modified ACT running on a mobile manipulator simulated with mujoco. Input is a single 640x480 image, plus joint angles. End goal is to get to a language-conditioned policy that can be guided by a VLM to do medium-horizon household tasks.

Is it nvidia Issac?

really cool! how are you teleoperating the robot?

It's unity with mujoco replacing the physics engine via DLL. So I can just hook up the quest 3 using the standard Unity integration, and then feed the positional values into an IK solver and then drive the joints from the solver.

-What is my purpose? You pick up cubes. -Oh my God. Who wouldn't feel like jumping off--