Lighting differences can make a huge difference in robotics. Today, I found a quirk in my model exemplifying this. > I collected 10h of training data. > 3h in, I notice that the left arm following the right arm for the final movement could be good for the final... insertion subtask. > I change behaviour. > But the first 3h didn't include much nighttime data (=dark room). 🎯Result The same policy reliably exhibits different behaviour on exactly this subtask depending on the lighting. So many microlearnings to be made in robotics.show more

Yep. I had to re-train one of my ACT models last night cuz when I recorded the original data I had a light placed in a weird position which gave me all sorts of reliability issues. Haha I hope the new model works cuz the stream is going live tomorrow 😅

Cool!

I've actually still got a few hours left til the training wraps up and I can test again 🤞

we should have better vision encoders for this, like having a loss func that doesn’t penalize difference in lighting condition in reconstruction so latents for 2 images with diff lighting is the same

perhaps should encase in a self-lighted box like some other startups do

Parts are already on their way!

There are so many little details to learn, but I am starting to feel confident in a few things and one is that you really should turn on image augmentations every time you train something. Specifically, this means the slight brightness color and contrast jitter settings. And without you having to do any extra work in data collection, this helps with things like lighting sensitivity.

Is that VLA based policy?

Yes, π0.5 finetune on 10h of data

Now imagine this in real production: every company, every site, different lighting, different conditions. The operator is definitely not going to lower the blinds just to make it work. Do you already have a solution for this?

Yeah, collect consistent data the next time 😂

Fair 😂 if it were only that simple, robotics would be half as fun....

haha so cool

Can't this be fixed by creating new training data based on original but with exposure changed? At least partly

Yes totally. But i need to collect that data now haha

I mean by creating variations of your training data programmatically with changed exposure, brightness, color etc? :)

I never train a real robot but I don’t want to spend any time optimising these micro learnings. ideally models should generalise to different lighting conditions. > I change behaviour can u explain your 3rd point. I didn’t get it. How you changed the behaviour?

I collected data in a different way. I originally did it the way you see in the first trial and then changed to the behaviour of the robot when its dark. If you don't want to spend time optimising these things I'd just wait another two years before getting into robotics haha

Got it. Makes sense to me now. haha I will surely enjoy it but we’ll see once I get my hands on some hardware. rn i m playing sim-sim only

Check it out - - they focused exactly on the problem of maintaining robustness under sensor noise, and every lab has implemented some notion of that

Cool thanks!

Not a problem! I can't wait to shake your robotic hand at the local manufacturer

JEPA is meant to solve this.

This is a great robotics example of a classic ML issue. It reminds us of the CNN that learned to classify wolves by detecting snow in the background instead of the animal itself. With black box models, dataset diversity is key to ensuring the policy learns the task, not accidental shortcuts.