Astra solved this hand-object reconstruction and tracking problem in one shot😂 It took us Jikai Wang Bowen Wen nearly two years to build a capture system and a pipeline to obtain accurate hand-object poses from multi-view videos. Some of our results:

A clarification on this: “one-shot” wasn’t the right word. See @SucarEdgar’s original post for details. Astra calls SAM3D, HaMeR, OpenCV, etc. to build the pipeline. The attached video is from our hand-crafted pipeline, not Astra.

@JwRobotics @bowenwen_me So astra found this piece of publication and implemented from scratch?

@JwRobotics Amazing!

@JwRobotics @bowenwen_me Two years of your pipeline is probably why Astra could do it in one shot. Someone had to prove the problem was tractable first.

@JwRobotics @bowenwen_me Hi Yu, how in the world is Astra able to one shot this? Like, what’s the process behind this? Does Astra use some intermediate tool to locate the joints frame by frame? If so, how is it important where all it does is to mimic given a clip? Can it generalize?

@JwRobotics @bowenwen_me Check details in the quoted tweet. I believe they have tried several tuning, but eventually make it work given a video input.

@JwRobotics @bowenwen_me Okay but this is kinda worthless, no? If this doesn’t generalize, I believe this is just an AI slop showboating

@JwRobotics @bowenwen_me Astra is calling different vision models such as SAM3D, HaMeR to get object pose and hand pose. I don’t see why it does not generalize

@JwRobotics @bowenwen_me What’s the cost of this “clip to mujoco” process btw?

@JwRobotics @bowenwen_me Once you have object-hand pose and shape, it is straightforward to load these objects into mujoco

@JwRobotics @bowenwen_me Thanks Mr. Xiang 🙏🏻

@JwRobotics @bowenwen_me very cool. thanks for sharing. what does one shot mean for Astra?

@JwRobotics @bowenwen_me Check details in the quoted tweet. I believe they have tried several tuning, but eventually make it work given a video input.

@JwRobotics @bowenwen_me Just out of curiosity. How does training go for this, is it virtual first, then physical and compare the difference, then retrain

@JwRobotics @bowenwen_me There is no training our pipeline. We used trained models for different computer vision tasks such as object segmentation, object pose estimation, hand pose estimation and multi-view geometry to solve this problem

So Astra generates this for the sim with the robot u give sit? Are these models on hugging face or proprietary? I just got open arm 2.0 robot and I’m building a generative cad and platform while teaching myself how to build this. I’m looking at dimensional for recording loops, then I was gonna let my ai create sims. Not sure with Astra if this is even the right approach