Can we learn a 3D world model that predicts object dynamics directly from videos?
@kaiwynd3631 位关注者
46,666 次观看
10 条评论
评论来自 X (Twitter) 上的原帖

@kaiwynd
Modeling ropes, cloth, bags, etc. is hard because of their complex physics and partial observability. Classical simulators struggle to construct exact digital twins from real observations. We overcome these challenges by learning neural dynamics directly from videos.
4 个赞

@kaiwynd
Our particle-based neural dynamics model represents objects as dense 3D particles and predicts their next-step velocities to simulate object dynamics. It features three stages: particle encoding, grid-velocity editing, and grid-to-particle velocity transfer.
3 个赞

@kaiwynd
Trained with videos including robot–object interactions under self-supervion, PGND can model diverse deformable objects—including ropes, cloth, stuffed animals, and paper bags—using <20 minutes of data per object.
3 个赞

@kaiwynd
PGND becomes a 3D action-conditioned video generator when 3D Gaussian Splatting is plugged in. It aligns better with ground truth, producing visually more realistic deformations than the baseline.
3 个赞

@kaiwynd
PGND can also act as a photorealistic deformable-object simulator with a complete scan of the scene. Given only a static reconstruction, we simulate the segmented object’s motion with a sequence of robot actions (red arrows).
3 个赞

@kaiwynd
Finally, PGND serves as a 3D world model within Model Predictive Control. It guides dual-arm cloth lifting, rope shaping, box closing, and plush-toy relocation, achieving fast convergence to target configurations.
3 个赞

@kaiwynd
This work is a close collaboration between Columbia University @ColumbiaCompSci and University of Illinois Urbana-Champaign @siebelschool. Huge thanks to my co-authors: @YunzhuLiYZ, Kris Hauser, @BaoyuLi6 !
5 个赞

@the_carlosdp
I love this, and that you made a hf space demo
1 个赞

@Hongyu_Lii
This is an exciting work. Congrats!!

@kaiwynd
Thank you Hongyu!