跳过导航
Can we learn a 3D world model that predicts object dynamics directly from videos?
0:00 / 0:34

Can we learn a 3D world model that predicts object dynamics directly from videos?

Kaifeng Zhang

@kaiwynd3631 位关注者

在 X 上查看
165 个赞下载原帖

46,666 次观看

Can we learn a 3D world model that predicts object dynamics directly from videos? Introducing Particle-Grid Neural Dynamics: a learning-based simulator for deformable objects that trains from real-world videos. Website: ArXiv: Code: Demo: To appear at #RSS2025

10 条评论

评论来自 X (Twitter) 上的原帖

  • Kaifeng Zhang 的头像

    @kaiwynd

    Modeling ropes, cloth, bags, etc. is hard because of their complex physics and partial observability. Classical simulators struggle to construct exact digital twins from real observations. We overcome these challenges by learning neural dynamics directly from videos.

    4 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    Our particle-based neural dynamics model represents objects as dense 3D particles and predicts their next-step velocities to simulate object dynamics. It features three stages: particle encoding, grid-velocity editing, and grid-to-particle velocity transfer.

    3 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    Trained with videos including robot–object interactions under self-supervion, PGND can model diverse deformable objects—including ropes, cloth, stuffed animals, and paper bags—using <20 minutes of data per object.

    3 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    PGND becomes a 3D action-conditioned video generator when 3D Gaussian Splatting is plugged in. It aligns better with ground truth, producing visually more realistic deformations than the baseline.

    3 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    PGND can also act as a photorealistic deformable-object simulator with a complete scan of the scene. Given only a static reconstruction, we simulate the segmented object’s motion with a sequence of robot actions (red arrows).

    3 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    Finally, PGND serves as a 3D world model within Model Predictive Control. It guides dual-arm cloth lifting, rope shaping, box closing, and plush-toy relocation, achieving fast convergence to target configurations.

    3 个赞

  • Kaifeng Zhang 的头像

    @kaiwynd

    This work is a close collaboration between Columbia University @ColumbiaCompSci and University of Illinois Urbana-Champaign @siebelschool. Huge thanks to my co-authors: @YunzhuLiYZ, Kris Hauser, @BaoyuLi6 !

    5 个赞

  • Carlos DP 的头像

    @the_carlosdp

    I love this, and that you made a hf space demo

    1 个赞

  • Hongyu Li 的头像

    @Hongyu_Lii

    This is an exciting work. Congrats!!

  • Kaifeng Zhang 的头像

    @kaiwynd

    Thank you Hongyu!