跳过导航
Your world model may know less about the world than you think.
0:00 / 1:08

Your world model may know less about the world than you think.

Hazel Nam

@hazel_heejeong1508 位关注者

在 X 上查看
186 个赞下载原帖

11,078 次观看

Your world model may know less about the world than you think. I've always wondered what world models really learn. Humans don't memorize every trajectory. We learn rules we can reuse. Can world models do the same? We built AnyGoal to find out.

28 条评论

评论来自 X (Twitter) 上的原帖

  • Hazel Nam 的头像

    @hazel_heejeong

    [1/8] Most world models learn from expert demonstrations. But experts don't explore every possible action: they take actions that serve a goal. So how can we tell whether a model learned how the world works, or just how experts behave?

    5 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [2/8] One way: change only the goal, not the world. If a model understands how actions affect the environment, it should be able to reuse that knowledge for new goals, even when those goals require actions rarely taken by the expert. That's what AnyGoal tests.

    5 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [3/8] Consider Push-T: the expert approaches a block to push it into place. Now ask the agent to avoid the block instead. Moving around is familiar. The actions are familiar. But the new goal breaks the state-action patterns the model learned.

    4 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [4/8] We tested world models across JEPA-style and generative approaches, measuring both future prediction and model-based planning. A recurring result: success on the original task does not reliably transfer to a new goal in the same environment.

    4 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [5/8] The kind of interaction matters. With true physical states provided, models often handle free-space goal shifts much better than contact-rich ones. Predicting how to move around an object is not the same as predicting what happens when you touch it.

    3 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [6/8] One revealing test: hold the state fixed and change only the action. Where experts strongly favor one behavior, models can become less sensitive to the supplied action. They may be predicting what an expert would do, rather than what your action would cause.

    5 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [7/8] A practical takeaway: don't judge a world model by familiar rollouts alone, but test it under AnyGoal. We provide a benchmark and open-source tools to do just that. 🖥️ pip install anygoal 🖥️ anygoal eval planning --model lewm --dataset avoidt

    7 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    [8/8] Work done with amazing collaborators: @divyat09, @quentinlldc, @randall_balestr, and @klindt_david 🐈‍⬛ Github: 🤗 HuggingFace: 🩶 Paper: 🌐 Website:

    6 个赞

  • Sonia Joseph 的头像

    @soniajoseph_

    @randall_balestr @klindt_david @quentinlldc @divyat09 Really cool work!

    3 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    @randall_balestr @klindt_david @quentinlldc @divyat09 Thank you Sonia!

    3 个赞

  • Junyeob Baek 的头像

    @JunyeobB

    @randall_balestr @klindt_david @quentinlldc @divyat09 Cool work! Compositionality in world models is a key yet missing feature right now. Congrats!

    2 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    @randall_balestr @klindt_david @quentinlldc @divyat09 Thank you! I've also seen some of your cool works 😎

    1 个赞

  • Tarun Kathuria 的头像

    @_TarunKathuria

    @randall_balestr @klindt_david @quentinlldc @divyat09 This looks really nice! I'm wondering if you can actually use this to modify goal conditioned RL to force learning of more robust policies as well and by extension better WMs

    2 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    @randall_balestr @klindt_david @quentinlldc @divyat09 Yes absolutely! We provide data collection tools for all AnyGoal tasks for that purpose 🙌 We also have pre-collected datasets on HuggingFace, and you can always collect more episodes yourself and use them for training too!

    3 个赞

  • Tarun Kathuria 的头像

    @_TarunKathuria

    @randall_balestr @klindt_david @quentinlldc @divyat09 Sounds great! Nice work again!

    1 个赞

  • Vandon 的头像

    @vandon123456789

    @randall_balestr @klindt_david @quentinlldc @divyat09 Very cool! If you train on more diverse data that covers those rare combinations, does the gap close, or is it a limit of the model architecture itself?"

    2 个赞

  • Hazel Nam 的头像

    @hazel_heejeong

    @randall_balestr @klindt_david @quentinlldc @divyat09 Thanks Vandon! We didn't try that yet but I am pretty sure it will perform better. But at the same time, we believe recognizing those gaps from training data and finding offline data that fills the gap are also tough in real world🙃

    1 个赞

  • NextSignal 的头像

    @Grok9527

    @randall_balestr @klindt_david @quentinlldc @divyat09 世界模型评测如果只看轨迹预测准不准,很容易高估理解。AnyGoal 这种组合泛化测试更关键,它查的是规则能不能迁移到没见过的组合。机器人进真实环境前,这关过不了,演示再流畅也不稳。

    2 个赞

  • Konrad 的头像

    @durazzzz

    @randall_balestr @klindt_david @quentinlldc @divyat09 Interesting idea, congrats!

    2 个赞

  • Anish 的头像

    @Anishfishhh

    @randall_balestr @klindt_david @quentinlldc @divyat09 yoooo this is sick!!!! Would love to chat :) world models are so cool, but would love to learn more about making them more general!!

    2 个赞