Can VLMs build Spatial Mental Models like humans? Reasoning from limited views? Reasoning from partial observations? Reasoning about unseen objects behind furniture / beyond current view? Check out MindCube! 🌐 📰 🤗 👩‍💻

We build MindCube with 21, 154 questions across 3, 268 images, where existing VLMs only show near-random performance☹️

Can VLMs approximate Spatial Mental Models? like Cognitive Maps, Reasoning Chains, View Interpolation?

Do these scaffolds improve spatial reasoning without training? Can we simply prompt the model to think spatially? How far can structure alone go? Although VLMs can output seemingly good cognitive maps, but the Isomorphic Rates is actually lower than 10%!

Can we teach VLMs to build and leverage spatial representations? We then train VLMs by providing 10, 000 reasoning chains and 10, 000 cognitive maps, investigating how to effectively guide their thinking process. We find: "Map then Reasoning" >> Only Reasoning, or Only Cognitive Maps

Do VLMs truly benefit from explicit training? SFT on "reasoning over cognitive maps" significantly boosts isomorphic similarity: 0.1% → 46.0% for the augmented cognitive maps 7.4% → 73.8% for the plain cognitive maps

Can Reinforcement Learning further refine spatial thought processes? We find that RL in a vacuum is not enough. Structured outputs provide modest benefits when learned from scratch. However, RL shines when it stands on an SFT-built scaffold:)

MindCube is the result of a joint effort of @NorthwesternEng @StanfordAILab @StanfordHAI @StanfordSVL @NYU_Courant @uwcse. Huge thanks to project leader @qineng_wang @Baiqiao_Yin and our incredible team @drfeifei @jiajunwu_cs @sainingxie @RanjayKrishna @HanLiu @WilliamZhangNU @SterZhang @James_KKW @wzihanw @JieyuZhang20 @keshigeyan ♥️ This is just the beginning—excited for the future of spatial reasoning from partial observations, and what’s next! Reply or email us with questions, ideas, or use cases Join us:

MindCube’s modular pipeline and cognitive mapping are pushing VLM spatial reasoning much closer to human levels. Open-source tools like this drive real progress in the field! 🤖

Maybe next up would be a torus.

A: Cognitive maps for VLMs, nice! 🗺️ I'm curious, how do you see this approach handling dynamic environments or changing object locations?

Our “what if” questions target this exactly by asking models to dynamically update cognitive maps, like humans maintaining a memory. However, cognitive maps lost low level visual details so I am excited to how far language can go for spatial reasoning.

People are racing to push math reasoning performance in #LLMs—but have we really asked why? The common assumption is that improving math reasoning should transfer to broader capabilities in other domains. But is that actually true? In our study ( we evaluated over 20 open-weight reasoning models and found that: ➡️Only models trained with RL exhibit broad transfer of math reasoning skills to other tasks. ➡️Models trained with SFT show limited or no transfer—especially to non-reasoning domains. To quantify this, we introduce the Transferability Index (TI), which measures how much gain in math could transfer to others. A positive score indicates effective transfer; a negative one suggests loss of general capability. We evaluate the models on three benchmark categories: - Math reasoning: MATH-500, AIME24/25, Olympiad - Other reasoning: GPQA-D (Science), LiveCodeBench2 (Code), ACPBench (Agent Planning), HeadQA (Medical) - Non-reasoning: CoQA (Conversational QA), IFEval (Instruction Following), HalluEval (Hallucination), MC-TACO (Commonsense) Our findings challenge the blind pursuit of leaderboard performance in math reasoning via SFT. Simply creating more math-like SFT data may inadvertently harm a model’s broader generalization. Instead, RL appears to be key for truly transferable reasoning development.

Many people may not know that Meta FAIR (Facebook AI Research) and Google DeepMind were the most prestigious industry research labs before OpenAI dropped ChatGPT. OpenAI was also very good, but not as good as the other two.

📢 today's scaling laws often don't work for predicting downstream task performance. For some pretraining setups, smooth and predictable scaling is the exception, not the rule. a quick read about scaling law fails: 📜 🧵1/5👇

Post-training of LLMs is increasingly important and RLHF remains a necessary step for an overall great model. Today we are releasing 6 new reward models, including GenRMs and multilingual. These models are used to post-train next *-nemotron models.