Manling Li's banner
Manling Li's profile picture

Manling Li

@ManlingLi_ • 11,259 subscribers

Assistant Professor@NWU, Amazon Scholar, Postdoc@Stanford, PhD@UIUC #Language+#Vision/#Robotics, #Reasoning, #Planning, RAGEN, VAGEN, BAGEN, MindCube, ViewAgent

Shorts

📍Theory of Space (accepted at #ICLR2026) Theory of Mind → hidden mental states Theory of Space → hidden spatial beliefs from passive observers “What do I know?” to active explorers “What don’t I know, and how do I reduce that uncertainty?” Theory of Space is to evaluate if foundation models can actively construct, revise, and exploit internal spatial beliefs. We quantify Active-Passive Gap. Not just measure task accuracy, but how much uncertainty is reduced per step, and how many steps are needed in total for agents to build stable spatial beliefs. Exploration should prioritize information gain and reduce uncertainty per step. Instead, we observe LLMs/VLMs explore redundantly with stalled belief updates. Key findings: 1. Active agents perform worse than rule based programs 2. Cognitive Map Failures & Belief Drift (beliefs about previously observed objects degrades over time; new updates corrupt earlier correct perceptions) 3. Poor Vision Identification & Belief Inertia in Belief Revision Website: Code: Data: Theory of Space is a joint effort of Northwestern Engineering, Stanford AI Lab, Allen School, Cornell Computer Science. Led by the amazing WilliamZhang, jointly done with Zihan Huang, yue wang, Jieyu Zhang, Lester Xue, @wzihanw, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Fei-Fei Li

📍Theory of Space (accepted at #ICLR2026) Theory of Mind → hidden mental states Theory of Space → hidden spatial beliefs from passive observers “What do I know?” to active explorers “What don’t I know, and how do I reduce that uncertainty?” Theory of Space is to evaluate if foundation models can actively construct, revise, and exploit internal spatial beliefs. We quantify Active-Passive Gap. Not just measure task accuracy, but how much uncertainty is reduced per step, and how many steps are needed in total for agents to build stable spatial beliefs. Exploration should prioritize information gain and reduce uncertainty per step. Instead, we observe LLMs/VLMs explore redundantly with stalled belief updates. Key findings: 1. Active agents perform worse than rule based programs 2. Cognitive Map Failures & Belief Drift (beliefs about previously observed objects degrades over time; new updates corrupt earlier correct perceptions) 3. Poor Vision Identification & Belief Inertia in Belief Revision Website: Code: Data: Theory of Space is a joint effort of Northwestern Engineering, Stanford AI Lab, Allen School, Cornell Computer Science. Led by the amazing WilliamZhang, jointly done with Zihan Huang, yue wang, Jieyu Zhang, Lester Xue, @wzihanw, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Fei-Fei Li

52,794 views

Videos

LIVE
1.2k

Anya Rossi

sweetdream.ai

SweetDream.aiSponsored•Livecam
Streaming Now

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

HD live stream
Exclusive private shows
1.2k viewers online

Free preview available • Premium content

Planning with the views: Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We introduce ViewSuite with 6 DoF camera control and ~165K task instances, testing: Path-to-View View-to-Path Interactive View Planning A sharp Planning Gap emerges: + can roughly "track" how camera action changes views - cannot "compose" a plan towards a target view at all We then try to teach VLMs with Reinforcement Learning. - RL cannot teach VLMs such planning ability, only 2.5% success rate with Qwen2.5-VL-7B. + With View Graph Distillation (our RL-Graph-SFT framework), 2.5% → 47.8% Below, we answer these questions: Q1. What are the failure modes? Q2. How can we make RL work? Q3. What has the model learned? Can we open up the model to see before/after? Can such spatial priors transfer to other view related tasks? Led by Kangrui Wang, great to work with Linjie (Lindsey) Li Zhengyuan Yang Shiqi Chen Zihan "Zenus" Wang ✈️ ICML Fei-Fei Li Jiajun Wu Leonidas Guibas, Lijuan Wang. A joint efforts with Stanford AI Lab Stanford Vision and Learning Lab Microsoft Research.
0:47
ManlingLi_'s profile picture

Planning with the views: Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We introduce ViewSuite with 6 DoF camera control and ~165K task instances, testing: Path-to-View View-to-Path Interactive View Planning A sharp Planning Gap emerges: + can roughly "track" how camera action changes views - cannot "compose" a plan towards a target view at all We then try to teach VLMs with Reinforcement Learning. - RL cannot teach VLMs such planning ability, only 2.5% success rate with Qwen2.5-VL-7B. + With View Graph Distillation (our RL-Graph-SFT framework), 2.5% → 47.8% Below, we answer these questions: Q1. What are the failure modes? Q2. How can we make RL work? Q3. What has the model learned? Can we open up the model to see before/after? Can such spatial priors transfer to other view related tasks? Led by Kangrui Wang, great to work with Linjie (Lindsey) Li Zhengyuan Yang Shiqi Chen Zihan "Zenus" Wang ✈️ ICML Fei-Fei Li Jiajun Wu Leonidas Guibas, Lijuan Wang. A joint efforts with Stanford AI Lab Stanford Vision and Learning Lab Microsoft Research.

Manling Li

58,249 views • 25 days ago

[NeurIPS D&B Oral] Embodied Agent Interface: Benchmarking LLMs for Embodied Agents A single line of code to evaluate your model! 🌟Standardize Goal Specifications: LTL 🌟Standardize Modules and Interfaces: 4 modules, 438 tasks, 1475 goals 🌟Standardize Fine-grained Metrics: 18 models, 42 metrics, 100+ page analysis Website: Dataset: Code: Docker: PyPI: Doc:
0:53
ManlingLi_'s profile picture

[NeurIPS D&B Oral] Embodied Agent Interface: Benchmarking LLMs for Embodied Agents A single line of code to evaluate your model! 🌟Standardize Goal Specifications: LTL 🌟Standardize Modules and Interfaces: 4 modules, 438 tasks, 1475 goals 🌟Standardize Fine-grained Metrics: 18 models, 42 metrics, 100+ page analysis Website: Dataset: Code: Docker: PyPI: Doc:

Manling Li

132,665 views • 1 year ago

World Model Reasoning for VLM Agents (NeurIPS 2025, Score 5544) We release VAGEN to teach VLMs to build internal world models via visual state reasoning: - StateEstimation: what is the current state? - TransitionModeling: what is next? MDP → POMDP shift to handle the partial observability from visual states! 🙌Led by Kangrui Wang WilliamZhang @wzihanw Yaning Gao Linjie (Lindsey) Li Qineng Wang Hanyang Chen w4nanch1 Yiping Lu Zhengyuan Yang lijuanwang,Ranjay Krishna Jiajun Wu Fei-Fei Li Yejin Choi 👍Grateful for the joint effort of Northwestern University Computer Science Allen School Stanford AI Lab Microsoft UW–Madison Computer Sciences Siebel School of Computing and Data Science.
0:21
ManlingLi_'s profile picture

World Model Reasoning for VLM Agents (NeurIPS 2025, Score 5544) We release VAGEN to teach VLMs to build internal world models via visual state reasoning: - StateEstimation: what is the current state? - TransitionModeling: what is next? MDP → POMDP shift to handle the partial observability from visual states! 🙌Led by Kangrui Wang WilliamZhang @wzihanw Yaning Gao Linjie (Lindsey) Li Qineng Wang Hanyang Chen w4nanch1 Yiping Lu Zhengyuan Yang lijuanwang,Ranjay Krishna Jiajun Wu Fei-Fei Li Yejin Choi 👍Grateful for the joint effort of Northwestern University Computer Science Allen School Stanford AI Lab Microsoft UW–Madison Computer Sciences Siebel School of Computing and Data Science.

Manling Li

63,619 views • 9 months ago

Can VLMs build Spatial Mental Models like humans? Reasoning from limited views? Reasoning from partial observations? Reasoning about unseen objects behind furniture / beyond current view? Check out MindCube! 🌐 📰 🤗 👩‍💻
1:26
ManlingLi_'s profile picture

Can VLMs build Spatial Mental Models like humans? Reasoning from limited views? Reasoning from partial observations? Reasoning about unseen objects behind furniture / beyond current view? Check out MindCube! 🌐 📰 🤗 👩‍💻

Manling Li

40,959 views • 1 year ago

No more content to load