Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Latent Spatial Memory for Video World Models

AK

530,057 subscribers

22,812 views • 3 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

AK3 months ago

paper:

Weijie Wang ✈️ ECCV3 months ago

Thanks @_akhaliq ! Author thread: Explore more at:

Ferbin3 months ago

This is the hard part for robotics. You can see the room once. Keeping track of where things are as you move, knowing what changed. That's the whole game. Video is easy. Memory is hard.

Victoria Darling3 months ago

just went from 115 to 120kg deadlift. felt like i was lifting a car

Truman3 months ago

Please check out the SO-101 benchmark I posted!

Related Videos

MindJourney Test-Time Scaling with World Models for Spatial Reasoning

MindJourney Test-Time Scaling with World Models for Spatial Reasoning

AK

61,826 views • 1 year ago

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 views • 6 months ago

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom

Jürgen Schmidhuber

129,852 views • 5 months ago

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.

Amir Haghighat

839,697 views • 7 months ago

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

Biwei Huang

15,084 views • 2 months ago

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity

The Longevist

284,313 views • 1 year ago

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:

AK

23,344 views • 7 months ago

experimenting with realtime latent consistency models in Figma

experimenting with realtime latent consistency models in Figma

Jordan Singer

374,693 views • 2 years ago

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Bardienus Duisterhof @ IROS 🌉

66,913 views • 3 months ago

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Xingang Pan

15,967 views • 9 months ago

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.

abdel

39,459 views • 4 months ago

Lion Dance trainees with absolutely impressive spatial awareness and muscle memory!

Lion Dance trainees with absolutely impressive spatial awareness and muscle memory!

Au〆

234,421 views • 18 days ago

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗

Niantic Spatial 🌎

16,582 views • 9 months ago

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.

Ying Shan

26,688 views • 14 days ago

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

AK

82,138 views • 3 years ago

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.

asim ᯅ

49,566 views • 4 months ago

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

stash

228,761 views • 5 months ago