Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Latent Spatial Memory for Video World Models

_akhaliq's profile picture

AK

530,057 subscribers

22,812 Aufrufe • vor 3 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

5 Kommentare

Profilbild von AK
AKvor 3 Monaten

paper:

Profilbild von Weijie Wang ✈️ ECCV
Weijie Wang ✈️ ECCVvor 3 Monaten

Thanks @_akhaliq ! Author thread: Explore more at:

Profilbild von Ferbin
Ferbinvor 3 Monaten

This is the hard part for robotics. You can see the room once. Keeping track of where things are as you move, knowing what changed. That's the whole game. Video is easy. Memory is hard.

Profilbild von Victoria Darling
Victoria Darlingvor 3 Monaten

just went from 115 to 120kg deadlift. felt like i was lifting a car

Profilbild von Truman
Trumanvor 3 Monaten

Please check out the SO-101 benchmark I posted!

Ähnliche Videos

MindJourney Test-Time Scaling with World Models for Spatial Reasoning
0:43

MindJourney Test-Time Scaling with World Models for Spatial Reasoning

AK

61,826 Aufrufe • vor 1 Jahr

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:
0:47

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 Aufrufe • vor 6 Monaten

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom
31:16

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom

Jürgen Schmidhuber

129,852 Aufrufe • vor 5 Monaten

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.
3:42

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.

Amir Haghighat

839,697 Aufrufe • vor 7 Monaten

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI
0:53

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

Biwei Huang

15,084 Aufrufe • vor 2 Monaten

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity
0:53

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity

The Longevist

284,313 Aufrufe • vor 1 Jahr

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:
0:24

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:

AK

23,344 Aufrufe • vor 7 Monaten

experimenting with realtime latent consistency models in Figma
0:30

experimenting with realtime latent consistency models in Figma

Jordan Singer

374,693 Aufrufe • vor 2 Jahren

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]
0:25

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Bardienus Duisterhof @ IROS 🌉

66,913 Aufrufe • vor 3 Monaten

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:
0:25

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Xingang Pan

15,967 Aufrufe • vor 9 Monaten

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.
1:16:13

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.

abdel

39,459 Aufrufe • vor 4 Monaten

Lion Dance trainees with absolutely impressive spatial awareness and muscle memory!
0:27

Lion Dance trainees with absolutely impressive spatial awareness and muscle memory!

Au〆

234,421 Aufrufe • vor 18 Tagen

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗
1:00

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗

Niantic Spatial 🌎

16,582 Aufrufe • vor 9 Monaten

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.
0:28

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.

Ying Shan

26,688 Aufrufe • vor 14 Tagen

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:
1:36

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

AK

82,138 Aufrufe • vor 3 Jahren

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.
1:00

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.

asim ᯅ

49,566 Aufrufe • vor 4 Monaten

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source
0:24

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

stash

228,761 Aufrufe • vor 5 Monaten