Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Latent Spatial Memory for Video World Models

AK

517,079 subscribers

22,745 Aufrufe • vor 2 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

MindJourney Test-Time Scaling with World Models for Spatial Reasoning

MindJourney Test-Time Scaling with World Models for Spatial Reasoning

AK

61,738 Aufrufe • vor 1 Jahr

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 Aufrufe • vor 5 Monaten

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom

Video of my opening keynote for the 2026 World Modeling Workshop at Mila, Quebec AI Institute: simple but powerful ways of using world models and their latent space (sorry for sound problem 0:20-0:43). Details and references in: The Neural World Model Boom

Jürgen Schmidhuber

129,852 Aufrufe • vor 4 Monaten

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.

You’ve used language models, image models, video models, and voice models. Now it’s time for world models, thanks to World Labs.

Amir Haghighat

839,267 Aufrufe • vor 5 Monaten

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

Biwei Huang

14,359 Aufrufe • vor 22 Tagen

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity

ทำไมผู้หญิงชอบหลงทาง อ่าน Google Maps ไม่ออก บางคนจำทางเก่ง เดินไปไหนก็ไม่หลง แต่บางคนต่อให้เปิดแผนที่ก็ยังงงว่าต้องเลี้ยวซ้ายหรือขวา นี่คือเรื่อง Spatial Memory หรือความจำเชิงพื้นที่ของสมอง Spatial Memory คือความสามารถในการจำตำแหน่ง วัตถุ และเส้นทางรอบตัว สมองส่วน Hippocampus จะสร้าง “แผนที่ในหัว” หรือ Cognitive Map ให้เราใช้ในการนำทาง หากสมองส่วนนี้ไม่แข็งแรง เราจะหลงทางง่าย และอาจเป็นสัญญาณเริ่มต้นของสมองเสื่อม วิธีฝึก Spatial Memory 1.สังเกต Landmark รอบตัวเวลาเดินทาง 2.วาดแผนที่ในหัว หรือหมุนแผนที่ให้ตรงกับทิศทางที่เราเดิน 3.ออกกำลังกายแบบแอโรบิกเพื่อกระตุ้น Hippocampus 4.นอนหลับเพียงพอและลดความเครียด เพื่อให้สมองเรียนรู้ทิศทางได้ดีที่สุด การฝึก Spatial Memory ไม่ใช่แค่ช่วยให้ไม่หลงทาง แต่ยังเป็นการป้องกันสมองเสื่อมในอนาคต #BrainLongevity #TheLongevist #หมออั๋น #longevity

The Longevist

284,213 Aufrufe • vor 1 Jahr

experimenting with realtime latent consistency models in Figma

experimenting with realtime latent consistency models in Figma

Jordan Singer

374,693 Aufrufe • vor 2 Jahren

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:

AnchorWeave World-Consistent Video Generation with Retrieved Local Spatial Memories paper:

AK

23,312 Aufrufe • vor 6 Monaten

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Bardienus Duisterhof

65,041 Aufrufe • vor 2 Monaten

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Xingang Pan

15,946 Aufrufe • vor 8 Monaten

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.

Why LLMs are a dead end for human-level intelligence, and especially for Physical AI / Robotics. The next leap isn’t bigger language models. It’s World Models. I just dropped a full 1-hour presentation from Shanghai: “World Models: the ChatGPT moment for robotics?” → Why LLMs hit a wall → Why action-conditioned world models planning in latent space are the real path → Live World Forge demo with LeWorldModel + Hugging Face LeRobot Watch here. The future of intelligence is embodied, not just chatty.

abdel

38,730 Aufrufe • vor 2 Monaten

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗

Real World AI doesn’t fail because of models. It fails because it lacks context. Project Jade shows true spatial intelligence through an embodied AI experience on Snap Spectacles. 👓 Ready for the next AI wave? Niantic Spatial is already here. (1/4) 🔗

Niantic Spatial 🌎

16,582 Aufrufe • vor 7 Monaten

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

AK

82,138 Aufrufe • vor 3 Jahren

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.

for physical AI to work in the real world, it needs high-fidelity 3D models of the real world. not just for visualization — for localization, spatial reasoning, and real-to-sim workflows that close the gap between training and deployment. at Niantic Spatial 🌎, we're turning Spexi drone imagery into exactly that — city-scale Gaussian splats grounded in the environments that matter most.

asim ᯅ

49,478 Aufrufe • vor 2 Monaten

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

What does JEPA actually learn? We can finally prove it 🌍 So excited to share our theory of identifiable World Models: LeJEPA recovers the latent variables of the world. Plan in the learned World Model as if it were real, same shortest path. 📄:

What does JEPA actually learn? We can finally prove it 🌍 So excited to share our theory of identifiable World Models: LeJEPA recovers the latent variables of the world. Plan in the learned World Model as if it were real, same shortest path. 📄:

David Klindt

217,428 Aufrufe • vor 2 Monaten

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

Announcing a new Memory system for robots on Dimensional Robots in production generate thousands of hours of video, lidar, odometry, far too large to fit into your Agent context SpatialMemory2 builds a multimodal data store in latent space for your Agents Fully open source

stash

228,452 Aufrufe • vor 3 Monaten

India’s PM’s got latent Full video -

India’s PM’s got latent Full video -

Official PeeingHuman

17,783 Aufrufe • vor 8 Monaten

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

We're releasing HY-Embodied-0.5, a family of foundation models for real-world embodied agents. The 2B model is now open source. It strengthens spatial-temporal perception and embodied reasoning for prediction, interaction, and planning. 🤖 The suite includes: 🔹 2B for edge deployment 🔹 32B for complex reasoning Key innovations: 🔹 Mixture-of-Transformers (MoT) architecture for modality-specific computation 🔹 Latent tokens for improved perceptual representation 🔹 Self-evolving post-training 🔹 On-policy distillation from large to small models Across 22 benchmarks, the 2B model outperforms similarly sized SOTA systems on 16 tasks. The 32B model approaches frontier-level performance. 🔗 GitHub: 🤗 Hugging Face:

Tencent Hy

34,317 Aufrufe • vor 4 Monaten