Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Yu et al., "MosaicMem: Hybrid Spatial Memory for Controllable Video World Models" A patch-based spatial memory that you raster into views + glues to make things work.

kwangmoo_yi's profile picture

Kwang Moo Yi

4,653 subscribers

11,170 görüntüleme • 4 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

SuperMap: A Living Spatial Memory for Embodied AI RSS 2026 Carnegie Mellon University SuperMap is a living spatial memory for embodied AI. It perceives the world, remembers its evolution, and supports reasoning and action. It is a training-free spatio-temporal SLAM system that builds a persistent semantic world model. It fuses high-frequency geometric SLAM with asynchronous open-vocabulary perception, producing a 4D scene graph: a queryable map carrying spatial and temporal information for every object, enabling visual-language navigation and long-horizon reasoning on real robots.
0:16

SuperMap: A Living Spatial Memory for Embodied AI RSS 2026 Carnegie Mellon University SuperMap is a living spatial memory for embodied AI. It perceives the world, remembers its evolution, and supports reasoning and action. It is a training-free spatio-temporal SLAM system that builds a persistent semantic world model. It fuses high-frequency geometric SLAM with asynchronous open-vocabulary perception, producing a 4D scene graph: a queryable map carrying spatial and temporal information for every object, enabling visual-language navigation and long-horizon reasoning on real robots.

Ryohei Sasaki@engineer

24,629 görüntüleme • 10 gün önce

The context size of video world models is only a few frames. Like a human with severe memory loss! We design a long-term memory for world models based on explicit 3D representations inspired by the human mind. This enables long-term consistency. 1/3
0:12

The context size of video world models is only a few frames. Like a human with severe memory loss! We design a long-term memory for world models based on explicit 3D representations inspired by the human mind. This enables long-term consistency. 1/3

Gordon Wetzstein

35,075 görüntüleme • 1 yıl önce

AI’s next frontier is Spatial Intelligence, a technology that will turn seeing into reasoning, perception into action, and imagination into creation. But what is it? Why does it matter? How do we build it? And how can we use it? Today, I want to share with you my thoughts on building and using world models to unlock spatial intelligence in this essay below. 1/n
0:11

AI’s next frontier is Spatial Intelligence, a technology that will turn seeing into reasoning, perception into action, and imagination into creation. But what is it? Why does it matter? How do we build it? And how can we use it? Today, I want to share with you my thoughts on building and using world models to unlock spatial intelligence in this essay below. 1/n

Fei-Fei Li

927,274 görüntüleme • 9 ay önce

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵
0:12

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,060 görüntüleme • 1 yıl önce

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.
0:12

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 görüntüleme • 1 yıl önce

Hope y’all vibing to that sunday feelin .. take a memory pic .. cause u never know when things make changes in the world u live in .. ✌️🫶 ❤️
0:14

Hope y’all vibing to that sunday feelin .. take a memory pic .. cause u never know when things make changes in the world u live in .. ✌️🫶 ❤️

the good boi 🐍

29,208 görüntüleme • 8 ay önce

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,052 görüntüleme • 1 yıl önce

Google 3D Maps aren’t as accurate as you think. Everyone knows Google dominates mapping—but here’s what they won’t tell you. Their 3D maps are built with car-mounted cameras, capturing images every ~10 meters. That’s fine for basic navigation, but it’s nowhere near enough for AR, robotics, autonomous systems, or AI-driven spatial intelligence. Now, imagine a map so detailed it captures the world at sub-5cm accuracy. That’s OVRMaps. 🌐 OVER 3D maps are built with 400-1,000 images per 300 sqm, taken from multiple angles at pedestrian level. The result? A new era of hyper-precise localization that changes everything. Why does this matter? ↳ VPS that actually works – Real-world AR anchoring, digital twins, and AI-powered spatial computing with pinpoint accuracy. ↳ Next-gen precision – Essential for ride-sharing, robotics, smart cities, and asset tracking—where even a small error makes a huge difference. ↳ AI-Driven Spatial Intelligence – Our maps fuel Large Geo-Spatial Models (LGMs), the AI revolution enabling machines to understand, navigate, and interact with the real world. ↳ The Spatial Computing revolution – Merging immersive experiences with real-world precision, unlocking applications beyond what Google’s dataset can support. OVRMaps are built for the future. 🌍 Start mapping today:
0:12

Google 3D Maps aren’t as accurate as you think. Everyone knows Google dominates mapping—but here’s what they won’t tell you. Their 3D maps are built with car-mounted cameras, capturing images every ~10 meters. That’s fine for basic navigation, but it’s nowhere near enough for AR, robotics, autonomous systems, or AI-driven spatial intelligence. Now, imagine a map so detailed it captures the world at sub-5cm accuracy. That’s OVRMaps. 🌐 OVER 3D maps are built with 400-1,000 images per 300 sqm, taken from multiple angles at pedestrian level. The result? A new era of hyper-precise localization that changes everything. Why does this matter? ↳ VPS that actually works – Real-world AR anchoring, digital twins, and AI-powered spatial computing with pinpoint accuracy. ↳ Next-gen precision – Essential for ride-sharing, robotics, smart cities, and asset tracking—where even a small error makes a huge difference. ↳ AI-Driven Spatial Intelligence – Our maps fuel Large Geo-Spatial Models (LGMs), the AI revolution enabling machines to understand, navigate, and interact with the real world. ↳ The Spatial Computing revolution – Merging immersive experiences with real-world precision, unlocking applications beyond what Google’s dataset can support. OVRMaps are built for the future. 🌍 Start mapping today:

Over the Reality 🌐

1,331,567 görüntüleme • 1 yıl önce

Robots struggle with strict action rules…memory and symbols help them learn fast. [Project + Full video link ⬇️] Robots struggle when tasks require specific steps in a fixed order. What if memory helped them think symbolically and learn faster? Solving tasks like unlocking a door then opening it is hard for deep RL. But by learning constraint relationships and storing them in memory, robots can solve these tasks much faster; with fewer trials and less training. Why it works ✅ Learns symbolic rules about action constraints ✅ Uses memory to transfer what it learned across tasks ✅ Handles real-world exploration with just 30 minutes of data ✅ Needs 10x fewer episodes than deep RL approaches This memory-based method shows a promising path forward for robots learning structured, real-world tasks. Full video: Paper: Thank you, Mrinal Verghese for sharing this amazing work! 🙏
0:18

Robots struggle with strict action rules…memory and symbols help them learn fast. [Project + Full video link ⬇️] Robots struggle when tasks require specific steps in a fixed order. What if memory helped them think symbolically and learn faster? Solving tasks like unlocking a door then opening it is hard for deep RL. But by learning constraint relationships and storing them in memory, robots can solve these tasks much faster; with fewer trials and less training. Why it works ✅ Learns symbolic rules about action constraints ✅ Uses memory to transfer what it learned across tasks ✅ Handles real-world exploration with just 30 minutes of data ✅ Needs 10x fewer episodes than deep RL approaches This memory-based method shows a promising path forward for robots learning structured, real-world tasks. Full video: Paper: Thank you, Mrinal Verghese for sharing this amazing work! 🙏

Ilir Aliu - eu/acc

10,241 görüntüleme • 1 yıl önce

A sneak peak of a complex and technically challenging experiment that my lab developed: Super proud of PhD candidate Hannah Johnson for showcasing our Whole-gut spatial genomic analysis in #zebrafish. This video illustrates one landmark in the protocol after multiple rounds of sequential #HCR and 3D imaging in zebrafish larvae to reveal spatial expression of numerous mRNAs in the same specimen. Data from these imaging data sets are then computationally analyzed for spatial cell groups, spatially variable genes, and differentially expressed genes along 3D. We are leveraging this systems-level SGA to uncover unappreciated mechanistic insight at the cell and tissue levels into #ENS construction. Stay tuned for our work that exploits this pipeline within various mutant and perturbation conditions. Reach out if you are interested in trying this! #fruitypebbles
0:18

A sneak peak of a complex and technically challenging experiment that my lab developed: Super proud of PhD candidate Hannah Johnson for showcasing our Whole-gut spatial genomic analysis in #zebrafish. This video illustrates one landmark in the protocol after multiple rounds of sequential #HCR and 3D imaging in zebrafish larvae to reveal spatial expression of numerous mRNAs in the same specimen. Data from these imaging data sets are then computationally analyzed for spatial cell groups, spatially variable genes, and differentially expressed genes along 3D. We are leveraging this systems-level SGA to uncover unappreciated mechanistic insight at the cell and tissue levels into #ENS construction. Stay tuned for our work that exploits this pipeline within various mutant and perturbation conditions. Reach out if you are interested in trying this! #fruitypebbles

Rosa Uribe, PhD

10,119 görüntüleme • 5 ay önce

LLMs require more GPU memory as they generate longer responses. Can we make GPU memory constant without significantly sacrificing accuracy? IceCache is a new method for managing KV caches that leverages Dynamic Continuous Indexing (DCI) to efficiently group and retrieve tokens by semantics. Joint work w/ Yuzhen Mao, Qitong Wang and Martin Ester. For details, check out the links below.
0:13

LLMs require more GPU memory as they generate longer responses. Can we make GPU memory constant without significantly sacrificing accuracy? IceCache is a new method for managing KV caches that leverages Dynamic Continuous Indexing (DCI) to efficiently group and retrieve tokens by semantics. Joint work w/ Yuzhen Mao, Qitong Wang and Martin Ester. For details, check out the links below.

Ke Li 🍁

21,163 görüntüleme • 3 ay önce

🎮 3D artists spend hours building the things AI video usually ignores: camera position, spatial relationships, character blocking, proportions, and scene structure. Then they flatten all that work into a single prompt — and hope the model understands what matters. That workflow is about to change. Dreamina has unveiled the world-first global launch of Dreamina Seedance 2.5, the official upgrade to Seedance 2.0. Early access is live on Dreamina, while US availability is expected roughly one week later. It’s whether AI video can finally build on the structured work 3D artists already created — at the lowest cost across the entire Seedance family.
0:10

🎮 3D artists spend hours building the things AI video usually ignores: camera position, spatial relationships, character blocking, proportions, and scene structure. Then they flatten all that work into a single prompt — and hope the model understands what matters. That workflow is about to change. Dreamina has unveiled the world-first global launch of Dreamina Seedance 2.5, the official upgrade to Seedance 2.0. Early access is live on Dreamina, while US availability is expected roughly one week later. It’s whether AI video can finally build on the structured work 3D artists already created — at the lowest cost across the entire Seedance family.

Alamin

472,595 görüntüleme • 9 gün önce

📢 Meta just posted this small video on Project Orion which looks awesome and (brings me back to ML1 days ☺️) except for the form-factor which makes it more compelling to a wider audience. - Full-color holographic lenses (waveguides) 70 degree FoV diagonal. - Lightweight and comfortable design for extended wear. - Hand tracking for natural gestures. - Voice commands to control apps hands-free. - Gesture recognition for quick interactions. - Neural interface wristband translates biometric signals into commands. - Spatial awareness to detect user movement and environment. - Eye tracking for focus-based input and selection. - Spatial audio enables immersive sound that matches visuals. - Computer puck similar to ML1 & ML2 to offload its processing. - Battery life is up to 3 hours for the AR Glasses & all day for the compute puck.
0:13

📢 Meta just posted this small video on Project Orion which looks awesome and (brings me back to ML1 days ☺️) except for the form-factor which makes it more compelling to a wider audience. - Full-color holographic lenses (waveguides) 70 degree FoV diagonal. - Lightweight and comfortable design for extended wear. - Hand tracking for natural gestures. - Voice commands to control apps hands-free. - Gesture recognition for quick interactions. - Neural interface wristband translates biometric signals into commands. - Spatial awareness to detect user movement and environment. - Eye tracking for focus-based input and selection. - Spatial audio enables immersive sound that matches visuals. - Computer puck similar to ML1 & ML2 to offload its processing. - Battery life is up to 3 hours for the AR Glasses & all day for the compute puck.

Dilmer

45,962 görüntüleme • 1 yıl önce

📢Introducing Generated Reality📢 A world model for XR that turns your tracked hand and head poses into an interactive, generative video experience. Take world models to the next level by interacting with the world using your own body! 🔗 1/4
0:15

📢Introducing Generated Reality📢 A world model for XR that turns your tracked hand and head poses into an interactive, generative video experience. Take world models to the next level by interacting with the world using your own body! 🔗 1/4

Gordon Wetzstein

20,045 görüntüleme • 5 ay önce

Here's a video of Bertie running, and a few other photos worth sharing. Thanks for reading through this thread, I want to do more of this where I explain all the work that goes into these models.
0:16

Here's a video of Bertie running, and a few other photos worth sharing. Thanks for reading through this thread, I want to do more of this where I explain all the work that goes into these models.

Covey

19,358 görüntüleme • 1 yıl önce

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.
0:17

This is some quietly impressive work on making video world models actually controllable in 4D space. VerseCrafter lets you take an input image, use something like Blender to animate the 3D camera path and object trajectories, then uses that to condition generation. Scribbling in 2D feels so crude in comparison. The authors represent everything in a shared 4D world state - static background as a point cloud, moving objects as 3D gaussian trajectories. The gaussians are an interesting choice because they capture position, shape, and orientation probabilistically rather than forcing rigid bounding boxes or category specific models like SMPL-X for human bodies. They bolt this onto frozen Wan2.1 with a lightweight adapter, so they get a strong video prior. They also built a pipeline to auto extract 4D annotations from real world videos to train this puppy. It doesn't look sexy yet, but IMO this is the interface video world models need - actual 3D authoring tools to exert control rather than crude scribbles and prompt incantations.

Bilawal Sidhu

25,802 görüntüleme • 6 ay önce

GLITCHED is my indie RPG where the protagonist & world are corrupted & a lowly NPC has to patch everything back together the 4th wall doesn't exist, so don't tell the NPC you control that he's in a video game my #indiedev life's work, coming soon #screenshotsaturday #pixelart
0:14

GLITCHED is my indie RPG where the protagonist & world are corrupted & a lowly NPC has to patch everything back together the 4th wall doesn't exist, so don't tell the NPC you control that he's in a video game my #indiedev life's work, coming soon #screenshotsaturday #pixelart

nikko (GLITCHED RPG)

20,201 görüntüleme • 2 yıl önce

🔒 WARNING: LEDGER CTO ISSUES ALERT FOR CRYPTO USERS AFTER CRITICAL CHROME UPDATE Ledger CTO Charles Guillemet flagged a major Google Chrome security patch addressing 26 vulnerabilities, including 4 critical and 22 high-severity flaws. These include memory management errors that could allow attackers to execute malicious code remotely through crafted webpages. Crypto users relying on browser-based wallets and extensions are particularly at risk. Users are urged to update Chrome immediately.
0:16

🔒 WARNING: LEDGER CTO ISSUES ALERT FOR CRYPTO USERS AFTER CRITICAL CHROME UPDATE Ledger CTO Charles Guillemet flagged a major Google Chrome security patch addressing 26 vulnerabilities, including 4 critical and 22 high-severity flaws. These include memory management errors that could allow attackers to execute malicious code remotely through crafted webpages. Crypto users relying on browser-based wallets and extensions are particularly at risk. Users are urged to update Chrome immediately.

BSCN

141,021 görüntüleme • 4 ay önce

Rocket Keeps Growing. Another 20,000 Android users joined the Rocket rollout. Every rollout helps us make Rocket better. Every day brings new capabilities. As access expands, Rocket continues improving through real-world usage while new features arrive every week. → One AI instead of dozens of disconnected agents → Long-term memory that grows with you → Semantic Video Search that understands videos—not just transcripts → Shared intelligence, where one Rocket's work benefits everyone We're rolling Rocket out in stages to ensure a great experience for every new user. The goal? Rocket in everyone's pocket.
0:10

Rocket Keeps Growing. Another 20,000 Android users joined the Rocket rollout. Every rollout helps us make Rocket better. Every day brings new capabilities. As access expands, Rocket continues improving through real-world usage while new features arrive every week. → One AI instead of dozens of disconnected agents → Long-term memory that grows with you → Semantic Video Search that understands videos—not just transcripts → Shared intelligence, where one Rocket's work benefits everyone We're rolling Rocket out in stages to ensure a great experience for every new user. The goal? Rocket in everyone's pocket.

UpRock

14,649 görüntüleme • 28 gün önce