Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Microsoft Research introduces Mirage Latent spatial memory stores 3D scenes directly as latent tokens, skipping the costly RGB render-and-reencode loop. The result is up to 10.57x faster video generation, 55x lower memory use, and state-of-the-art consistency on WorldScore.

DailyPapers

19,698 subscribers

18,678 views • 1 month ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:

Tencent Hy

198,289 views • 10 months ago

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 views • 3 months ago

Shadows of The Shining Exploring the latent space of The Shining is to approach the film as one approaches a memory. Here, artificial intelligence acts as a second consciousness: it does not reproduce — it remembers. It reconstructs The Shining from its traces, its echoes, from what has survived the narrative. The latent space becomes a mental territory, an inner Overlook where the ghosts of the image, the logic of dreams, and the mechanisms of visual memory unfold once again. The aim is to turn the exploration of the latent space into a mnemonic experience of cinema itself.

Shadows of The Shining Exploring the latent space of The Shining is to approach the film as one approaches a memory. Here, artificial intelligence acts as a second consciousness: it does not reproduce — it remembers. It reconstructs The Shining from its traces, its echoes, from what has survived the narrative. The latent space becomes a mental territory, an inner Overlook where the ghosts of the image, the logic of dreams, and the mechanisms of visual memory unfold once again. The aim is to turn the exploration of the latent space into a mnemonic experience of cinema itself.

Benjamin Bardou

37,491 views • 8 months ago

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Robots Digest 🤖

22,933 views • 5 months ago

How can we generate high-fidelity, complex 3D scenes? Quan Meng's LT3SD decomposes 3D scenes into latent tree representations, with diffusion on the latent trees enabling seamless infinite 3D scene synthesis! w/ Lei Li, Matthias Niessner

How can we generate high-fidelity, complex 3D scenes? Quan Meng's LT3SD decomposes 3D scenes into latent tree representations, with diffusion on the latent trees enabling seamless infinite 3D scene synthesis! w/ Lei Li, Matthias Niessner

Angela Dai

29,422 views • 1 year ago

We've been obsessed with the 3D genome lately. Our preprint will tell you why. We mapped 3D topology, layer by layer, across Development and Injury, guess what?! Neurons retain latent cellular memory that triggers rewiring of the 3D genome upon injury!!!

We've been obsessed with the 3D genome lately. Our preprint will tell you why. We mapped 3D topology, layer by layer, across Development and Injury, guess what?! Neurons retain latent cellular memory that triggers rewiring of the 3D genome upon injury!!!

Ishwariya Venkatesh

10,985 views • 3 months ago

Honcho has fully determined the Pareto Frontier in Agent Memory. It’s state-of-the-art on LoCoMo, LongMem, and BEAM. Honcho is also the fastest, cheapest and most token efficient memory solution.

Honcho has fully determined the Pareto Frontier in Agent Memory. It’s state-of-the-art on LoCoMo, LongMem, and BEAM. Honcho is also the fastest, cheapest and most token efficient memory solution.

Honcho

64,438 views • 6 months ago

Introducing Latent-X — our all-atom frontier AI model for protein binder design. State-of-the-art lab performance, widely accessible via the Latent Labs Platform. Free tier: Blog: Technical report:

Introducing Latent-X — our all-atom frontier AI model for protein binder design. State-of-the-art lab performance, widely accessible via the Latent Labs Platform. Free tier: Blog: Technical report:

Simon Kohl

56,880 views • 11 months ago

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy

Varun

187,489 views • 11 months ago

What if robots could think longer on harder problems without saying a single word?🤔 We introduce RD-VLA (Recurrent-Depth VLA): a latent, iterative reasoning architecture for robot control. ❌No Chain-of-Thought tokens. ❌No extra memory overhead. ✅Just reasoning—directly in latent space. 🧠🤖 Project page: 👇🧵

What if robots could think longer on harder problems without saying a single word?🤔 We introduce RD-VLA (Recurrent-Depth VLA): a latent, iterative reasoning architecture for robot control. ❌No Chain-of-Thought tokens. ❌No extra memory overhead. ✅Just reasoning—directly in latent space. 🧠🤖 Project page: 👇🧵

Jiafei Duan

13,833 views • 5 months ago

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:

AK

718,760 views • 3 years ago

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

Tom Yeh

238,166 views • 2 years ago

Excited to announce 🚀gsplat v1.0🚀: a ⏩efficient⏩ CUDA backend for 3D Gaussian Splatting! A drop-in replacement of the official impl. with: - Up to 2x faster training; - Up to 4x less GPU memory; - Render millions of GSs in real-time; - And more;

Excited to announce 🚀gsplat v1.0🚀: a ⏩efficient⏩ CUDA backend for 3D Gaussian Splatting! A drop-in replacement of the official impl. with: - Up to 2x faster training; - Up to 4x less GPU memory; - Render millions of GSs in real-time; - And more;

Ruilong Li

87,982 views • 2 years ago

"MeshFlow: Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformer" TL;DR: learns a continuous mesh latent space and generates vertices and connectivity in parallel with flow matching, producing quality 3D meshes up to 18× faster than autoregressive.

"MeshFlow: Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformer" TL;DR: learns a continuous mesh latent space and generates vertices and connectivity in parallel with flow matching, producing quality 3D meshes up to 18× faster than autoregressive.

Alexandre Morgand

43,016 views • 1 month ago

We are partnering up with zauth to secure the next generation of 3D AI agents. These agents have memory, emotion, voice, and on-chain identity and can be embedded anywhere. These agents use the x402 protocol to make real payments for skills and services. To make this ecosystem production-ready and safe, is teaming up with zauth, the security layer for the agentic internet. Bringing safety and trust to the next wave of 3D AI agents 🔐

We are partnering up with zauth to secure the next generation of 3D AI agents. These agents have memory, emotion, voice, and on-chain identity and can be embedded anywhere. These agents use the x402 protocol to make real payments for skills and services. To make this ecosystem production-ready and safe, is teaming up with zauth, the security layer for the agentic internet. Bringing safety and trust to the next wave of 3D AI agents 🔐

three.ws

50,602 views • 1 month ago

This week Jules Urbach joined CASEY and @permissionless to discuss how DePIN networks like The Render Network can leverage latent consumer GPU compute for next generation AI inference. Watch the full #Permissionless2024 panel here:

This week Jules Urbach joined CASEY and @permissionless to discuss how DePIN networks like The Render Network can leverage latent consumer GPU compute for next generation AI inference. Watch the full #Permissionless2024 panel here:

The Render Network

96,653 views • 1 year ago

Ubisoft has shared a video dedicated to the design of the first cutscene between Basim and Nimlot in the Valley of Memory DLC of Assassin’s Creed Mirage!

Ubisoft has shared a video dedicated to the design of the first cutscene between Basim and Nimlot in the Valley of Memory DLC of Assassin’s Creed Mirage!

Access The Animus

11,999 views • 7 months ago

James is putting Antigravity to the test on the Commodore 64. By handling device-specific memory mapping and raw code generation, Antigravity is making 8-bit game dev faster (and cooler) than ever. 🕹️

James is putting Antigravity to the test on the Commodore 64. By handling device-specific memory mapping and raw code generation, Antigravity is making 8-bit game dev faster (and cooler) than ever. 🕹️

Google Antigravity

51,853 views • 4 months ago

New work with Yide Shentu! We present Latent Code as Bridges🌉 or LCB. We leverage LLMs for robotics by finetuning the LLM to output latent embeddings to control the lower level policy. This hierarchical approach enables end2end training while maintaining the advantages of LLMs!

New work with Yide Shentu! We present Latent Code as Bridges🌉 or LCB. We leverage LLMs for robotics by finetuning the LLM to output latent embeddings to control the lower level policy. This hierarchical approach enables end2end training while maintaining the advantages of LLMs!

Philipp Wu

21,078 views • 2 years ago

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

AK

32,840 views • 2 years ago