Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Microsoft Research introduces Mirage Latent spatial memory stores 3D scenes directly as latent tokens, skipping the costly RGB render-and-reencode loop. The result is up to 10.57x faster video generation, 55x lower memory use, and state-of-the-art consistency on WorldScore.

DailyPapers

21,705 subscribers

18,760 Aufrufe • vor 4 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

4 Kommentare

Profilbild von DailyPapers

DailyPapersvor 4 Monaten

Paper: Project: Code:

Profilbild von BoldHyun

BoldHyunvor 4 Monaten

Mirage stores 3D scenes as latent tokens, skipping the costly RGB render-and-reencode loop. 10.57x faster generation, 55x lower memory, state-of-the-art consistency on WorldScore. #MicrosoftResearch

Profilbild von Michał Piszczek

Michał Piszczekvor 4 Monaten

xxrender-reencode was always infrastructure tax. 55x memory reduction is why this shipsxrender-reencode was always infrastructure tax. 55x memory reduction is why this ships.render-reencode was always infrastructure tax. 55x memory reduction is why this ships.

Profilbild von Abhishek Verma

Abhishek Vermavor 4 Monaten

@_akhaliq This is amazing

Ähnliche Videos

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.

Introducing WorldCrafter: a world model with persistent memory—not rigid 3D geometry, not just latent memory, but a “soft skeleton” of 3D-aware latents. The result: striking consistency as you explore, look away, and return—even in dynamic scenes.

Ying Shan

27,642 Aufrufe • vor 20 Tagen

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:

HunyuanWorld-Voyager is here and fully open-source! The world’s first ultra-long-range world model with native 3D reconstruction, redefining AI-driven spatial intelligence for VR, gaming, and simulations. ✅Direct 3D Output: Exports point cloud videos to 3D formats without tools like COLMAP, enabling instant 3D application use. ✅Innovative 3D Memory: Introduces a scalable world caching mechanism, ensuring geometric consistency across any camera trajectory. ✅Top-Ranked Performance: #1 on Stanford’s WorldScore, excelling in video generation and 3D reconstruction benchmarks.( Built on HunyuanWorld 1.0, Voyager blends video generation with 3D modeling, delivering camera-controlled, high-fidelity RGB-D sequences. Control scenes via keyboard or joystick for unmatched 3D consistency. Explore now: 🌐Project Page: 🔗GitHub: 🤗HuggingFace: 📝Technical Details:

Tencent Hy

198,391 Aufrufe • vor 1 Jahr

Tencent ARC releases GAE: a geometry-native latent space for 3D-consistent world generation GAE compresses 3,072-channel geometry features into a compact 128-channel latent that decodes into RGB, depth, camera trajectories, and point clouds from the same generated state. It halves camera-trajectory error and cuts FVD by up to 23% on standard benchmarks.

Tencent ARC releases GAE: a geometry-native latent space for 3D-consistent world generation GAE compresses 3,072-channel geometry features into a compact 128-channel latent that decodes into RGB, depth, camera trajectories, and point clouds from the same generated state. It halves camera-trajectory error and cuts FVD by up to 23% on standard benchmarks.

DailyPapers

25,488 Aufrufe • vor 19 Tagen

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

New research with Tsinghua University: Spatial-TTT. A framework for streaming visual-based spatial intelligence with test-time training (TTT). Spatial-TTT adapts fast weights to capture and organize spatial evidence from long video streams, enabling models to build structured 3D spatial memory over time. Highlights: 🔹Efficient streaming memory. Fast weights act as compact spatial memory with sublinear memory growth over 7000+ frames and more than 40% lower compute. 🔹Spatial-predictive mechanism. TTT layers with 3D spatiotemporal convolution capture geometric correspondence and temporal continuity. 🔹SOTA results on long-horizon video spatial understanding (VSI-Bench). The paper ranked #1 on Hugging Face Daily Papers on March 13. Project page: GitHub: Paper: Model & Data:

Tencent Hy

20,792 Aufrufe • vor 7 Monaten

Shadows of The Shining Exploring the latent space of The Shining is to approach the film as one approaches a memory. Here, artificial intelligence acts as a second consciousness: it does not reproduce — it remembers. It reconstructs The Shining from its traces, its echoes, from what has survived the narrative. The latent space becomes a mental territory, an inner Overlook where the ghosts of the image, the logic of dreams, and the mechanisms of visual memory unfold once again. The aim is to turn the exploration of the latent space into a mnemonic experience of cinema itself.

Shadows of The Shining Exploring the latent space of The Shining is to approach the film as one approaches a memory. Here, artificial intelligence acts as a second consciousness: it does not reproduce — it remembers. It reconstructs The Shining from its traces, its echoes, from what has survived the narrative. The latent space becomes a mental territory, an inner Overlook where the ghosts of the image, the logic of dreams, and the mechanisms of visual memory unfold once again. The aim is to turn the exploration of the latent space into a mnemonic experience of cinema itself.

Benjamin Bardou

37,536 Aufrufe • vor 11 Monaten

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Can 1,000 real-world 3D scenes make a state-of-the-art model better? At OVER Research, we tested it by fine-tuning R3, a state-of-the-art streaming 3D reconstruction model, on roughly 1,000 scenes from the OVER dataset. The result: 25% lower trajectory error on our real-world environments compared to the deployed public checkpoint. And we did it using only 1,000 scenes, the same open dataset anyone can download for free, out of the 285,000+ real-world 3D scenes OVER has today. Read the full OVER Research publication: Download the 1,000-scene dataset on Hugging Face:

Over the Reality 🌐

34,785 Aufrufe • vor 1 Monat

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Robots Digest 🤖

22,933 Aufrufe • vor 8 Monaten

Honcho has fully determined the Pareto Frontier in Agent Memory. It’s state-of-the-art on LoCoMo, LongMem, and BEAM. Honcho is also the fastest, cheapest and most token efficient memory solution.

Honcho has fully determined the Pareto Frontier in Agent Memory. It’s state-of-the-art on LoCoMo, LongMem, and BEAM. Honcho is also the fastest, cheapest and most token efficient memory solution.

Honcho

66,350 Aufrufe • vor 9 Monaten

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy

Here’s a demo of our Agentic Memory system, inspired by how our own brain holds information in a 3D spatial space. This feels, natural. Extending this further, we announced the Agentic Memory Protocol on July 15th in SF - which enables memory to be local, encrypted and available to other agents and apps based on your permission-only. We believe this is the future of memory - not owned by any one app, spatial, and always improving. Aaron Levie Andrej Karpathy

Varun

187,489 Aufrufe • vor 1 Jahr

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:

AK

718,760 Aufrufe • vor 3 Jahren

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

MICROSOFT QUIETLY BUILT ONE MEMORY MODULE THAT PLUGS INTO ANY AI AGENT AND CUTS ITS CONTEXT BY UP TO 100X no retraining. no task-specific redesign. you just plug it in Microsoft's insight: agents don't need to remember what happened. they need the knowledge pulled from it - the facts and the reusable skills, not the raw logs so PlugMem stops storing history. it compiles experience into a knowledge graph, then feeds the agent only what matters the result: > up to 1-2 orders of magnitude fewer tokens > highest decision-utility per token of any system tested > one unchanged module beats task-specific memory on chat QA, multi-hop search, and web agents more memory was never the answer. better memory is - and Microsoft just shipped it as a plug-in bookmark this. the agents that win won't remember more, they'll remember smarter.

NO1ennn

227,683 Aufrufe • vor 2 Monaten

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation. Co-led w/ Zhengyang Geng 🧵

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation. Co-led w/ Zhengyang Geng 🧵

Mihir Prabhudesai

82,974 Aufrufe • vor 2 Monaten

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

SORA by Hand ✍️ OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie. How does DiT work? 𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps [1] Given ↳ Video ↳ Prompt: "sora is sky" ↳ Diffusion step: t = 3 [2] Video → Patches ↳ Divide all pixels in all frames into 4 spacetime patches [3] Visual Encoder: Pixels 🟨 → Latent 🟩 ↳ Multiply the patches with weights and biases, followed by ReLU ↳ The result is a latent feature vector per patch ↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1). ↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4) [4] ⬛ Add Noise ↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise. ↳ Add the Sampled Noise to latent features to obtain Noised Latent. ↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is. ↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was. [5-7] 🟪 Conditioning by Adaptive Layer Norm [5] Encode Conditions ↳ Encode "sora is sky" into a text embedding vector [0,1,-1]. ↳ Encode t = 3 to as a binary vector [1,1]. ↳ Concatenate the two vectors in to a 5D column vector. [6] Estimate Scale/Shift ↳ Multiply the combined vector with weights and biases ↳ The goal is to estimate the scale [2,-1] and shift [-1,5]. ↳ Copy the result to (X) and (+) [7] Apply Scale/Sift ↳ Scale the noised latent by [2,-1] ↳ Shifted the scaled noised latent by [-1, 5] ↳ The result is "conditioned" noise latent. [8-10] Transformer [8] Self-Attention ↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix ↳ Value is omitted for simplicity [9] Attention Pooling ↳ Multiply the conditioned noised latent with the self-attention matrix ↳ The result are attention weighted features [10] Pointwise Feed Forward Network ↳ Multiply the attention weighted features with weights and biases ↳ The result is the Predicted Noise 🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻 [11] ↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth). ↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders) ↳ Note the visual encoder and decoder's parameters are frozen (blue borders) 🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲) [12] Denoise ↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent [13] Visual Decoder: Latent 🟩 → Pixels 🟨 ↳ Multiply the patches with weights and biases, followed by ReLU [14] Patches → Video ↳ Rearrange patches into a sequence of video frames.

Tom Yeh

238,485 Aufrufe • vor 2 Jahren

Excited to announce 🚀gsplat v1.0🚀: a ⏩efficient⏩ CUDA backend for 3D Gaussian Splatting! A drop-in replacement of the official impl. with: - Up to 2x faster training; - Up to 4x less GPU memory; - Render millions of GSs in real-time; - And more;

Excited to announce 🚀gsplat v1.0🚀: a ⏩efficient⏩ CUDA backend for 3D Gaussian Splatting! A drop-in replacement of the official impl. with: - Up to 2x faster training; - Up to 4x less GPU memory; - Render millions of GSs in real-time; - And more;

Ruilong Li

88,201 Aufrufe • vor 2 Jahren

Microsoft ran its graph system against vector RAG on 8k, 120k, and a full million token context window, and the million token giant lost Jonathan Larson from Microsoft Research put up the numbers → LazyGraphRAG won 92, 90, and 91 percent of data-local questions, the exact place plain RAG was supposed to be strong → and did it at a tenth of the cost of the million token run the lesson buried in the benchmark → a bigger context window is not memory, structured memory is throwing more tokens at the problem was just the expensive way to be wrong open source on GitHub, the lesson a $1500 course would charge you for

Microsoft ran its graph system against vector RAG on 8k, 120k, and a full million token context window, and the million token giant lost Jonathan Larson from Microsoft Research put up the numbers → LazyGraphRAG won 92, 90, and 91 percent of data-local questions, the exact place plain RAG was supposed to be strong → and did it at a tenth of the cost of the million token run the lesson buried in the benchmark → a bigger context window is not memory, structured memory is throwing more tokens at the problem was just the expensive way to be wrong open source on GitHub, the lesson a $1500 course would charge you for

NO1ennn

29,189 Aufrufe • vor 2 Monaten

We are partnering up with zauth to secure the next generation of 3D AI agents. These agents have memory, emotion, voice, and on-chain identity and can be embedded anywhere. These agents use the x402 protocol to make real payments for skills and services. To make this ecosystem production-ready and safe, is teaming up with zauth, the security layer for the agentic internet. Bringing safety and trust to the next wave of 3D AI agents 🔐

We are partnering up with zauth to secure the next generation of 3D AI agents. These agents have memory, emotion, voice, and on-chain identity and can be embedded anywhere. These agents use the x402 protocol to make real payments for skills and services. To make this ecosystem production-ready and safe, is teaming up with zauth, the security layer for the agentic internet. Bringing safety and trust to the next wave of 3D AI agents 🔐

three.ws

51,086 Aufrufe • vor 4 Monaten

James is putting Antigravity to the test on the Commodore 64. By handling device-specific memory mapping and raw code generation, Antigravity is making 8-bit game dev faster (and cooler) than ever. 🕹️

James is putting Antigravity to the test on the Commodore 64. By handling device-specific memory mapping and raw code generation, Antigravity is making 8-bit game dev faster (and cooler) than ever. 🕹️

Google Antigravity

52,027 Aufrufe • vor 7 Monaten

We are getting 2 new devices from NVIDIA and Microsoft: 1. The DGX Station, with GB300 superchip and up to 748GB of memory. 2. The RTX Spark laptop, with 1 petaflop of AI performance and 128GB of unified memory. The hardware here is pretty bonkers! Think about it this way:

We are getting 2 new devices from NVIDIA and Microsoft: 1. The DGX Station, with GB300 superchip and up to 748GB of memory. 2. The RTX Spark laptop, with 1 petaflop of AI performance and 128GB of unified memory. The hardware here is pretty bonkers! Think about it this way:

Santiago

15,307 Aufrufe • vor 4 Monaten

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

AK

32,849 Aufrufe • vor 2 Jahren

Meta announces Movie Gen A Cast of Media Foundation Models We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models

Meta announces Movie Gen A Cast of Media Foundation Models We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models

AK

62,719 Aufrufe • vor 2 Jahren