Introducing DI-PCG: A diffusion-based Image2PCG framework that automates Procedural... Content Generation, and our first step towards neural symbolic 3D generation. The demo showcases six object categories and can be extended to support customized PCGs. Links in comments. CC: Wang Zhaoshow more

DimensionX: Create Any 3D and 4D Scenes from a... Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.show more

Unreal Engine 5.8 preview is live, introducing Mesh Terrain,... updates to Procedural Content Generation, the Procedural Vegetation Editor, MetaHumans, and much more. Download: Video by Dylan Browneshow more

NVIDIA AI Released DiffusionRenderer: An AI Model for Editable,... Photorealistic 3D Scenes from a Single Video In a groundbreaking new paper, researchers at NVIDIA, University of Toronto, Vector Institute and the University of Illinois Urbana-Champaign have unveiled a framework that directly tackles this challenge. DiffusionRenderer represents a revolutionary leap forward, moving beyond mere generation to offer a unified solution for understanding and manipulating 3D scenes from a single video. It effectively bridges the gap between generation and editing, unlocking the true creative potential of AI-driven content. DiffusionRenderer treats the “what” (the scene’s properties) and the “how” (the rendering) in one unified framework built on the same powerful video diffusion architecture that underpins models like Stable Video Diffusion..... Read full article here: Paper: GitHub Page: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA AIDevshow more

Excited to share our new work "AvatarPopUp"! "Instant 3D... Human Avatar Generation using Image Diffusion Models" 💻 📄show more

Introducing Inversion, our family of structured LLMs. Our first... generation models excel in structured tasks, offering unmatched speed, latency, reliability, and efficiency, with the most comprehensive typed JSON output support available anywhere.show more

🔥Spatial intelligence requires world generation, and now we have... the first comprehensive evaluation benchmark📏 for it! Introducing WorldScore: Unifying evaluation for 3D, 4D, and video models on world generation! 🧵1/7 Web: arxiv:show more

Introducing the world's highest railroad, thanks to procedural generation... not working out exactly like I'd hoped😄# #bugsHappenshow more

🔥Spatial intelligence needs fast, *interactive* 3D world generation 🎮... — introducing WonderWorld: generating 3D scenes interactively following your movement and content requests, and see them in <10 seconds! 🧵1/6 Web: arXiv:show more

Wonderland: Navigating 3D Scenes from a Single Image Contributions:... • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.show more

LLaDA (the first Large Language Diffusion Model) is *just*... out 💥 and I've built a demo, try out now 👨‍💻 It's mesmerizing to watch the diffusion process 🌀, and it being a diffusion model gives you superpowers like "the 4th word has to be pineapple" 🦸 Demo and weights 👇show more

My first PhD paper!🎉We learn *diffusion* models for code... generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/nshow more

📢DNF: Generating 4D animations with dictionary-based neural fields! Xinyi... Zhang presents a new dictionary-based neural field for unconditional 4D generation of deforming shapes -- generating motions with high-quality shape and temporal consistency.show more

Game generation → diffusion output, end-to-end. From prompt to... scene: a moon-like environment created in our engine and reskinned into flexible media assets. Built on our web app.show more

Real-time generation isn’t about predicting the next pixel, it... requires understanding the structure and context of a 3D world. That’s the shift. Read more about our real-time game-native diffusion model on our blog. Moonlake's Reverie v0.1.0show more

Chop the gradients ✂️! We found that truncating decoder... gradients in latent video diffusion to a fixed window allows us to finetune on videos with pixel-wise perceptual losses without running out of memory. Pixel losses have been essential for image generation and reconstruction, but until now, they haven't scaled to long-duration, high-resolution video diffusion due to recursive activation accumulation in causal decoders, leading to OOM during training 💥📉. Project: Video diffusion models can do a lot more 🚀 when you can backprop the decoder! Post-process neural rendered scenes, super-resolve videos, harmonize lighting in controlled synthetic driving scenes, and inpaint videos — all in a single step ⚡ with a quick finetune from a standard diffusion model.show more

The architecture of thought. GPT-2's neural pathways visualized in... topological 3D space. Language generation as pure mathematical movement.show more

New Generation Model! 🚨 We're introducing the Mistral model... to our expanding lineup of generation models. Mistral brings efficient performance and strong language understanding capabilities to our platform. Initial testing shows promising results in code comprehension and generation tasks, making it a valuable addition to development workflows. While we continue to optimize its implementation, early benchmarks demonstrate consistent and reliable outputs across various programming tasks.show more

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the... rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7show more