Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing DI-PCG: A diffusion-based Image2PCG framework that automates Procedural Content Generation, and our first step towards neural symbolic 3D generation. The demo showcases six object categories and can be extended to support customized PCGs. Links in comments. CC: Wang Zhao

Ying Shan

2,366 subscribers

12,420 views • 1 year ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,017 views • 1 year ago

Unreal Engine 5.8 preview is live, introducing Mesh Terrain, updates to Procedural Content Generation, the Procedural Vegetation Editor, MetaHumans, and much more. Download: Video by Dylan Browne

Unreal Engine 5.8 preview is live, introducing Mesh Terrain, updates to Procedural Content Generation, the Procedural Vegetation Editor, MetaHumans, and much more. Download: Video by Dylan Browne

80 LEVEL

19,226 views • 29 days ago

NVIDIA AI Released DiffusionRenderer: An AI Model for Editable, Photorealistic 3D Scenes from a Single Video In a groundbreaking new paper, researchers at NVIDIA, University of Toronto, Vector Institute and the University of Illinois Urbana-Champaign have unveiled a framework that directly tackles this challenge. DiffusionRenderer represents a revolutionary leap forward, moving beyond mere generation to offer a unified solution for understanding and manipulating 3D scenes from a single video. It effectively bridges the gap between generation and editing, unlocking the true creative potential of AI-driven content. DiffusionRenderer treats the “what” (the scene’s properties) and the “how” (the rendering) in one unified framework built on the same powerful video diffusion architecture that underpins models like Stable Video Diffusion..... Read full article here: Paper: GitHub Page: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA AIDev

NVIDIA AI Released DiffusionRenderer: An AI Model for Editable, Photorealistic 3D Scenes from a Single Video In a groundbreaking new paper, researchers at NVIDIA, University of Toronto, Vector Institute and the University of Illinois Urbana-Champaign have unveiled a framework that directly tackles this challenge. DiffusionRenderer represents a revolutionary leap forward, moving beyond mere generation to offer a unified solution for understanding and manipulating 3D scenes from a single video. It effectively bridges the gap between generation and editing, unlocking the true creative potential of AI-driven content. DiffusionRenderer treats the “what” (the scene’s properties) and the “how” (the rendering) in one unified framework built on the same powerful video diffusion architecture that underpins models like Stable Video Diffusion..... Read full article here: Paper: GitHub Page: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA AIDev

Marktechpost AI Dev News ⚡

104,741 views • 11 months ago

Excited to share our new work "AvatarPopUp"! "Instant 3D Human Avatar Generation using Image Diffusion Models" 💻 📄

Nikos Kolotouros

14,081 views • 2 years ago

Introducing Inversion, our family of structured LLMs. Our first generation models excel in structured tasks, offering unmatched speed, latency, reliability, and efficiency, with the most comprehensive typed JSON output support available anywhere.

Introducing Inversion, our family of structured LLMs. Our first generation models excel in structured tasks, offering unmatched speed, latency, reliability, and efficiency, with the most comprehensive typed JSON output support available anywhere.

John

212,921 views • 2 years ago

🔥Spatial intelligence requires world generation, and now we have the first comprehensive evaluation benchmark📏 for it! Introducing WorldScore: Unifying evaluation for 3D, 4D, and video models on world generation! 🧵1/7 Web: arxiv:

🔥Spatial intelligence requires world generation, and now we have the first comprehensive evaluation benchmark📏 for it! Introducing WorldScore: Unifying evaluation for 3D, 4D, and video models on world generation! 🧵1/7 Web: arxiv:

Hong-Xing (Koven) Yu

66,615 views • 1 year ago

Introducing the world's highest railroad, thanks to procedural generation not working out exactly like I'd hoped😄# #bugsHappen

Introducing the world's highest railroad, thanks to procedural generation not working out exactly like I'd hoped😄# #bugsHappen

Protopop Games

45,371 views • 5 months ago

🔥Spatial intelligence needs fast, *interactive* 3D world generation 🎮 — introducing WonderWorld: generating 3D scenes interactively following your movement and content requests, and see them in <10 seconds! 🧵1/6 Web: arXiv:

🔥Spatial intelligence needs fast, interactive 3D world generation 🎮 — introducing WonderWorld: generating 3D scenes interactively following your movement and content requests, and see them in <10 seconds! 🧵1/6 Web: arXiv:

Hong-Xing (Koven) Yu

169,078 views • 1 year ago

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 views • 1 year ago

LLaDA (the first Large Language Diffusion Model) is *just* out 💥 and I've built a demo, try out now 👨‍💻 It's mesmerizing to watch the diffusion process 🌀, and it being a diffusion model gives you superpowers like "the 4th word has to be pineapple" 🦸 Demo and weights 👇

LLaDA (the first Large Language Diffusion Model) is just out 💥 and I've built a demo, try out now 👨‍💻 It's mesmerizing to watch the diffusion process 🌀, and it being a diffusion model gives you superpowers like "the 4th word has to be pineapple" 🦸 Demo and weights 👇

apolinario (poli)

82,599 views • 1 year ago

My first PhD paper!🎉We learn *diffusion* models for code generation that learn to directly *edit* syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n

My first PhD paper!🎉We learn diffusion models for code generation that learn to directly edit syntax trees of programs. The result is a system that can incrementally write code, see the execution output, and debug it. 🧵1/n

Shreyas Kapur

742,386 views • 2 years ago

📢DNF: Generating 4D animations with dictionary-based neural fields! Xinyi Zhang presents a new dictionary-based neural field for unconditional 4D generation of deforming shapes -- generating motions with high-quality shape and temporal consistency.

📢DNF: Generating 4D animations with dictionary-based neural fields! Xinyi Zhang presents a new dictionary-based neural field for unconditional 4D generation of deforming shapes -- generating motions with high-quality shape and temporal consistency.

Angela Dai

14,637 views • 1 year ago

Game generation → diffusion output, end-to-end. From prompt to scene: a moon-like environment created in our engine and reskinned into flexible media assets. Built on our web app.

Game generation → diffusion output, end-to-end. From prompt to scene: a moon-like environment created in our engine and reskinned into flexible media assets. Built on our web app.

Moonlake

20,014 views • 4 months ago

Real-time generation isn’t about predicting the next pixel, it requires understanding the structure and context of a 3D world. That’s the shift. Read more about our real-time game-native diffusion model on our blog. Moonlake's Reverie v0.1.0

Real-time generation isn’t about predicting the next pixel, it requires understanding the structure and context of a 3D world. That’s the shift. Read more about our real-time game-native diffusion model on our blog. Moonlake's Reverie v0.1.0

Moonlake

14,439 views • 5 months ago

Chop the gradients ✂️! We found that truncating decoder gradients in latent video diffusion to a fixed window allows us to finetune on videos with pixel-wise perceptual losses without running out of memory. Pixel losses have been essential for image generation and reconstruction, but until now, they haven't scaled to long-duration, high-resolution video diffusion due to recursive activation accumulation in causal decoders, leading to OOM during training 💥📉. Project: Video diffusion models can do a lot more 🚀 when you can backprop the decoder! Post-process neural rendered scenes, super-resolve videos, harmonize lighting in controlled synthetic driving scenes, and inpaint videos — all in a single step ⚡ with a quick finetune from a standard diffusion model.

Chop the gradients ✂️! We found that truncating decoder gradients in latent video diffusion to a fixed window allows us to finetune on videos with pixel-wise perceptual losses without running out of memory. Pixel losses have been essential for image generation and reconstruction, but until now, they haven't scaled to long-duration, high-resolution video diffusion due to recursive activation accumulation in causal decoders, leading to OOM during training 💥📉. Project: Video diffusion models can do a lot more 🚀 when you can backprop the decoder! Post-process neural rendered scenes, super-resolve videos, harmonize lighting in controlled synthetic driving scenes, and inpaint videos — all in a single step ⚡ with a quick finetune from a standard diffusion model.

Felix Heide

28,282 views • 1 month ago

The architecture of thought. GPT-2's neural pathways visualized in topological 3D space. Language generation as pure mathematical movement.

The architecture of thought. GPT-2's neural pathways visualized in topological 3D space. Language generation as pure mathematical movement.

michielh.eth

21,966 views • 1 year ago

New Generation Model! 🚨 We're introducing the Mistral model to our expanding lineup of generation models. Mistral brings efficient performance and strong language understanding capabilities to our platform. Initial testing shows promising results in code comprehension and generation tasks, making it a valuable addition to development workflows. While we continue to optimize its implementation, early benchmarks demonstrate consistent and reliable outputs across various programming tasks.

New Generation Model! 🚨 We're introducing the Mistral model to our expanding lineup of generation models. Mistral brings efficient performance and strong language understanding capabilities to our platform. Initial testing shows promising results in code comprehension and generation tasks, making it a valuable addition to development workflows. While we continue to optimize its implementation, early benchmarks demonstrate consistent and reliable outputs across various programming tasks.

ALCHEMIST AI 🔮

16,378 views • 1 year ago

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7

🚨Announcing our #ICLR2025 Oral! 🔥Diffusion LMs are on the rise for parallel text generation! But unlike autoregressive LMs, they struggle with quality, fixed-length constraints & lack of KV caching. 🚀Introducing Block Diffusion—combining autoregressive and diffusion models for the best of both worlds! 👇1/7

Marianne Arriola

148,147 views • 1 year ago