Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Can we apply diffusion directly on MLP weights? Yes!!! We show generalization to new 3D shapes and 4D mesh animations: "HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion" #ICCV2023! Project: Video:

MattNiessner's profile picture

Matthias Niessner

49,215 subscribers

86,084 görüntüleme • 3 yıl önce •via X (Twitter)

#ICCV2023
LIVE

Anya Rossi• Live Now

Private livecam show

6 Yorum

Matthias Niessner profil fotoğrafı
Matthias Niessner3 yıl önce

Quite interesting that neural fields can be directly used as representations for generative models. One open question is what MLP weight structure works best for generalization. Super cool work by @ErkocZiya in collaboration with @angelaqdai, @fangchangma, and Qi Shan.

Michael Black profil fotoğrafı
Michael Black3 yıl önce

@CasualEffects Cool idea.

Efstratios Gavves profil fotoğrafı
Efstratios Gavves3 yıl önce

We had a similar idea in our brainstorming sessions, curious to read in detail. Sounds excellent!!

Ruoshi Liu profil fotoğrafı
Ruoshi Liu3 yıl önce

Very cool work! How does it compare with MeshDiffusion whose geometric representation is more explicit?

田中義弘 | taziku CEO / AI × Creative profil fotoğrafı
田中義弘 | taziku CEO / AI × Creative3 yıl önce

Potential technologies. Thanks for the info!

Tianyu He profil fotoğrafı
Tianyu He3 yıl önce

One question: why not generate triplane representation? since it more straightforward for diffusion models.

Benzer Videolar

Excited to share Norman Müller's DiffRF: Rendering-guided 3D Radiance Field Diffusion #CVPR2023 highlight! 2D diffusion is great, but what about 3D? We show radiance field diffusion with rendering guidance for consistent and editable 3D synthesis. Vid:
0:27

Excited to share Norman Müller's DiffRF: Rendering-guided 3D Radiance Field Diffusion #CVPR2023 highlight! 2D diffusion is great, but what about 3D? We show radiance field diffusion with rendering guidance for consistent and editable 3D synthesis. Vid:

Matthias Niessner

118,156 görüntüleme • 3 yıl önce

ActionMesh Animated 3D Mesh Generation with Temporal 3D Diffusion
0:48

ActionMesh Animated 3D Mesh Generation with Temporal 3D Diffusion

AK

12,347 görüntüleme • 9 ay önce

DimensionX Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
1:49

DimensionX Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion

AK

58,273 görüntüleme • 1 yıl önce

SCoPE: We found a surprisingly simple way to encode the 3D camera directly into video diffusion — and it works remarkably well.
2:00

SCoPE: We found a surprisingly simple way to encode the 3D camera directly into video diffusion — and it works remarkably well.

Ying Shan

13,402 görüntüleme • 2 ay önce

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.
0:30

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.

Matthias Niessner

16,835 görüntüleme • 2 yıl önce

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.
0:34

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,769 görüntüleme • 2 yıl önce

Nvidia just announced Align Your Gaussians Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
0:22

Nvidia just announced Align Your Gaussians Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

AK

131,358 görüntüleme • 2 yıl önce

3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models. SIGGRAPH 2023. Zhang (Biao Zhang ) Tang (Jiapeng Tang) Niessner (Matthias Niessner) Wonka.
0:49

3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models. SIGGRAPH 2023. Zhang (Biao Zhang ) Tang (Jiapeng Tang) Niessner (Matthias Niessner) Wonka.

Peter Wonka

11,765 görüntüleme • 3 yıl önce

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.
0:23

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

AK

117,091 görüntüleme • 2 yıl önce

Slowly getting somewhere with generating images (soon 3D scenes?) with Stable Diffusion directly inside Unity with local inference. no API, no subscription, no python #madewithunity #unitysentis
1:22

Slowly getting somewhere with generating images (soon 3D scenes?) with Stable Diffusion directly inside Unity with local inference. no API, no subscription, no python #madewithunity #unitysentis

Julien Kipp

12,366 görüntüleme • 2 yıl önce

"MeshFlow: Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformer" TL;DR: learns a continuous mesh latent space and generates vertices and connectivity in parallel with flow matching, producing quality 3D meshes up to 18× faster than autoregressive.
0:29

"MeshFlow: Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformer" TL;DR: learns a continuous mesh latent space and generates vertices and connectivity in parallel with flow matching, producing quality 3D meshes up to 18× faster than autoregressive.

Alexandre Morgand

43,375 görüntüleme • 4 ay önce

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:
0:24

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

AK

304,708 görüntüleme • 3 yıl önce

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:
0:46

I'm excited to share our new work Diffusion as Shader (DaS), a versatile controllable video generation method for various tasks: object manipulation, camera control, mesh-to-video, and motion transfer. Project page: Github:

Yuan Liu

33,490 görüntüleme • 1 yıl önce

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.
0:43

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 görüntüleme • 3 yıl önce

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency
0:21

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 görüntüleme • 2 yıl önce

InsertAnywhere Bridging 4D Scene Geometry and Diffusion Models for Realistic Video Object Insertion
0:21

InsertAnywhere Bridging 4D Scene Geometry and Diffusion Models for Realistic Video Object Insertion

AK

26,123 görüntüleme • 10 ay önce

Google Deepmind presents CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
2:12

Google Deepmind presents CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models

Chubby♨️

12,843 görüntüleme • 1 yıl önce

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model abs: project page:
2:16

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model abs: project page:

AK

90,089 görüntüleme • 3 yıl önce