Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models. SIGGRAPH 2023. Zhang (Biao Zhang ) Tang (Jiapeng Tang) Niessner (Matthias Niessner) Wonka.

Peter Wonka

1,006 subscribers

11,765 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

1 Comments

Andrea Tagliasacchi3 years ago

@MattNiessner @_BiaoZhang @jiapeng_tang would have loved to see the what, before the how…

Related Videos

Check out 𝐌𝐨𝐭𝐢𝐨𝐧𝟐𝐕𝐞𝐜𝐒𝐞𝐭𝐬, a 4D diffusion model for dynamic surface reconstruction from imperfect observations of sparse, noisy, or partial point clouds. Main idea: we represent time-varying shapes via 4D neural representation with latent vector sets, and then explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of latent vector sets. Great work by Wei Cao るちゃん Biao Zhang Jiapeng Tang

Check out 𝐌𝐨𝐭𝐢𝐨𝐧𝟐𝐕𝐞𝐜𝐒𝐞𝐭𝐬, a 4D diffusion model for dynamic surface reconstruction from imperfect observations of sparse, noisy, or partial point clouds. Main idea: we represent time-varying shapes via 4D neural representation with latent vector sets, and then explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of latent vector sets. Great work by Wei Cao るちゃん Biao Zhang Jiapeng Tang

Matthias Niessner

29,149 views • 2 years ago

How can we generate high-fidelity, complex 3D scenes? Quan Meng's LT3SD decomposes 3D scenes into latent tree representations, with diffusion on the latent trees enabling seamless infinite 3D scene synthesis! w/ Lei Li, Matthias Niessner

How can we generate high-fidelity, complex 3D scenes? Quan Meng's LT3SD decomposes 3D scenes into latent tree representations, with diffusion on the latent trees enabling seamless infinite 3D scene synthesis! w/ Lei Li, Matthias Niessner

Angela Dai

29,469 views • 2 years ago

Gaussian Shell Maps are a new neural scene representation that connects fields and 3D Gaussians. This representation unlocks the full potential of 3D Gaussian splatting for generative AI applications, such as 3D avatar generation. 1/2

Gordon Wetzstein

52,555 views • 2 years ago

zhang boheng & tang chia-hung medal ceremony cut

zhang boheng & tang chia-hung medal ceremony cut

noé 🩵

117,138 views • 2 years ago

Looking for a challenging dataset for novel view synthesis and 3D semantics? Check out ScanNet++ at #ICCV23 (Oral)! 460+ scenes w/ 1mm laser scans, semantics, DSLR images, iPhone RGBD video Chandan Yeshwanth Yueh-Cheng Liu Matthias Niessner

Looking for a challenging dataset for novel view synthesis and 3D semantics? Check out ScanNet++ at #ICCV23 (Oral)! 460+ scenes w/ 1mm laser scans, semantics, DSLR images, iPhone RGBD video Chandan Yeshwanth Yueh-Cheng Liu Matthias Niessner

Angela Dai

10,289 views • 3 years ago

Synthesia (Synthesia 🎥) has raised $200 million in its latest funding round, bringing the company's valuation to $4 billion. Synthesia (Synthesia 🎥), headquartered in London, England, United Kingdom, was founded in 2017 by Victor Riparbelli (Victor Riparbelli), Lourdes Agapito (Lourdes Agapito), Matthias Niessner (Matthias Niessner), and Steffen Tjerrild (Steffen Tjerrild).

Synthesia (Synthesia 🎥) has raised $200 million in its latest funding round, bringing the company's valuation to $4 billion. Synthesia (Synthesia 🎥), headquartered in London, England, United Kingdom, was founded in 2017 by Victor Riparbelli (Victor Riparbelli), Lourdes Agapito (Lourdes Agapito), Matthias Niessner (Matthias Niessner), and Steffen Tjerrild (Steffen Tjerrild).

Silicon Valley Investclub

15,606 views • 11 months ago

We present Neural Riemannian Motion Fields (NRMF) as a promising paradigm for generative modeling of articulated motion. Unlike various diffusion based models, NRMF is useful both in generation and in deployment as a prior across tasks: #CVPR #AI #CVML👇

We present Neural Riemannian Motion Fields (NRMF) as a promising paradigm for generative modeling of articulated motion. Unlike various diffusion based models, NRMF is useful both in generation and in deployment as a prior across tasks: #CVPR #AI #CVML👇

Tolga Birdal

16,795 views • 1 year ago

Excited to announce ScanNet++ v2!🎉 Chandan Yeshwanth and Yueh-Cheng Liu have been working tirelessly to bring: 🔹1006 high-fidelity 3D scans 🔹+ DSLR & iPhone captures 🔹+ rich semantics Elevating 3D scene understanding to the next level!🚀 w/ Matthias Niessner

Excited to announce ScanNet++ v2!🎉 Chandan Yeshwanth and Yueh-Cheng Liu have been working tirelessly to bring: 🔹1006 high-fidelity 3D scans 🔹+ DSLR & iPhone captures 🔹+ rich semantics Elevating 3D scene understanding to the next level!🚀 w/ Matthias Niessner

Angela Dai

47,629 views • 1 year ago

New AAA game from China, "Blood Message." A little background to this true historic event. The story is based in the year 848AD, after the Anshi rebellion, Tang dynasty was weakened and lost control of the Western Region (modern day Gansu and Xinjiang). With a shift in global temperature, the Tibetan empire rose up and occupied the region. After years of struggle, with one Tang city holding out a siege for 11 years. The Tibetan empire practiced a very oppressive form of serfdom and slavery, very much opposed to the ideals of Tang Dynasty of openness (many Tang officials, even a couple of prime ministers were foreigners). the guy giving the speech at 0:50 is Zhang Yichao, who led a local rebellion against the Tibetan empire. To seek help from the Tang Dynasty, Zhang Yichao sent 10 teams of messengers to Tang capital Chang'An modern day Xi'An. To tell the people of Tang, the rebellion of the Western Region and seek reinforcement. Of the 10 messenger teams, only 1 team arrived in Chang'An, the player is in that team. Historically, Tang Emperor gave Zhang Yichao a bunch of titles and all the land in the Western region, but the dynasty was unable to help, but through years of war, Zhang Yichao's rebel army was able to reconnect the Western region with Tang territory.

New AAA game from China, "Blood Message." A little background to this true historic event. The story is based in the year 848AD, after the Anshi rebellion, Tang dynasty was weakened and lost control of the Western Region (modern day Gansu and Xinjiang). With a shift in global temperature, the Tibetan empire rose up and occupied the region. After years of struggle, with one Tang city holding out a siege for 11 years. The Tibetan empire practiced a very oppressive form of serfdom and slavery, very much opposed to the ideals of Tang Dynasty of openness (many Tang officials, even a couple of prime ministers were foreigners). the guy giving the speech at 0:50 is Zhang Yichao, who led a local rebellion against the Tibetan empire. To seek help from the Tang Dynasty, Zhang Yichao sent 10 teams of messengers to Tang capital Chang'An modern day Xi'An. To tell the people of Tang, the rebellion of the Western Region and seek reinforcement. Of the 10 messenger teams, only 1 team arrived in Chang'An, the player is in that team. Historically, Tang Emperor gave Zhang Yichao a bunch of titles and all the land in the Western region, but the dynasty was unable to help, but through years of war, Zhang Yichao's rebel army was able to reconnect the Western region with Tang territory.

Zhao DaShuai 东北进修🇨🇳

72,049 views • 1 year ago

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

AK

304,708 views • 3 years ago

We also present another paper at ACM SIGGRAPH 2023 on neural implicit 3D Morphable Models that can be used to create a dynamic 3D avatar from a single in-the-wild image. (Lead author Connor Lin).

Koki Nagano

12,786 views • 3 years ago

Check out the Neural Graph Mapping for Dense SLAM with Efficient Loop Closure paper by Leonard Bruns, Jun Zhang, and Patric Jensfelt, visualized with Rerun. “Existing neural field-based SLAM methods typically employ a single monolithic field as their scene representation. This prevents efficient incorporation of loop closure constraints and limits scalability. To address these shortcomings, we propose a neural mapping framework which anchors lightweight neural fields to the pose graph of a sparse visual SLAM system.”

Check out the Neural Graph Mapping for Dense SLAM with Efficient Loop Closure paper by Leonard Bruns, Jun Zhang, and Patric Jensfelt, visualized with Rerun. “Existing neural field-based SLAM methods typically employ a single monolithic field as their scene representation. This prevents efficient incorporation of loop closure constraints and limits scalability. To address these shortcomings, we propose a neural mapping framework which anchors lightweight neural fields to the pose graph of a sparse visual SLAM system.”

Rerun

12,865 views • 2 years ago

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.

Matthias Niessner

16,835 views • 2 years ago

📢ProcGen3D: Learning Neural Procedural Graphs for Image-to-3D Reconstruction Xinyi Zhang learns neural procedural graphs to generate high-fidelity 3D - MCTS-guided sampling maintains consistency with the input image, even from real images! Check it out:

📢ProcGen3D: Learning Neural Procedural Graphs for Image-to-3D Reconstruction Xinyi Zhang learns neural procedural graphs to generate high-fidelity 3D - MCTS-guided sampling maintains consistency with the input image, even from real images! Check it out:

Angela Dai

14,702 views • 11 months ago

Can we apply diffusion directly on MLP weights? Yes!!! We show generalization to new 3D shapes and 4D mesh animations: "HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion" #ICCV2023! Project: Video:

Can we apply diffusion directly on MLP weights? Yes!!! We show generalization to new 3D shapes and 4D mesh animations: "HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion" #ICCV2023! Project: Video:

Matthias Niessner

86,084 views • 3 years ago

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:

AK

121,020 views • 3 years ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,918 views • 1 year ago

Today we are sharing three new research papers, each exploring a new way to generate 3D content by leveraging large-scale generative models and 2D priors. These projects were led by our incredible interns Hao Zhang @BDuisterhof @DrTunnels [1/4]

Today we are sharing three new research papers, each exploring a new way to generate 3D content by leveraging large-scale generative models and 2D priors. These projects were led by our incredible interns Hao Zhang @BDuisterhof @DrTunnels [1/4]

World Labs

123,355 views • 4 months ago

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:

AK

263,035 views • 3 years ago

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,769 views • 2 years ago