Loading video...

Video Failed to Load

Go Home

๐ŸŒŸ Create anything in 3D! ๐ŸŒŸ Introducing CAT3D: a new method that generates high-fidelity 3D scenes from any number of real or generated images in one minute, powered by multi-view diffusion models. w/ lovely coauthors Aleksander Holynski, Ben Poole and an amazing team!

153,337 views โ€ข 2 years ago โ€ขvia X (Twitter)

10 Comments

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

Website: Paper: Also, check out our gallery for hundreds of diverse generated 3D scenes!

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

CAT3D uses a multi-view diffusion model to synthesize a large set of consistent novel views from the input image(s). These views are then used to reconstruct a detailed 3D scene using robust reconstruction methods.

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

CAT3D can create entire 3D scenes in as little as one minute! โšก Plus, it achieves state-of-the-art performance for few-view 3D reconstruction.

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

Big shout out to the incredible team: @holynski_ @philipphenzler @ArthurBrussee @rmbrualla @_pratul_ @jon_barron @poolio

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

We are also sincerely thankful to @shlomifruchter, @sirbayes, @babaeizadeh, @Han_Zhang_ and Amir Hertz for training the base text-to-image model that supports our work!

Durk Kingma's profile picture
Durk Kingma2 years ago

@holynski_ @poolio Congrats, great work!

Ruiqi Gao's profile picture
Ruiqi Gao2 years ago

@holynski_ @poolio Thanks Durk! โ˜บ๏ธ

Abubakar Abid's profile picture
Abubakar Abid2 years ago

@holynski_ @poolio So cool! @Gradio demo when ;)

Rundi Wu's profile picture
Rundi Wu2 years ago

@holynski_ @poolio Amazing results!!! Congrats!

Dejia Xu's profile picture
Dejia Xu2 years ago

@holynski_ @poolio Congrats! Amazing results!

Related Videos

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 views โ€ข 3 years ago