Loading video...

Video Failed to Load

Go Home

#SIGGRAPH2024 🏆Best Paper Honorable Mentioned Introducing the magic behind #Rodin ( CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (1/n) CLAY is 3D #stablediffusion + #ControlNet + #LoRA.

DeemosTech's profile picture

Deemos

10,655 subscribers

30,855 views • 2 years ago •via X (Twitter)

Science & TechnologyEducationArts#SIGGRAPH2024#Rodin#stablediffusion#ControlNet#LoRA
LIVE

Anya Rossi• Live Now

Private livecam show

10 Comments

Deemos's profile picture
Deemos2 years ago

(2/n) The core of CLAY is the Large-scale Generative Mode, which is up to 1.5B paramaters. This model was trained on #3D Native Data using a Diffusion Transformer (DiT), just like #Sora.

Deemos's profile picture
Deemos2 years ago

(3/n) CLAY foundation model is designed for Text-to-3D. 🔧However, we designed ControlNet&LoRA for CLAY, allowing controls across various modalities like image, voxel, multi-view images, point cloud, and bounding box, enabling efficient fine-tuning for specific styles.

Deemos's profile picture
Deemos2 years ago

(4/n) We even made a #Blender Plugin for it. Coming Soon at

Deemos's profile picture
Deemos2 years ago

(5/n) 📷We also introduce a multiview material diffusion. After generating the geometry, we process it into quad-faces. Then utilize material diffusion model with advanced back-projection techniques to achieve rapid and efficient creation of high-quality PBR textures.

Deemos's profile picture
Deemos2 years ago

(6/n) For more details, please refer to our paper. 📑Paper: 📦Project: 🔮Product: 💻Open-Clay: 🪀Demo @Gradio :

Martin Nebelong's profile picture
Martin Nebelong2 years ago

Congrats, well deserved!

Deemos's profile picture
Deemos2 years ago

Thanks Martin!

Risichad 🦾's profile picture
Risichad 🦾2 years ago

WAW !

PDS_B2BMGMT's profile picture
PDS_B2BMGMT2 years ago

You have to sign-up to a website to get download.

sanchay's profile picture
sanchay2 years ago

Woah , congratulations! This is so damn cool

Related Videos

#SIGGRAPH2024 🚀So glad to announce that the foundation model behind #3D GenAI #Rodin Gen-1, CLAY has been officially accepted by ACM SIGGRAPH Technical Paper Journal Track. 🔥The final paper is still being prepared, but here's a sneak peek of its Technical Video❤️
0:45

#SIGGRAPH2024 🚀So glad to announce that the foundation model behind #3D GenAI #Rodin Gen-1, CLAY has been officially accepted by ACM SIGGRAPH Technical Paper Journal Track. 🔥The final paper is still being prepared, but here's a sneak peek of its Technical Video❤️

Hyper3D by Deemos

22,132 views • 2 years ago

Current 3D generative models are slow and low quality. We present GRM, a large-scale model that reconstructs 3D Gaussians in 0.1s and generates high-quality 3D assets from text or single images in a few seconds. Demo: 1/4
0:37

Current 3D generative models are slow and low quality. We present GRM, a large-scale model that reconstructs 3D Gaussians in 0.1s and generates high-quality 3D assets from text or single images in a few seconds. Demo: 1/4

Gordon Wetzstein

19,189 views • 2 years ago

1/ 🚀 Introducing AIDO.StructureDiffusion: A generative model for structural protein design—enabling high-quality, controllable generation of monomers, complexes, and antibodies. 🧵
0:33

1/ 🚀 Introducing AIDO.StructureDiffusion: A generative model for structural protein design—enabling high-quality, controllable generation of monomers, complexes, and antibodies. 🧵

GenBio AI

918,215 views • 11 months ago

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:
1:22

Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion abs: project page:

AK

262,963 views • 3 years ago

Breaking news 🚨: A Brand New AI Service on OctaSpace ( yes another one ☝️ ) Hunyuan3D 2.0 Advanced large-scale 3D synthesis system for generating high-resolution textured 3D assets using AI. Generate high quality 3D assets with textures which you can then import into 3D tools.
0:44

Breaking news 🚨: A Brand New AI Service on OctaSpace ( yes another one ☝️ ) Hunyuan3D 2.0 Advanced large-scale 3D synthesis system for generating high-resolution textured 3D assets using AI. Generate high quality 3D assets with textures which you can then import into 3D tools.

OctaSpace

15,495 views • 1 year ago

3D ControlNets for #Rodin Gen-1 is now open for early access🚀 DM us if you already have a deep dive on Rodin and want more🎁 #ControlNet #3D #CG
0:50

3D ControlNets for #Rodin Gen-1 is now open for early access🚀 DM us if you already have a deep dive on Rodin and want more🎁 #ControlNet #3D #CG

Deemos

15,626 views • 1 year ago

Introducing the Mint 3D asset pack generator. Generate 20 high quality 3D assets from one prompt!
0:56

Introducing the Mint 3D asset pack generator. Generate 20 high quality 3D assets from one prompt!

tamrat

369,508 views • 11 days ago

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.
0:36

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.

Dreaming Tulpa 🥓👑

85,115 views • 1 year ago

🚀 Rodin Gen-1 is now open trials at 🔥Powered by 1.5B #3D Native Large Model, #Rodin creates assets with quad-faces and PBR in 40s. 🔮Rodin is closer than ever to being a production-ready 3D #GenAI tool. We believe it will bridge community to AI era.
0:39

🚀 Rodin Gen-1 is now open trials at 🔥Powered by 1.5B #3D Native Large Model, #Rodin creates assets with quad-faces and PBR in 40s. 🔮Rodin is closer than ever to being a production-ready 3D #GenAI tool. We believe it will bridge community to AI era.

Hyper3D by Deemos

206,091 views • 2 years ago

Step1X-3D is out on Hugging Face Towards High-Fidelity and Controllable Generation of Textured 3D Assets
0:45

Step1X-3D is out on Hugging Face Towards High-Fidelity and Controllable Generation of Textured 3D Assets

AK

79,559 views • 1 year ago

Text-to-3D with Blocks📦 Striving towards real-time creation on #Rodin Gen-1 with multiple 3D ControlNets! 🚀 Rodin Gen-1 will be the largest ever 3D large model, and we are planing to open it to public🎉 Stay tuned!
1:22

Text-to-3D with Blocks📦 Striving towards real-time creation on #Rodin Gen-1 with multiple 3D ControlNets! 🚀 Rodin Gen-1 will be the largest ever 3D large model, and we are planing to open it to public🎉 Stay tuned!

Hyper3D by Deemos

41,255 views • 2 years ago

Personalized 3D Generative Avatars from a Single Portrait Contributions: 1. Generate a personalized 3D avatar from a reference portrait image with controllable facial attributes. 2. Create high-quality synthetic 2D video datasets with diverse attribute editing from a reference portrait image. 3. Use latent space regularization with face morphing supervision for a continuous and smooth latent space, enhancing the generative ability for unseen or interpolated attribute appearances. 4. Employ an efficient fine-tuning technique via Low-Rank Adaptation (LoRA) [26] to integrate any new facial attribute into the avatar model.
0:45

Personalized 3D Generative Avatars from a Single Portrait Contributions: 1. Generate a personalized 3D avatar from a reference portrait image with controllable facial attributes. 2. Create high-quality synthetic 2D video datasets with diverse attribute editing from a reference portrait image. 3. Use latent space regularization with face morphing supervision for a continuous and smooth latent space, enhancing the generative ability for unseen or interpolated attribute appearances. 4. Employ an efficient fine-tuning technique via Low-Rank Adaptation (LoRA) [26] to integrate any new facial attribute into the avatar model.

MrNeRF

20,507 views • 1 year ago

🔥Text-to-3D Foundation Model🔥 We are excited to announce #3DTopia, a generalist 🧊text-to-3D🧊 foundation model, which produces ** high-quality 3D assets within 5 minutes ** - Code: - Video:
0:58

🔥Text-to-3D Foundation Model🔥 We are excited to announce #3DTopia, a generalist 🧊text-to-3D🧊 foundation model, which produces ** high-quality 3D assets within 5 minutes ** - Code: - Video:

Ziwei Liu

62,424 views • 2 years ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:
0:50

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,417 views • 9 months ago

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.
1:53

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.

AK

25,120 views • 1 year ago

Introducing FLARE #CVPR2026 2025 FLARE is a feed-forward model that simultaneously estimates high-quality camera poses, 3D geometry, and appearance from sparse uncalibrated images. 1/4
0:29

Introducing FLARE #CVPR2026 2025 FLARE is a feed-forward model that simultaneously estimates high-quality camera poses, 3D geometry, and appearance from sparse uncalibrated images. 1/4

Gordon Wetzstein

29,518 views • 1 year ago

Accurate and controllable scene generation has been difficult with natural language alone. You instead need a language for scenes. Introducing the Scene Language — a visual representation for high-quality 3D/4D generation by integrating programs, words, and embeddings — 🧵(1/6)
0:31

Accurate and controllable scene generation has been difficult with natural language alone. You instead need a language for scenes. Introducing the Scene Language — a visual representation for high-quality 3D/4D generation by integrating programs, words, and embeddings — 🧵(1/6)

Yunzhi Zhang

63,813 views • 1 year ago

Here’s how Jim Morren created the disappearing clay effect of the Cinematic Trailer, for our Master 3D Characters in Blender course. Full trailer here: #3d #3dsculpting #clay #3dsculpture #3dart #blender3d
1:00

Here’s how Jim Morren created the disappearing clay effect of the Cinematic Trailer, for our Master 3D Characters in Blender course. Full trailer here: #3d #3dsculpting #clay #3dsculpture #3dart #blender3d

CG Boost

41,630 views • 1 year ago

Nvidia presents Edify 3D! The method can generate high-quality 3D assets from text descriptions. It uses a diffusion model to create detailed quad-mesh topologies and high-resolution textures in under 2 minutes.
0:43

Nvidia presents Edify 3D! The method can generate high-quality 3D assets from text descriptions. It uses a diffusion model to create detailed quad-mesh topologies and high-resolution textures in under 2 minutes.

Dreaming Tulpa 🥓👑

39,517 views • 1 year ago

We’re also publishing two research papers on the approaches we used for high quality 3D model generation & texture generation based on text prompts that are behind the Meta 3D Gen system. Meta 3D AssetGen ➡️ Meta 3D TextureGen ➡️
0:37

We’re also publishing two research papers on the approaches we used for high quality 3D model generation & texture generation based on text prompts that are behind the Meta 3D Gen system. Meta 3D AssetGen ➡️ Meta 3D TextureGen ➡️

AI at Meta

72,914 views • 2 years ago