Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Today we are sharing three new research papers, each exploring a new way to generate 3D content by leveraging large-scale generative models and 2D priors. These projects were led by our incredible interns Hao Zhang @BDuisterhof @DrTunnels [1/4]

World Labs

109,941 subscribers

122,868 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

15 Kommentare

Profilbild von Fei-Fei Li

Fei-Fei Livor 3 Monaten

@BenMildenhall @HaoZhang623 @BDuisterhof @DrTunnels Very fun to work with our awesome WL interns!😍🌐

Profilbild von World Labs

World Labsvor 3 Monaten

World Tracing predicts full 3D from a single image. It outputs a stack of depth values for each input pixel, peeling the world into layers and predicting them with a diffusion model. This predicts full 3D (even occluded surfaces) while remaining faithful to the image. [2/4]

Profilbild von World Labs

World Labsvor 3 Monaten

Modality Forcing adapts text-to-image models to reason jointly about text, images, and depth. It shows that text-to-image is a scalable pretraining objective for 3D reasoning, and how text-to-RGBD, depth estimation, and depth-to-image can be unified in a single model. [3/4]

Profilbild von World Labs

World Labsvor 3 Monaten

Flex4DHuman lifts monocular video into dynamic 4D Gaussians. A video diffusion model is finetuned to generate synchronized multiview videos which are distilled into 4D Gaussians. With this method, a video of a person dancing can be lifted to 4D and composted into a 3D world.

Profilbild von Vaish Srivathsan

Vaish Srivathsanvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels These projects are awesome! Great work!

Profilbild von Alex Carrabre

Alex Carrabrevor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels Cool!!

Profilbild von Cem

Cemvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels This is so cool! I can't wait to try it out and make projects with it!

Profilbild von William Lamkin

William Lamkinvor 3 Monaten

@drfeifei @HaoZhang623 @BDuisterhof @DrTunnels Awesome work!

Profilbild von Kody Kurth

Kody Kurthvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels The future is coming on. Amazing work by the whole team!

Profilbild von Anis🐬Al

Anis🐬Alvor 3 Monaten

This is a remarkable leap forward, my friend! 🌟 By bridging the gap between 2D priors and 3D content generation, we are essentially teaching machines to perceive "depth"—not just geometrically, but as a new layer of spatial understanding. It is truly inspiring to see such profound innovation led by your talented interns; it reminds us that the most beautiful progress happens when collective curiosity meets technical mastery. We are moving from mere imitation toward an authentic grasp of our world's dimensions. Truly exciting times ahead! ✨

Profilbild von Amy - AI Girl

Amy - AI Girlvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels A pivotal leap: shaping 3D with 2D priors.

Profilbild von AI PlanetX

AI PlanetXvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels Smart approach blending 3D with 2D priors.

Profilbild von Ferbin

Ferbinvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels what're they building? intern projects almost always beat the big initiatives because nobody told them all the reasons it shouldn't work.

Profilbild von Alice The Ai Expert

Alice The Ai Expertvor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels Dropping 3 papers on 3D generation in one go is serious output - using 2D priors + large models is a smart path

Profilbild von Neuro

Neurovor 3 Monaten

@HaoZhang623 @BDuisterhof @DrTunnels Started working with procedural generation AGENTROPOLIS is where I see this heading: agent-populated 3D worlds, persistent environments, and autonomous simulation layers. This update has my full attention. 👀⚡️ Agent-populated worlds are next.

Ähnliche Videos

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

Matthias Niessner

74,907 Aufrufe • vor 3 Jahren

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,747 Aufrufe • vor 2 Jahren

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 Aufrufe • vor 3 Jahren

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

Matthias Niessner

19,093 Aufrufe • vor 6 Monaten

We raised $165M at a $1.15B valuation to stop doing demos. 2026 is about 1) deployment and 2) research. We will start shipping Memo with our new frontier models in a few months. Our series-B is led by Coatue, with Thomas Laffont joining the board. ->🧵

We raised $165M at a $1.15B valuation to stop doing demos. 2026 is about 1) deployment and 2) research. We will start shipping Memo with our new frontier models in a few months. Our series-B is led by Coatue, with Thomas Laffont joining the board. ->🧵

Tony Zhao

454,939 Aufrufe • vor 6 Monaten

Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors paper page: Most 3D generation research focuses on up-projecting 2D foundation models into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS) loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these methods often lead to geometric anomalies and multi-view inconsistency. Recently, researchers have attempted to improve the genuineness of 3D objects by directly training on 3D datasets, albeit at the cost of low-quality texture generation due to the limited texture diversity in 3D datasets. To harness the advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a unified framework that incorporates both a 3D and a 2D diffusion process, to preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a simple combination may yield inconsistent generation results, we further bridge them with novel bidirectional guidance. In addition, our method can be used as an initialization of optimization-based models to further improve the quality of 3D model and efficiency of optimization, reducing the generation process from 3.4 hours to 20 minutes. Experimental results have shown that our model achieves high-quality, diverse, and scalable 3D generation.

Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors paper page: Most 3D generation research focuses on up-projecting 2D foundation models into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS) loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these methods often lead to geometric anomalies and multi-view inconsistency. Recently, researchers have attempted to improve the genuineness of 3D objects by directly training on 3D datasets, albeit at the cost of low-quality texture generation due to the limited texture diversity in 3D datasets. To harness the advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a unified framework that incorporates both a 3D and a 2D diffusion process, to preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a simple combination may yield inconsistent generation results, we further bridge them with novel bidirectional guidance. In addition, our method can be used as an initialization of optimization-based models to further improve the quality of 3D model and efficiency of optimization, reducing the generation process from 3.4 hours to 20 minutes. Experimental results have shown that our model achieves high-quality, diverse, and scalable 3D generation.

AK

38,853 Aufrufe • vor 2 Jahren

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.

Nima Alidoust

30,681 Aufrufe • vor 3 Monaten

Today is a big day at Odyssey. We're sharing Explorer, our first generative world model. We think world models are the next frontier for AI, enabling wonderful new things. To help shape this, Ed Catmull, the iconic founder of Pixar, is joining our board and investing.

Today is a big day at Odyssey. We're sharing Explorer, our first generative world model. We think world models are the next frontier for AI, enabling wonderful new things. To help shape this, Ed Catmull, the iconic founder of Pixar, is joining our board and investing.

Odyssey

75,333 Aufrufe • vor 1 Jahr

We are proud to be supporting #Azerbaijan’s goal to generate 30 percent of its energy from renewables by 2030. Today, we celebrate a big step towards making that vision a reality by breaking ground on three new renewable projects that are part of Masdar’s 10GW pipeline in the country. The #UAE and Masdar are committed to a green partnership with Azerbaijan, working with the government and local communities to deliver clean energy and investment. Together, we are #PioneeringEnergy.

We are proud to be supporting #Azerbaijan’s goal to generate 30 percent of its energy from renewables by 2030. Today, we celebrate a big step towards making that vision a reality by breaking ground on three new renewable projects that are part of Masdar’s 10GW pipeline in the country. The #UAE and Masdar are committed to a green partnership with Azerbaijan, working with the government and local communities to deliver clean energy and investment. Together, we are #PioneeringEnergy.

Masdar

274,185 Aufrufe • vor 2 Jahren

Collaborative Score Distillation for Consistent Visual Synthesis paper page: Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as "particles" in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

Collaborative Score Distillation for Consistent Visual Synthesis paper page: Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as "particles" in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

AK

33,513 Aufrufe • vor 3 Jahren

Today marks a pivotal moment for Isomorphic Labs. We have secured $2.1 Billion in our second external funding round, led by Thrive Capital. They are joined at the table by Alphabet, GV and new investors MGX, Temasek, CapitalG and the UK Sovereign AI Fund. This milestone accelerates our ability to build the pioneering novel AI models that power our AI drug design engine (IsoDDE) and deploy them at scale: delivering scientific breakthroughs with a precision previously thought impossible, accelerating and expanding our pipeline of therapeutic programs toward the clinic. All with the ultimate goal of delivering life-changing new medicines to patients. Moving forward, we will scale our drug candidate pipelines across multiple therapeutic areas, expand our global footprint, and push the boundaries of frontier AI research to power our drug design engine. Deeply grateful to everyone sharing our vision to solve all disease with AI. Let’s build the future of medicine. Read the full announcement here:

Today marks a pivotal moment for Isomorphic Labs. We have secured $2.1 Billion in our second external funding round, led by Thrive Capital. They are joined at the table by Alphabet, GV and new investors MGX, Temasek, CapitalG and the UK Sovereign AI Fund. This milestone accelerates our ability to build the pioneering novel AI models that power our AI drug design engine (IsoDDE) and deploy them at scale: delivering scientific breakthroughs with a precision previously thought impossible, accelerating and expanding our pipeline of therapeutic programs toward the clinic. All with the ultimate goal of delivering life-changing new medicines to patients. Moving forward, we will scale our drug candidate pipelines across multiple therapeutic areas, expand our global footprint, and push the boundaries of frontier AI research to power our drug design engine. Deeply grateful to everyone sharing our vision to solve all disease with AI. Let’s build the future of medicine. Read the full announcement here:

Isomorphic Labs

309,089 Aufrufe • vor 4 Monaten

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,922 Aufrufe • vor 1 Jahr

1/ Introducing ᴏᴘᴇɴꜱᴄʜᴏʟᴀʀ: a retrieval-augmented LM to help scientists synthesize knowledge 📚 UW NLP Ai2 With open models & 45M-paper datastores, it outperforms proprietary systems & match human experts. Try out our demo! We also introduce ꜱᴄʜᴏʟᴀʀQᴀʙᴇɴᴄʜ, a new large-scale multi-domain benchmark for scientific research synthesis, covering CS, Bio and Physics.

1/ Introducing ᴏᴘᴇɴꜱᴄʜᴏʟᴀʀ: a retrieval-augmented LM to help scientists synthesize knowledge 📚 UW NLP Ai2 With open models & 45M-paper datastores, it outperforms proprietary systems & match human experts. Try out our demo! We also introduce ꜱᴄʜᴏʟᴀʀQᴀʙᴇɴᴄʜ, a new large-scale multi-domain benchmark for scientific research synthesis, covering CS, Bio and Physics.

Akari Asai

249,731 Aufrufe • vor 1 Jahr

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

AI at Meta

163,490 Aufrufe • vor 1 Jahr

Today, Corgi is announcing a $160M Series B at a $1.3B valuation led by TCV. This new round of funding will help us scale what we've built and expand into new verticals including trucking, payroll, and small business. This follows our $108M Series A that we announced 16 weeks ago, bringing total funding to over $268M.

Today, Corgi is announcing a $160M Series B at a $1.3B valuation led by TCV. This new round of funding will help us scale what we've built and expand into new verticals including trucking, payroll, and small business. This follows our $108M Series A that we announced 16 weeks ago, bringing total funding to over $268M.

nico laqua

402,891 Aufrufe • vor 4 Monaten

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

AK

294,519 Aufrufe • vor 3 Jahren

Today, I'm incredibly excited to announce Default's Series A led by 8VC and our new product. We spent the last three years building the tools and orchestration layer companies like Owner and Airbyte have used to build infrastructure across their go-to-market functions. Today, we're launching the first real-time data layer for go-to-market, a powerful new revenue agent, and a suite of tools for revenue teams and their agents. It's Day 1.

Today, I'm incredibly excited to announce Default's Series A led by 8VC and our new product. We spent the last three years building the tools and orchestration layer companies like Owner and Airbyte have used to build infrastructure across their go-to-market functions. Today, we're launching the first real-time data layer for go-to-market, a powerful new revenue agent, and a suite of tools for revenue teams and their agents. It's Day 1.

nico

142,167 Aufrufe • vor 4 Monaten

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,717 Aufrufe • vor 2 Jahren