正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Accurate and controllable scene generation has been difficult with natural language alone. You instead need a language for scenes. Introducing the Scene Language — a visual representation for high-quality 3D/4D generation by integrating programs, words, and embeddings — 🧵(1/6)

Yunzhi Zhang

2,626 subscribers

63,956 次观看 • 1 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

9 条评论

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

The Scene Language uses programs, words, and neural embeddings to encode scene structures, semantics, and visual identities, respectively. It can be inferred using pre-trained LMs/VLMs to generate scenes from text and image prompts. (2/6)

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

The representation also applies to 4D scenes—many dynamic effects are simple to write in programs! Some text-to-4D synthesis results here: (3/6)

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

When given image prompts, the pipeline converts input images into 3D scenes while preserving the structure and content: (4/6)

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

The representation is not tied to one specific renderer; instead, it can be consumed by renderers ranging from end-to-end, neural generative models to traditional graphics engines. (5/6)

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

More results on project page: Paper: This is a very fun collaboration with the wonderful @zizhang_li, @Mattzh1314, @elliottszwu, and @jiajunwu_cs. (6/6)

Blockchainer K 的头像

Blockchainer K1 年前

I love how Scene Language combines the power of programs, words, and embeddings to create stunning 3D/4D visuals. Can't wait to explore its possibilities for urban planning and smart cities!

Jos van der Westhuizen 的头像

Jos van der Westhuizen1 年前

The results look crazy good! Can the semantic components be treated as separate 3D objects in something like Unity? Would love to get access to an API or something to integrate this into my tool.

Yunzhi Zhang 的头像

Yunzhi Zhang1 年前

Yes, semantic components are separate and can be individually imported as meshes. We'll release the code in November. Stay tuned! :)

IC4 的头像

IC41 年前

That's why we have different courts for sailors.

相关视频

Introducing ClickDiffusion! We developed a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface.

Introducing ClickDiffusion! We developed a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface.

Alec Helbling

36,302 次观看 • 2 年前

1/ 🚀 Introducing AIDO.StructureDiffusion: A generative model for structural protein design—enabling high-quality, controllable generation of monomers, complexes, and antibodies. 🧵

1/ 🚀 Introducing AIDO.StructureDiffusion: A generative model for structural protein design—enabling high-quality, controllable generation of monomers, complexes, and antibodies. 🧵

GenBio AI

918,237 次观看 • 1 年前

Meet Gemini Omni Flash on SeaArt. A natively multimodal model for video generation and conversational editing. Edit with natural language. Keep scenes and story consistent across edits. Motion follows real-world physics and logic. From prompt to scene — in one flow.

Meet Gemini Omni Flash on SeaArt. A natively multimodal model for video generation and conversational editing. Edit with natural language. Keep scenes and story consistent across edits. Motion follows real-world physics and logic. From prompt to scene — in one flow.

SeaArt.Ai🐋

30,455 次观看 • 23 天前

SceneScript treats 3D reconstruction as a language problem rather than a geometry one. The model watches a video of a room and just learns to write a script for it. It autoregressively spits out text commands like make_wall(...) or make_bbox(...) that define the scene. Stanford's new "Scene Language" paper goes a step further adding CLIP embeddings to capture visual appearance too. The fact that language models already understand spatial relationships well enough to write out scene graphs is pretty wild.

SceneScript treats 3D reconstruction as a language problem rather than a geometry one. The model watches a video of a room and just learns to write a script for it. It autoregressively spits out text commands like make_wall(...) or make_bbox(...) that define the scene. Stanford's new "Scene Language" paper goes a step further adding CLIP embeddings to capture visual appearance too. The fact that language models already understand spatial relationships well enough to write out scene graphs is pretty wild.

Bilawal Sidhu

107,060 次观看 • 1 年前

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

LLaVA-3D A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D-awareness for 3D scene understanding has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we employ a simple yet effective representation, 3D Patch, which connects 2D CLIP patch features with their corresponding positions in 3D space. By integrating the 3D Patches into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D image understanding and 3D scene understanding. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D image understanding and vision-language conversation capabilities with LLaVA.

AK

41,736 次观看 • 1 年前

Speed and quality can finally coexist in diffusion-based language generation. Introducing DiDi-Instruct, a Discrete Diffusion Divergence Instruct method that distills a pre-trained discrete diffusion language model (dLLM) into a few-step student for ultra-fast generation. Built on integral KL-divergence minimization, DiDi-Instruct achieves up to 64× faster decoding, surpasses both its teacher and GPT-2, and cuts training time by 20×. Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct Paper: Code: Project: Our report: 📬 #PapersAccepted by Jiqizhixin

Speed and quality can finally coexist in diffusion-based language generation. Introducing DiDi-Instruct, a Discrete Diffusion Divergence Instruct method that distills a pre-trained discrete diffusion language model (dLLM) into a few-step student for ultra-fast generation. Built on integral KL-divergence minimization, DiDi-Instruct achieves up to 64× faster decoding, surpasses both its teacher and GPT-2, and cuts training time by 20×. Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct Paper: Code: Project: Our report: 📬 #PapersAccepted by Jiqizhixin

机器之心 JIQIZHIXIN

18,174 次观看 • 9 个月前

LEGO-SLAM: Language-Embedded Gaussian Optimization SLAM LEGO-SLAM running at 15 FPS on a ScanNet scene with language-based loop closing for drift correction. LEGO-SLAM is a 3DGS-based SLAM framework that supports open-vocabulary semantic querying and rendering. It tracks via G-ICP and efficiently builds a map by embedding Gaussians with scene-adaptive 16D language features. Map management is achieved through Language Pruning and Language-Based Loop Detection. The generated map enables open-vocabulary 3D Object Localization.

LEGO-SLAM: Language-Embedded Gaussian Optimization SLAM LEGO-SLAM running at 15 FPS on a ScanNet scene with language-based loop closing for drift correction. LEGO-SLAM is a 3DGS-based SLAM framework that supports open-vocabulary semantic querying and rendering. It tracks via G-ICP and efficiently builds a map by embedding Gaussians with scene-adaptive 16D language features. Map management is achieved through Language Pruning and Language-Based Loop Detection. The generated map enables open-vocabulary 3D Object Localization.

Ryohei Sasaki@engineer

15,060 次观看 • 4 个月前

Gaussian Shell Maps are a new neural scene representation that connects fields and 3D Gaussians. This representation unlocks the full potential of 3D Gaussian splatting for generative AI applications, such as 3D avatar generation. 1/2

Gordon Wetzstein

52,480 次观看 • 2 年前

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Alex Rives

207,204 次观看 • 1 年前

Got five papers accepted by #ECCV2024 European Conference on Computer Vision #ECCV2026 ! Huge thanks to all my collaborators! 😃 See you in Milan 🇮🇹 Summary of Selected Works (I made a fast-forward for them 😄) - [Shape Generation] Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models, ECCV 2024. - [Efficient Motion Generation] EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, ECCV 2024. - [Controllable Motion Generation] TLControl: Trajectory and Language Control for Human Motion Synthesis, ECCV 2024. - [Avatar Generation] Disentangled Clothed Avatar Generation from Text Descriptions, ECCV 2024. Project Page: Surf-D: EMDM: TLControl: SOSMPL:

Zhiyang (Frank) Dou

18,187 次观看 • 2 年前

“Math is code. And code is math.” Carina Hong CEO Axiom on why that idea matters for AI: “For centuries, mathematicians reasoned in natural language. But because of the Curry–Howard correspondence, a proof can also be a program.” “You can translate natural-language math into formal code using systems like Lean. Lean is both a theorem-proving language and a programming language.” “That means you can generate solutions as code and then formally verify them. You can even use the same system to verify programs themselves.” “This creates a flywheel of generation and verification, where AI can both write solutions and prove that they’re correct.”

“Math is code. And code is math.” Carina Hong CEO Axiom on why that idea matters for AI: “For centuries, mathematicians reasoned in natural language. But because of the Curry–Howard correspondence, a proof can also be a program.” “You can translate natural-language math into formal code using systems like Lean. Lean is both a theorem-proving language and a programming language.” “That means you can generate solutions as code and then formally verify them. You can even use the same system to verify programs themselves.” “This creates a flywheel of generation and verification, where AI can both write solutions and prove that they’re correct.”

Forward Future

24,729 次观看 • 5 个月前

🚀 Excited to announce the first release of a novel open source programming language and platform for language model interaction! Combining prompts, constraints & scripting, LMQL elevates the capabilities of large language models. 🧵1/6 A quick tour.

🚀 Excited to announce the first release of a novel open source programming language and platform for language model interaction! Combining prompts, constraints & scripting, LMQL elevates the capabilities of large language models. 🧵1/6 A quick tour.

LMQL (Language Model Query Language)

198,966 次观看 • 3 年前

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

(1/n) Time to unify your favorite visual generative models, VLMs, and simulators for controllable visual generation—Introducing a Product of Experts (PoE) framework for inference-time knowledge composition from heterogeneous models.

Yunzhi Zhang

49,009 次观看 • 1 年前

🥰Super excited that SceneWeaver ( won the best paper award at the IROS25 RoboGen workshop. SceneWeaver provides an agentic framework for tool-based 3D scene generation, given a language description as input, you can generate or edit a corresponding details with lots of details.

🥰Super excited that SceneWeaver ( won the best paper award at the IROS25 RoboGen workshop. SceneWeaver provides an agentic framework for tool-based 3D scene generation, given a language description as input, you can generate or edit a corresponding details with lots of details.

Siyuan Huang

43,681 次观看 • 9 个月前

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Introducing Modality Forcing, a recipe for post-training T2I models for SOTA RGB-Depth generation! Text-to-image (T2I) models learn rich representations of the spatial world. How do we build on this prior for high-quality depth generation? 🧵 [1/6]

Bardienus Duisterhof

64,875 次观看 • 1 个月前

SuperSplat's voxel collision is a game changer for 3D Gaussian splat creators. Generate a highly accurate, inescapable voxel representation of any splat, regardless of quality. Generation is performed on the GPU, so it's super-fast. The voxel representation is tiny - only 418KB for this scene. And all this magic is free and open source, courtesy of the splat-transform library from PlayCanvas.

SuperSplat's voxel collision is a game changer for 3D Gaussian splat creators. Generate a highly accurate, inescapable voxel representation of any splat, regardless of quality. Generation is performed on the GPU, so it's super-fast. The voxel representation is tiny - only 418KB for this scene. And all this magic is free and open source, courtesy of the splat-transform library from PlayCanvas.

Will Eastcott

12,653 次观看 • 2 个月前

Introducing Lasso: The natural language search engine for blockchain data ✨ Search the chain, get data, and find alpha in your own words -- no SQL or complex smart contract understanding needed! Sign up for early access here -> A 🧵

Introducing Lasso: The natural language search engine for blockchain data ✨ Search the chain, get data, and find alpha in your own words -- no SQL or complex smart contract understanding needed! Sign up for early access here -> A 🧵

Lasso

130,136 次观看 • 3 年前

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,953 次观看 • 1 年前

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 次观看 • 3 年前

Stop by the #ECCV2024 Google Booth at 4:30pm CEST where Googlers Francis Engelmann and Federico Tombari will demo language-guided 3D search ( and SceneFun3D, a new benchmark dataset for functional 3D scene understanding (

Stop by the #ECCV2024 Google Booth at 4:30pm CEST where Googlers Francis Engelmann and Federico Tombari will demo language-guided 3D search ( and SceneFun3D, a new benchmark dataset for functional 3D scene understanding (

Google AI

27,556 次观看 • 1 年前