Загрузка видео...

Не удалось загрузить видео

На главную

Using the new WordPress Command Palette to call an assistant that adds LLM generated text to a 3D world using natural language commands! "add text: write a short poem about the metaverse" This extends to image, audio and 3D objects in the future. WebXR holodeck style editing!

xrpublisher's profile picture

XR Publisher

2,086 subscribers

17,891 просмотров • 3 лет назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

wow.. text to 3D would AI is here you can generate 3D assets using text/image and build up a 3D world.. and even auto rig characters. link in comment
0:20

wow.. text to 3D would AI is here you can generate 3D assets using text/image and build up a 3D world.. and even auto rig characters. link in comment

el.cine

212,681 просмотров • 1 год назад

Tencent presents TIP-Editor An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts paper page: Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIPEditor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.
2:00

Tencent presents TIP-Editor An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts paper page: Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIPEditor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.

AK

56,428 просмотров • 2 лет назад

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.
0:42

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 просмотров • 3 лет назад

Introducing StyleDrop, a model that allows a significantly higher level of stylized text-to-image synthesis by using a few style reference images that describe the style for text-to-image generation, bypassing the burden of text prompt engineering. More→
0:42

Introducing StyleDrop, a model that allows a significantly higher level of stylized text-to-image synthesis by using a few style reference images that describe the style for text-to-image generation, bypassing the burden of text prompt engineering. More→

Google AI

80,379 просмотров • 2 лет назад

I just used Luma's text-to-3D to generate a DeLorean from Back to the Future and placed it in my living room using the Vision Pro! The future is wild. 🤩
1:32

I just used Luma's text-to-3D to generate a DeLorean from Back to the Future and placed it in my living room using the Vision Pro! The future is wild. 🤩

Ammaar Reshi

265,563 просмотров • 2 лет назад

Generate an interactive 3D world ✨ I opened this project in Cursor and uploaded an image of a cozy bookstore. The agent then generated the environment, chose objects to turn into 3D meshes, added colliders, and created sound effects 🤯
0:20

Generate an interactive 3D world ✨ I opened this project in Cursor and uploaded an image of a cozy bookstore. The agent then generated the environment, chose objects to turn into 3D meshes, added colliders, and created sound effects 🤯

Justine Moore

77,484 просмотров • 4 месяцев назад

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.
0:30

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI Breakfast

33,725 просмотров • 3 лет назад

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.
1:11

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

The Humanoid Hub

25,125 просмотров • 9 месяцев назад

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓
1:14

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Google AI

203,357 просмотров • 5 месяцев назад

To generate 3D models in Codex with text-to-cad: Start a new project, write your prompt for a 3D model, and include a link to clone the text-to-cad repo. Codex handles the rest. The repo is open source and all scripts / ui run locally. Link below.
0:21

To generate 3D models in Codex with text-to-cad: Start a new project, write your prompt for a 3D model, and include a link to clone the text-to-cad repo. Codex handles the rest. The repo is open source and all scripts / ui run locally. Link below.

Jake Fitzgerald

22,113 просмотров • 4 месяцев назад

Speedsculpt + AI image gen. What an incredibly powerful combo compared to text to image. Made using Dreams on my PS5, by far the fastest way to sculpt in 3d anywhere. Transformed with Krea ai. Imagine when it's possible to convert this to an actual 3d model, to animate it, and to integrate it into projects. #ThePowerofDreams #kreaai
0:25

Speedsculpt + AI image gen. What an incredibly powerful combo compared to text to image. Made using Dreams on my PS5, by far the fastest way to sculpt in 3d anywhere. Transformed with Krea ai. Imagine when it's possible to convert this to an actual 3d model, to animate it, and to integrate it into projects. #ThePowerofDreams #kreaai

Martin Nebelong

56,601 просмотров • 2 лет назад

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.
0:22

3D editing is hard: you need to ground an image + instruction and generate a faithful 3D shape in one forward pass -- no test-time optimization. So, we steer pretrained image-to-3D representations to do text-guided 3D edits; no massive 3D edit-pair dataset needed. Key trap: the “no-edit” solution is a nasty local minimum. We fix it with preference optimization, pushing the model to actually edit. Steer3D is the second work that adapts alignment ideas from LLMs to the 3D modality. SAM 3D also used DPO to improve its 3D generations.

Georgia Gkioxari

116,203 просмотров • 9 месяцев назад

Using ImGui's font atlas to render text into an RGBA texture, then mapping it to a simple quad for in-game 3D text. Super useful for debugging and prototyping.
0:30

Using ImGui's font atlas to render text into an RGBA texture, then mapping it to a simple quad for in-game 3D text. Super useful for debugging and prototyping.

Ioannis Koukourakis

12,443 просмотров • 8 месяцев назад

You can now use AI to make an interactive 3D world from text prompts 🤯 Watch me use KREA AI's new Scene feature - I created the underground environment, and then prompted to drop in 3D meshes for objects. You can move, resize, or rotate the objects as you please!
0:25

You can now use AI to make an interactive 3D world from text prompts 🤯 Watch me use KREA AI's new Scene feature - I created the underground environment, and then prompted to drop in 3D meshes for objects. You can move, resize, or rotate the objects as you please!

Justine Moore

30,851 просмотров • 1 год назад

We think text-to-image AI is pretty interesting, so here's text-to-BIM! It won’t “create a museum in the style of Zaha Hadid.” (Yet.) But you can describe your building and get an editable 3D model in return. Coming soon from Hypar !
2:02

We think text-to-image AI is pretty interesting, so here's text-to-BIM! It won’t “create a museum in the style of Zaha Hadid.” (Yet.) But you can describe your building and get an editable 3D model in return. Coming soon from Hypar !

Hypar

28,752 просмотров • 3 лет назад