Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🔴 Finally! NVIDIA has finally made the code for Neuralangelo public! It has the ability to transform any video into a highly detailed 3D environment, and it's a technology related to but DIFFERENT from NeRF. 💡 Here's how it works: It takes a 2D video as input, showing an... object, monument, building, landscape, etc., from various perspectives and analyzes details such as depth, size, and the shapes of objects. From this, the AI sketches an initial 3D model, similar to how an artist molds a figure. This representation is then refined to highlight more details, just as an artist would make the final touches when sculpting. The result is a 3D environment/model, perfect for use in any environment. Imagine the applications it will have for video games, cinema, virtual environments, VR, and more! 📽️🎮 💡 More details: A year ago, an article was presented on a groundbreaking technique called NVIDIA's Instant NeRF. This technique turns images into stunning 3D scenes in a short time, ideal for creating realistic models for video games and other applications. Although Instant NeRF had a lot of potential, the generated models were not perfect and often lacked detailed structures, appearing somewhat cartoonish. A year on, NVIDIA releases a new technique based on Instant NeRF, named Neuralangelo. This enhances the fidelity of surface structures. While NeRF reconstructs real objects in virtual environments from images or videos, Instant NeRF speeds up this process, and Neuralangelo further improves the quality, making the generated objects appear even more realistic when examined up close. Neuralangelo improves Instant NeRF's approach in two key ways related to the hash grid encoding technique: 1⃣ Numerical gradients have been used to compute higher-order derivatives as a smoothing operation. This optimizes the "hash grid" encoding using numerical rather than analytical gradients, providing a smoother input to the network that produces the 3D model. 2⃣ A "coarse-to-fine" optimization has been implemented in the hash grids to control different levels of detail. That is, they first focus on a smoothed version of the scene, and then refine it with more detailed updates. Well, as Arthur C. Clarke said, "Any sufficiently advanced technology is indistinguishable from magic."show more

Javi Lopez ⛩️

128,327 subscribers

689,046 views • 2 years ago •via X (Twitter)

News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Javi Lopez ⛩️2 years ago

LINKs: 🔗 Paper: 🔗 Code:

Javi Lopez ⛩️2 years ago

And if it doesn't bore you, the full video. Mine with techno music is more entertaining 😂

Javi Lopez ⛩️2 years ago

And you can join my newsletter for Generative AI free tuts, tools and news!

Javi Lopez ⛩️2 years ago

More AI News!

Anders Hjemdahl2 years ago

Very cool - basically an AI-improved photogrammety technique which cleverly fills in gaps and creates smooth surfaces from gradient algorithms? Looking forward to getting my hands on this!

Javi Lopez ⛩️2 years ago

That's a better summary than the one I did! 😅👌

Ali Hero2 years ago

I think they pretty soon create games with random infinite AI maps

Javi Lopez ⛩️2 years ago

Agree!

john2 years ago

everyday I start to think we’re living in a simulation more and more

🇺🇦 Brad Gashler2 years ago

@0xUnum Non commercial license only 😭

Related Videos

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 views • 3 years ago

Here's a look behind the scenes of how I used a 3d model, generated from an image, as a "driver" for ai animation. In the near future we'll see more 3d tools that support these types of workflows, with much more emphasis on enhancing user input instead of just letting the ai do everything. I used the 3d model to generate keyframes that I then animated in Luma Dream. By using a 3d model with a diffusion "layer" done with Krea, I got quite high quality frames with a relatively high consistency as well since the AI didn't have to "hallucinate" that much. I upscaled and detailed the frames using Magnific. Using semi-detailed 3d models as drivers for gen ai is very powerful, and it allows us to use models that are sculpted in a more free-flowing type of workflow that doesn't rely so much on high surface detailing or very time-consuming finish, but instead relies more on gestural 3d sculpting. #art #ai

Here's a look behind the scenes of how I used a 3d model, generated from an image, as a "driver" for ai animation. In the near future we'll see more 3d tools that support these types of workflows, with much more emphasis on enhancing user input instead of just letting the ai do everything. I used the 3d model to generate keyframes that I then animated in Luma Dream. By using a 3d model with a diffusion "layer" done with Krea, I got quite high quality frames with a relatively high consistency as well since the AI didn't have to "hallucinate" that much. I upscaled and detailed the frames using Magnific. Using semi-detailed 3d models as drivers for gen ai is very powerful, and it allows us to use models that are sculpted in a more free-flowing type of workflow that doesn't rely so much on high surface detailing or very time-consuming finish, but instead relies more on gestural 3d sculpting. #art #ai

Martin Nebelong

48,200 views • 2 years ago

"#AI will never create clean meshes" AI 👇 This is the new #AI model from NVIDIA called Meshtron (jeez what a name 😅). It generates meshes of complex 3D objects closely resembling those created by professional artists. It opens the door to a more realistic generation of detailed 3D assets for animation, gaming, and virtual environments. The current version accepts the following control inputs: - Point cloud: Determines the shape of the output mesh. - Face count: Determines the density of the output mesh. - Quad ratio: Switches between quad and triangle tessellation. - Creativity: Can be adjusted to generate extra details in the model. Meshtron can either serve as a stand-alone remesher to improve the quality of existing models, or be used in tandem with a text-to-3D or image-to-3D model to generate artist-grade meshes from scratch.

"#AI will never create clean meshes" AI 👇 This is the new #AI model from NVIDIA called Meshtron (jeez what a name 😅). It generates meshes of complex 3D objects closely resembling those created by professional artists. It opens the door to a more realistic generation of detailed 3D assets for animation, gaming, and virtual environments. The current version accepts the following control inputs: - Point cloud: Determines the shape of the output mesh. - Face count: Determines the density of the output mesh. - Quad ratio: Switches between quad and triangle tessellation. - Creativity: Can be adjusted to generate extra details in the model. Meshtron can either serve as a stand-alone remesher to improve the quality of existing models, or be used in tandem with a text-to-3D or image-to-3D model to generate artist-grade meshes from scratch.

Gabriele Romagnoli

2,176,749 views • 1 year ago

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic paper page:

AK

46,151 views • 3 years ago

This might be the craziest workflow you'll see this week. From #VR greyboxing to 3D worlds with the help of #AI. The 2 tools I used are ShapesXR and Marble from World Labs. Let me tell you why and how. - ShapesXR is the most powerful and fastest way to assemble 3D scenes at scale. - The library of assets + ability to sketch from procedural primitives gives the perfect balance between speed and freedom. - ShapesXR allows you to export your creation as a glTF that can be used as a "3D prompt" into Marble - Marble now allows you to create a panorama using a 3D scene as input increasing DRAMATICALLY the control you have during the creation process - Through the workflow you can adjust textures and objects in the scene using AI inpainting - There are tons of options on how to use the final output (create a #GaussianSplatting, export collision mesh and even a detailed mesh 🤯) This is the perfect examples of how VR creative tools improve and accelerate the design process... so make sure to follow me if that's your jam 😉.

This might be the craziest workflow you'll see this week. From #VR greyboxing to 3D worlds with the help of #AI. The 2 tools I used are ShapesXR and Marble from World Labs. Let me tell you why and how. - ShapesXR is the most powerful and fastest way to assemble 3D scenes at scale. - The library of assets + ability to sketch from procedural primitives gives the perfect balance between speed and freedom. - ShapesXR allows you to export your creation as a glTF that can be used as a "3D prompt" into Marble - Marble now allows you to create a panorama using a 3D scene as input increasing DRAMATICALLY the control you have during the creation process - Through the workflow you can adjust textures and objects in the scene using AI inpainting - There are tons of options on how to use the final output (create a #GaussianSplatting, export collision mesh and even a detailed mesh 🤯) This is the perfect examples of how VR creative tools improve and accelerate the design process... so make sure to follow me if that's your jam 😉.

Gabriele Romagnoli

20,126 views • 7 months ago

Finally got #gaussiansplatting working! I’m really impressed by how well it captured the fine details of the sneaker fabric and shoelaces, as well as the text. This was generated from 75 DSLR photos I shot on a 24mm lens, took about 30 minutes to process on an RTX 3090, and is running at a smooth 60fps on the SIBR Viewer. Going to run more tests to see how this compares to some of my experiments in photogrammetry. I’m hoping a plugin for #UnrealEngine5 comes out soon! #nerf

Finally got #gaussiansplatting working! I’m really impressed by how well it captured the fine details of the sneaker fabric and shoelaces, as well as the text. This was generated from 75 DSLR photos I shot on a 24mm lens, took about 30 minutes to process on an RTX 3090, and is running at a smooth 60fps on the SIBR Viewer. Going to run more tests to see how this compares to some of my experiments in photogrammetry. I’m hoping a plugin for #UnrealEngine5 comes out soon! #nerf

CoffeeVectors

183,486 views • 2 years ago

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,698 views • 1 year ago

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

@levelsio

119,210 views • 1 year ago

This is a pretty wild model! You can use it to turn an image into a 3D object with texture. The quality is out of this world! I'm not even a designer, and I've been using this nonstop for the last 2 hours. The model is Hunyuan 3D 2.1. It's open source. You'll find model weights, training/inference code, data pipelines, and architecture on their repository. You can even fine-tune it if you want! GitHub Repository: By the way, the model runs on consumer-grade GPUs. You don't need a datacenter for this! I've been using the model from the HuggingFace demo page: To use it, go to the link and upload an image. That's it! Check out the video I recorded for a couple of examples.

This is a pretty wild model! You can use it to turn an image into a 3D object with texture. The quality is out of this world! I'm not even a designer, and I've been using this nonstop for the last 2 hours. The model is Hunyuan 3D 2.1. It's open source. You'll find model weights, training/inference code, data pipelines, and architecture on their repository. You can even fine-tune it if you want! GitHub Repository: By the way, the model runs on consumer-grade GPUs. You don't need a datacenter for this! I've been using the model from the HuggingFace demo page: To use it, go to the link and upload an image. That's it! Check out the video I recorded for a couple of examples.

Santiago

44,783 views • 1 year ago

Introducing Hybrid Models Alchemist AI now allows users to choose one model for ideation and another for completion, making app generation smarter, more structured, and highly detailed. With an ideation model, users can provide a simple prompt, and Alchemist AI does the heavy lifting—expanding on the idea with precise technical details. When creating a game, for example, it defines map size, terrain complexity, physics, AI behavior, and game mechanics before passing it to the completion model, which then builds the final product. This hybrid approach elevates the quality of generated applications, making it easier than ever to create highly polished games, tools, and applications—with minimal effort.

Introducing Hybrid Models Alchemist AI now allows users to choose one model for ideation and another for completion, making app generation smarter, more structured, and highly detailed. With an ideation model, users can provide a simple prompt, and Alchemist AI does the heavy lifting—expanding on the idea with precise technical details. When creating a game, for example, it defines map size, terrain complexity, physics, AI behavior, and game mechanics before passing it to the completion model, which then builds the final product. This hybrid approach elevates the quality of generated applications, making it easier than ever to create highly polished games, tools, and applications—with minimal effort.

ALCHEMIST AI 🔮

17,905 views • 1 year ago

Meta released SAM3 a model to reconstruct objects from segmented images. I tried it and here are 4 things I found out - The generation is incredibly fast when compared with other 3D GenAI models - The models generated are not meshes but Gaussian Splatting. I was surprised that in the paper it was not mentioned anywhere - When generating multiple objects the arrangement is a bit messed up. - Unfortunately, it is impossible to download any of the .ply generated 😔 I cannot stop thinking about the implications of this in XR. Objects in the real world could be almost instantly turned into photorealistic copies and even rearranged to the user's wish.

Meta released SAM3 a model to reconstruct objects from segmented images. I tried it and here are 4 things I found out - The generation is incredibly fast when compared with other 3D GenAI models - The models generated are not meshes but Gaussian Splatting. I was surprised that in the paper it was not mentioned anywhere - When generating multiple objects the arrangement is a bit messed up. - Unfortunately, it is impossible to download any of the .ply generated 😔 I cannot stop thinking about the implications of this in XR. Objects in the real world could be almost instantly turned into photorealistic copies and even rearranged to the user's wish.

Gabriele Romagnoli

32,428 views • 7 months ago

When I saw the mask "Tribes of the Calf" from Kanbas I knew I had to make it into reality. The jewelry and gold really made it stand out for me. Since Sam Spratt's The Masquerade was revealed, I have been spending time sculpting and dissecting the mask to recreate it in 3D as faithfully as possible. I delved into the creation of this mask for many reasons. I love a good challenge and this mask surely was one for me. Creating something in 3D from a 2D image is not easy, and especially when the source has generative nature, some stuff is hard to interpret, but I tried my best to make sure the visual integrity of the mask is as close to the original as possible. Splitting the whole mask into parts, filling the missing pieces so I can build the textures was quite a lot of work. I tried to present the mask in my own style with a slightly different colorway to adapt to the mask itself. Please enjoy this short animation, and turn on sound🔊 This piece is my statement that I am here to stay. That I have a voice that often feels being lost in the void. That I have been creating and posting digital art for over 20 years now and will continue until I'm gone. I have a story to tell and I want to be heard. The space we have here is small, and is shrinking day by day. It doesn't have to be like that. We need to support each other and push ourselves and people here, otherwise we are all doomed. As Kanbas has put in their observation of the mask: "Inspirational. Emotional. Natural." This is what our space can be, and this is me making a statement with this homage. I will share a 4k still below as well as a short video showing the 3D GLB interactive model together with a yt link to the 4k video since compression here is pretty bad.

When I saw the mask "Tribes of the Calf" from Kanbas I knew I had to make it into reality. The jewelry and gold really made it stand out for me. Since Sam Spratt's The Masquerade was revealed, I have been spending time sculpting and dissecting the mask to recreate it in 3D as faithfully as possible. I delved into the creation of this mask for many reasons. I love a good challenge and this mask surely was one for me. Creating something in 3D from a 2D image is not easy, and especially when the source has generative nature, some stuff is hard to interpret, but I tried my best to make sure the visual integrity of the mask is as close to the original as possible. Splitting the whole mask into parts, filling the missing pieces so I can build the textures was quite a lot of work. I tried to present the mask in my own style with a slightly different colorway to adapt to the mask itself. Please enjoy this short animation, and turn on sound🔊 This piece is my statement that I am here to stay. That I have a voice that often feels being lost in the void. That I have been creating and posting digital art for over 20 years now and will continue until I'm gone. I have a story to tell and I want to be heard. The space we have here is small, and is shrinking day by day. It doesn't have to be like that. We need to support each other and push ourselves and people here, otherwise we are all doomed. As Kanbas has put in their observation of the mask: "Inspirational. Emotional. Natural." This is what our space can be, and this is me making a statement with this homage. I will share a 4k still below as well as a short video showing the 3D GLB interactive model together with a yt link to the 4k video since compression here is pretty bad.

shoneec

17,531 views • 1 year ago

WOW. 😳 Apple just quietly won the 3D maps war at WWDC. Gaussian Splatting is coming to Apple Maps Flyover this fall. Apple Maps Flyover covers 300+ cities. Until yesterday, every single one was built on standard drone photogrammetry. The technology captures photos from the air and reconstructs 3D geometry from them. Gaussian Splatting does not reconstruct geometry. It represents the scene as millions of tiny 3D ellipsoids, each one carrying its own color and opacity information based on how light actually behaves in that location. The output is not a mesh model. It is a field of light. When you move through it, it does not crumble at the edges. The detail holds because it was never geometry to begin with. Apple has been hiring for this for years. Their SHARP model, published in research last year, generates photorealistic 3D scenes from a single image in under a second. Google has more sensor data than anyone. More Street View cars, more satellites, more capture history. On navigation accuracy and geodata depth, Google Maps is still ahead by most measures. But fidelity in 3D city rendering is a different competition, and Apple just set a bar in that. Most people will experience this in the fall without knowing the name of the technology. They will open Flyover, look at a city they know, and notice it looks different. Real, not rendered. That is the moment Gaussian Splatting stops being a research term and becomes something a billion people use. Bookmark this. It will look prescient by October.

WOW. 😳 Apple just quietly won the 3D maps war at WWDC. Gaussian Splatting is coming to Apple Maps Flyover this fall. Apple Maps Flyover covers 300+ cities. Until yesterday, every single one was built on standard drone photogrammetry. The technology captures photos from the air and reconstructs 3D geometry from them. Gaussian Splatting does not reconstruct geometry. It represents the scene as millions of tiny 3D ellipsoids, each one carrying its own color and opacity information based on how light actually behaves in that location. The output is not a mesh model. It is a field of light. When you move through it, it does not crumble at the edges. The detail holds because it was never geometry to begin with. Apple has been hiring for this for years. Their SHARP model, published in research last year, generates photorealistic 3D scenes from a single image in under a second. Google has more sensor data than anyone. More Street View cars, more satellites, more capture history. On navigation accuracy and geodata depth, Google Maps is still ahead by most measures. But fidelity in 3D city rendering is a different competition, and Apple just set a bar in that. Most people will experience this in the fall without knowing the name of the technology. They will open Flyover, look at a city they know, and notice it looks different. Real, not rendered. That is the moment Gaussian Splatting stops being a research term and becomes something a billion people use. Bookmark this. It will look prescient by October.

Shruti

19,784 views • 1 month ago

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 views • 2 years ago

Virtual try-on is the future of retail. Soon, we won’t even remember a time when we shopped without it. With Doji I created a fashion campaign with my favorite new products with myself as the model, an AI generated digital twin, and brought it to life additional AI video tools. Open to see more experiments and use cases for this technology:

Virtual try-on is the future of retail. Soon, we won’t even remember a time when we shopped without it. With Doji I created a fashion campaign with my favorite new products with myself as the model, an AI generated digital twin, and brought it to life additional AI video tools. Open to see more experiments and use cases for this technology:

Paige Piskin XR

52,666 views • 1 year ago

Adobe.. has released a tool that combines Generative Gaussian Splats with a Diffusion layer and it's not all over the internet? WHAT IS GOING ON :D I had to test this out ofc! Substance 3D viewer, the new 3d viewer just released by Adobe, not only supports viewing of a large, large list of 3d formats, but it also supports generating of 3d models as Gaussian Splats. Combine that with the built-in 3d to image functionality and you basically have at least a few of the steps I've been doing for the last year or so using multiple tools. Exciting stuff! The new Photoshop Beta also supports the 3D viewer and you can import 3d files directly in photoshop as smart objects linked to the viewer. The elements you prompt in based on your models, can be easily exported to your clipboard without backgrounds making it very easy to use this in your normal image editing workflows. I have a suspicion that we'll see much more of this in other Adobe tools and I'm very curious to see this used for things like this in more 3d tools than "just" Adobe Neo (which is fantastic btw). #adobe #art #gaussiansplatting

Adobe.. has released a tool that combines Generative Gaussian Splats with a Diffusion layer and it's not all over the internet? WHAT IS GOING ON :D I had to test this out ofc! Substance 3D viewer, the new 3d viewer just released by Adobe, not only supports viewing of a large, large list of 3d formats, but it also supports generating of 3d models as Gaussian Splats. Combine that with the built-in 3d to image functionality and you basically have at least a few of the steps I've been doing for the last year or so using multiple tools. Exciting stuff! The new Photoshop Beta also supports the 3D viewer and you can import 3d files directly in photoshop as smart objects linked to the viewer. The elements you prompt in based on your models, can be easily exported to your clipboard without backgrounds making it very easy to use this in your normal image editing workflows. I have a suspicion that we'll see much more of this in other Adobe tools and I'm very curious to see this used for things like this in more 3d tools than "just" Adobe Neo (which is fantastic btw). #adobe #art #gaussiansplatting

Martin Nebelong

279,983 views • 1 year ago

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion discuss: We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.

AK

66,435 views • 1 year ago

WUNDERKAMMER by Isakost 🪲💀🫀 108 pieces / 100% on-chain / 14 Jan A capsule for the presence of an artist who has planted seeds of magic since the foundational days of cryptoart, Isa Kost's WUNDERKAMMER emerges from the lineage of collecting as a creative practice. Building a Wunderkammer, a Cabinet of Curiosities, is a practice that began in the 16th century as a way to honor objects of strange value. Now, we place such objects on blockchain. Each of the 108 objects in this Cabinet of Curiosities is a dead memory, a dead experience that has called out to be found, and Isa Kost transforms them one by one. The permanence of death becomes timeless presence on-chain. We look at our curated serial collections as a way to make a vessel that can hold an artist's presence, a means of creating a 'place' that can serve as a home base and coherent core for their body of work. The beauty of this collection and the meaning-making that it does in relation to its medium on-chain make us proud to bring it into the world. Find the collection page and further details below.

WUNDERKAMMER by Isakost 🪲💀🫀 108 pieces / 100% on-chain / 14 Jan A capsule for the presence of an artist who has planted seeds of magic since the foundational days of cryptoart, Isa Kost's WUNDERKAMMER emerges from the lineage of collecting as a creative practice. Building a Wunderkammer, a Cabinet of Curiosities, is a practice that began in the 16th century as a way to honor objects of strange value. Now, we place such objects on blockchain. Each of the 108 objects in this Cabinet of Curiosities is a dead memory, a dead experience that has called out to be found, and Isa Kost transforms them one by one. The permanence of death becomes timeless presence on-chain. We look at our curated serial collections as a way to make a vessel that can hold an artist's presence, a means of creating a 'place' that can serve as a home base and coherent core for their body of work. The beauty of this collection and the meaning-making that it does in relation to its medium on-chain make us proud to bring it into the world. Find the collection page and further details below.

sovrn.art

33,950 views • 1 year ago

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.

LUͦʷCͦk

18,417 views • 9 months ago