Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Invisible Stitch Generating Smooth 3D Scenes with Depth Inpainting 3D scene generation has quickly become a challenging new research direction, fueled by consistent improvements of 2D generative diffusion models. Most prior work in this area generates scenes by

AK

524,644 subscribers

44,611 просмотров • 2 лет назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 просмотров • 1 год назад

one 2d photo --> 3d gaussian splat quick test with Echo-2 by SpAItial -- these 3d scene generation models are getting better! already at a sufficient quality to serve as a virtual set / backdrop in your 3d tool of choice

one 2d photo --> 3d gaussian splat quick test with Echo-2 by SpAItial -- these 3d scene generation models are getting better! already at a sufficient quality to serve as a virtual set / backdrop in your 3d tool of choice

Bilawal Sidhu

23,775 просмотров • 5 месяцев назад

3D $ANIME ? Using 3D in anime or a complete series in 3D style, how does this affect the anime industry? Probably the most hated anime after which people hated 3D in anime was Berserk in 2016, it was really terrible, but 3D in anime is no longer what it was in 2010-20. Let's look at the pros and cons of 3D in anime and what series can be called good 3D anime Pros of 3D Animation: 🟢Dynamic camera angles & smooth motion 🟢Saves time & production costs 🟢Consistent character proportions Cons of 3D Animation: 🔴Can look stiff or "plastic" 🔴Limited facial expressions 🔴Often disliked by traditional anime fans Top 3D $ANIME series in my opinion: 🔵Houseki no Kuni – stunning visuals & fluid animation 🔵Dorohedoro – dark, gritty style perfectly fits CGI 🔵Beastars – expressive character animation 🔵Trigun: Stampede – cinematic action & high-quality CGI The best studio that combines 2D and 3D well is, I think, Ufotable Ufotable blends 2D & 3D seamlessly: 🟡3D backgrounds + 2D character animation 🟡Anime-style shading makes 3D look hand-drawn 🟡Frame-by-frame touch-ups for smooth motion 🟡Mixing 2D effects (fire, water) with 3D for impact Example: "Demon Slayer" – epic battle scenes like Tanjiro’s Water Breathing, using 3D camera movement while preserving 2D aesthetics. Which 3D $ANIME impressed you the most?

3D $ANIME ? Using 3D in anime or a complete series in 3D style, how does this affect the anime industry? Probably the most hated anime after which people hated 3D in anime was Berserk in 2016, it was really terrible, but 3D in anime is no longer what it was in 2010-20. Let's look at the pros and cons of 3D in anime and what series can be called good 3D anime Pros of 3D Animation: 🟢Dynamic camera angles & smooth motion 🟢Saves time & production costs 🟢Consistent character proportions Cons of 3D Animation: 🔴Can look stiff or "plastic" 🔴Limited facial expressions 🔴Often disliked by traditional anime fans Top 3D $ANIME series in my opinion: 🔵Houseki no Kuni – stunning visuals & fluid animation 🔵Dorohedoro – dark, gritty style perfectly fits CGI 🔵Beastars – expressive character animation 🔵Trigun: Stampede – cinematic action & high-quality CGI The best studio that combines 2D and 3D well is, I think, Ufotable Ufotable blends 2D & 3D seamlessly: 🟡3D backgrounds + 2D character animation 🟡Anime-style shading makes 3D look hand-drawn 🟡Frame-by-frame touch-ups for smooth motion 🟡Mixing 2D effects (fire, water) with 3D for impact Example: "Demon Slayer" – epic battle scenes like Tanjiro’s Water Breathing, using 3D camera movement while preserving 2D aesthetics. Which 3D $ANIME impressed you the most?

Kvintestar⛩️ Anime Artist

15,666 просмотров • 1 год назад

Combining the explicit control of 3D software with the creativity of generative AI models is a promising yet underrated workflow. Build your 3D scenes procedurally by describing them in natural language, then take them all the way with your image & video models of choice. Tools like intangible are built around such a workflow so you don't need to duct-tape apps together. Pretty cool!

Combining the explicit control of 3D software with the creativity of generative AI models is a promising yet underrated workflow. Build your 3D scenes procedurally by describing them in natural language, then take them all the way with your image & video models of choice. Tools like intangible are built around such a workflow so you don't need to duct-tape apps together. Pretty cool!

Bilawal Sidhu

37,629 просмотров • 1 год назад

With Hunyuan3D World Model 1.0 now released and open-sourced, we're excited to showcase the technical highlights behind this impressive innovation: ✅360° Panoramic Generation: Creates complete, immersive “world scenes”, far beyond localized views. ✅Explorable 3D Scene Generation: Generates diverse, spatially consistent 3D worlds from text/image for truly immersive exploration. ✅Interactive/Editable: Achieves separation of foreground objects, background terrain, ground, and sky, for seamless secondary editing. ✅Exportable Mesh: Generated scenes can be exported as 3D meshes for direct import into mainstream game engines and modeling software. ✅Industry-Leading SOTA Evaluation: Surpasses state-of-the-art open-source models in generation quality. As the industry's first open-source model for physical simulation and explorable world generation, Hunyuan3D World Model 1.0 aims to foster a collaborative community ecosystem with developers and enthusiasts. ✨ Try it now: 🤗 Hugging Face:

With Hunyuan3D World Model 1.0 now released and open-sourced, we're excited to showcase the technical highlights behind this impressive innovation: ✅360° Panoramic Generation: Creates complete, immersive “world scenes”, far beyond localized views. ✅Explorable 3D Scene Generation: Generates diverse, spatially consistent 3D worlds from text/image for truly immersive exploration. ✅Interactive/Editable: Achieves separation of foreground objects, background terrain, ground, and sky, for seamless secondary editing. ✅Exportable Mesh: Generated scenes can be exported as 3D meshes for direct import into mainstream game engines and modeling software. ✅Industry-Leading SOTA Evaluation: Surpasses state-of-the-art open-source models in generation quality. As the industry's first open-source model for physical simulation and explorable world generation, Hunyuan3D World Model 1.0 aims to foster a collaborative community ecosystem with developers and enthusiasts. ✨ Try it now: 🤗 Hugging Face:

Tencent Hy

23,240 просмотров • 1 год назад

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google actually decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:

Bilawal Sidhu

44,822 просмотров • 2 лет назад

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

DroneSplat: 3D Gaussian Splatting for Robust 3D Reconstruction from In-the-Wild Drone Imagery Abstract: Drones have become essential tools for reconstructing wild scenes due to their outstanding maneuverability. Recent advances in radiance field methods have achieved remarkable rendering quality, providing a new avenue for 3D reconstruction from drone imagery. However, dynamic distractors in wild environments challenge the static scene assumption in radiance fields, while limited view constraints hinder the accurate capture of underlying scene geometry. To address these challenges, we introduce DroneSplat, a novel framework designed for robust 3D reconstruction from in-the-wild drone imagery. Our method adaptively adjusts masking thresholds by integrating local-global segmentation heuristics with statistical approaches, enabling precise identification and elimination of dynamic distractors in static scenes. We enhance 3D Gaussian Splatting with multi-view stereo predictions and a voxel-guided optimization strategy, supporting high-quality rendering under limited view constraints. For comprehensive evaluation, we provide a drone-captured 3D reconstruction dataset encompassing both dynamic and static scenes. Extensive experiments demonstrate that DroneSplat outperforms both 3DGS and NeRF baselines in handling in-the-wild drone imagery.

MrNeRF

21,389 просмотров • 1 год назад

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢 We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_pris

Matthias Niessner

12,499 просмотров • 1 год назад

I am blown away 🤯. Check this out! CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models TL;DR: "To enable broader exploration of dynamic scenes, our model can generate new video clips of the same scene based on previously generated content and user-provided camera trajectories. This approach maintains dynamic capabilities, accurate camera control, and scene consistency throughout the extended exploration." "Our model enables precise camera control across diverse scenarios while preserving dynamic scene elements, e.g." "Our method can generate videos with strong 3D consistency, which enables high-quality 3D reconstruction using the camera-controlled videos." Contributions: 1) A systematic data curation pipeline for constructing a dynamic video dataset with camera trajectory annotations; 2) A lightweight camera control injection module and corresponding training strategy that preserves dynamic video generation capabilities while adding camera control effect; 3) A clip-wise autoregressive generation recipe that enables extended range exploration of generated scenes.

I am blown away 🤯. Check this out! CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models TL;DR: "To enable broader exploration of dynamic scenes, our model can generate new video clips of the same scene based on previously generated content and user-provided camera trajectories. This approach maintains dynamic capabilities, accurate camera control, and scene consistency throughout the extended exploration." "Our model enables precise camera control across diverse scenarios while preserving dynamic scene elements, e.g." "Our method can generate videos with strong 3D consistency, which enables high-quality 3D reconstruction using the camera-controlled videos." Contributions: 1) A systematic data curation pipeline for constructing a dynamic video dataset with camera trajectory annotations; 2) A lightweight camera control injection module and corresponding training strategy that preserves dynamic video generation capabilities while adding camera control effect; 3) A clip-wise autoregressive generation recipe that enables extended range exploration of generated scenes.

MrNeRF

12,633 просмотров • 1 год назад

NVIDIA just published ArtiFixer. It tackles one of the most frustrating parts of 3D Gaussian Splatting: what happens in areas you didn't capture. Standard reconstruction falls apart in under-observed regions. Floaters, blurry holes, missing geometry. ArtiFixer patches those gaps with an autoregressive diffusion model that stays consistent with what exists while generating plausible content where there's nothing. 1-3 dB PSNR improvement across all MipNeRF 360 scenes. On some of them, it's the only approach that produces usable output at all. Open source:

NVIDIA just published ArtiFixer. It tackles one of the most frustrating parts of 3D Gaussian Splatting: what happens in areas you didn't capture. Standard reconstruction falls apart in under-observed regions. Floaters, blurry holes, missing geometry. ArtiFixer patches those gaps with an autoregressive diffusion model that stays consistent with what exists while generating plausible content where there's nothing. 1-3 dB PSNR improvement across all MipNeRF 360 scenes. On some of them, it's the only approach that produces usable output at all. Open source:

KIRI Engine - 3D Scanner App

49,927 просмотров • 3 месяцев назад

GSTAR: Gaussian Surface Tracking and Reconstruction Contributions: • A new framework for tracking and reconstructing dynamic scenes, combining 3D Gaussians and meshes to effectively manage changes in topology. • A method for Gaussian unbinding and surface re-meshing, allowing for the generation of new surfaces as topologies evolve. • A method for handling large or fast deformations of surfaces between frames using scene flow warping. Abstract (excerpt): However, tracking dynamic surfaces with 3D Gaussians remains challenging due to complex topology changes, such as surfaces appearing, disappearing, or splitting. To address these challenges, we propose GSTAR, a novel method that achieves photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking for general dynamic scenes with changing topology. Given multi-view captures as input, GSTAR binds Gaussians to mesh faces to represent dynamic objects. For surfaces with consistent topology, GSTAR maintains the mesh topology and tracks the meshes using Gaussians.

GSTAR: Gaussian Surface Tracking and Reconstruction Contributions: • A new framework for tracking and reconstructing dynamic scenes, combining 3D Gaussians and meshes to effectively manage changes in topology. • A method for Gaussian unbinding and surface re-meshing, allowing for the generation of new surfaces as topologies evolve. • A method for handling large or fast deformations of surfaces between frames using scene flow warping. Abstract (excerpt): However, tracking dynamic surfaces with 3D Gaussians remains challenging due to complex topology changes, such as surfaces appearing, disappearing, or splitting. To address these challenges, we propose GSTAR, a novel method that achieves photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking for general dynamic scenes with changing topology. Given multi-view captures as input, GSTAR binds Gaussians to mesh faces to represent dynamic objects. For surfaces with consistent topology, GSTAR maintains the mesh topology and tracks the meshes using Gaussians.

MrNeRF

22,698 просмотров • 1 год назад

There's a lot of both 2D and 3D effort that we're putting into Star Birds together with our friends at Toukana Interactive! Here's a quick behind the scenes look at some of the assets, 3D models, birb shenanigans and cutscenes that you might have already seen in the Early Access version! 🚀✨ Have you already joined us on this space adventure? Don't miss out! Star Birds in Early Access is available on Steam:

There's a lot of both 2D and 3D effort that we're putting into Star Birds together with our friends at Toukana Interactive! Here's a quick behind the scenes look at some of the assets, 3D models, birb shenanigans and cutscenes that you might have already seen in the Early Access version! 🚀✨ Have you already joined us on this space adventure? Don't miss out! Star Birds in Early Access is available on Steam:

Kurzgesagt

100,543 просмотров • 11 месяцев назад

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

Yanjie Ze

75,420 просмотров • 1 год назад

HTML enters 3D! Or vice versa? With the new HTML in Canvas by WICG, we can finally put native DOM elements directly into WebGL/WebGPU scenes. It is experimental for now, but the possibilities for 3D interfaces and special effects are huge. This demo was built using Three.js and Omma AI (tool by Spline ) It’s a fun new way to explore what the web can do! Are you interested in seeing the demo?

HTML enters 3D! Or vice versa? With the new HTML in Canvas by WICG, we can finally put native DOM elements directly into WebGL/WebGPU scenes. It is experimental for now, but the possibilities for 3D interfaces and special effects are huge. This demo was built using Three.js and Omma AI (tool by Spline ) It’s a fun new way to explore what the web can do! Are you interested in seeing the demo?

Gábor Pribék

176,669 просмотров • 5 месяцев назад

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions: 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.

MrNeRF

10,691 просмотров • 1 год назад

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Shikun Liu

22,471 просмотров • 11 месяцев назад

AI video tools give you generation, but they rarely give you control. Pippit just launched 3D Director Studio. Instead of relying on prompts alone, you can plan and previsualize complex cinematic scenes in 3D, then use those references to guide video generation with Seedance 2.5. 🎥 Total Camera Control: Set up exact angles, FOV, and precise camera moves. 👥Character & Scene Blocking: Position character models, adjust postures, and place 150+ props in 3D space to solve multi-person positioning and spatial continuity. ⏱️Action Timeline: Sequence character movements and actions frame-by-frame using keyframes. No physical setups, green screens, or heavy production crews required—just complete creative control from concept to scene. Move from prompting to directing: Pippitofficial #3DDirectorStudio #PippitAI #Seedance25 #PippitPartner

AI video tools give you generation, but they rarely give you control. Pippit just launched 3D Director Studio. Instead of relying on prompts alone, you can plan and previsualize complex cinematic scenes in 3D, then use those references to guide video generation with Seedance 2.5. 🎥 Total Camera Control: Set up exact angles, FOV, and precise camera moves. 👥Character & Scene Blocking: Position character models, adjust postures, and place 150+ props in 3D space to solve multi-person positioning and spatial continuity. ⏱️Action Timeline: Sequence character movements and actions frame-by-frame using keyframes. No physical setups, green screens, or heavy production crews required—just complete creative control from concept to scene. Move from prompting to directing: Pippitofficial #3DDirectorStudio #PippitAI #Seedance25 #PippitPartner

kamran Hassan

16,458 просмотров • 22 дней назад

Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models Contributions: • We introduce Diffuman4D, a novel diffusion model that generates spatio-temporally consistent and high-resolution (1024p) human videos from sparse-view video inputs. • We propose a sliding iterative denoising mechanism that enhances both the spatial and temporal consistency of generated long-term videos while maintaining efficient inference. • We design a human pose conditioning scheme to enhance the appearance quality and motion accuracy of generated human videos. • We plan to release our processed version of the DNA-Rendering dataset, which we believe will benefit future research in this area.

Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models Contributions: • We introduce Diffuman4D, a novel diffusion model that generates spatio-temporally consistent and high-resolution (1024p) human videos from sparse-view video inputs. • We propose a sliding iterative denoising mechanism that enhances both the spatial and temporal consistency of generated long-term videos while maintaining efficient inference. • We design a human pose conditioning scheme to enhance the appearance quality and motion accuracy of generated human videos. • We plan to release our processed version of the DNA-Rendering dataset, which we believe will benefit future research in this area.

MrNeRF

24,729 просмотров • 1 год назад