Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

📢Introducing 360Anything, our method for lifting any perspective image or video to gravity-aligned 360° panoramas without using any camera or 3D information. This enables consistent novel view synthesis and 3D scene reconstruction. Project page: 🧵

Ziyi Wu

1,291 subscribers

63,573 Aufrufe • vor 9 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

23 Kommentare

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

*What is the task?* Unlike standard perspective data with a limited field of view, 360° panoramas capture the full scene with 360° horizontal and 180° vertical FoV. Our task: We take a perspective image or video and "hallucinate" the rest of that missing sphere. (2/N)

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

*Why panorama?* Standard video models struggle with object permanence—if a camera pans away and comes back, objects may disappear. With panoramas, the model is forced to generate everything in the scene. This serves as a "working memory" for consistent world generation. (3/N)

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

*Is our result 3D consistent?* Yes! We lift monocular video to 360° and directly train 3D Gaussian Splatting (3DGS) on it. 360Anything generates unobserved regions (e.g., behind the camera) with high consistency which blends seamlessly with observed areas in 3D. (4/N)

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

*Gravity-aligned generation* We always produce panoramas in a gravity-aligned canonical pose. Below we show results with large camera or object motion in the input video. 360Anything consistently tracks the camera trajectory indicated by the red border. (5/N)

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

*Method* Embracing the "bitter lesson", we replace geometric inductive biases in existing methods (e.g. camera projection) with a simple sequence-to-sequence DiT that learns camera geometry purely from data. 360Anything achieves new SOTA without requiring any camera info. (6/N)

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

Check out more results at: This work was done during my Student Researcher internship at @GoogleDeepMind Toronto, and it’s likely the last work of my PhD! Shout out to the amazing team @watson_nn, @taiyasaki, @fleet_dj, @marcusabrubaker, @srbhsxn! (N/N)

Profilbild von Sherwin Bahmani

Sherwin Bahmanivor 9 Monaten

Big guy 🚀

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

Bonus: seam artifact is a long-standing issue in pano gen. We attribute this to the zero-padding in Conv when encoding image/video latents with VAE, and propose a fix. Below, we concat both sides of the panorama to show they are continuous. Check out our paper for details!

Profilbild von Himanshu Kumar

Himanshu Kumarvor 9 Monaten

@Dazitu_616, your method for 360° panoramas is a clever approach for novel view synthesis; the project page will be useful.

Profilbild von Yong Zhang

Yong Zhangvor 9 Monaten

cool work! Where can I experience the model with custom inputs?

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

Thanks for your interest! We are kicking off the code release process internally but nothing is guaranteed. Google has been strict about weight release of generative models (safety & copyright issues). Stay tuned!

Profilbild von Jared Lambert

Jared Lambertvor 8 Monaten

Hey ziyi! I'm working on a project with @abundanceinst and this model is exactly what we need - I would love to talk to you about potential access! DM me!

Profilbild von Ulrich Forkel

Ulrich Forkelvor 6 Monaten

It's been almost three months—haven't you decided to release it as open source yet?

Profilbild von IAndrés

IAndrésvor 9 Monaten

🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯

Profilbild von Debitto Lozano // MetAI CEO

Debitto Lozano // MetAI CEOvor 9 Monaten

Hi. If you are looking for AI testers in Japan let me know. I would love to try it(if possible).

Profilbild von Jason Diamond

Jason Diamondvor 9 Monaten

Will this generate Stereo 360°? Can it generate 180° stereo from a non fisheye mono or non fisheye stereo pair?

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

That would be an interesting direction--having more than a single input/output. Could be a future direction to explore (if the right data exists)

Profilbild von Mattias Johansson

Mattias Johanssonvor 9 Monaten

Wow. Super. Any change we can get the code to play with?

Profilbild von Ziyi Wu

Ziyi Wuvor 9 Monaten

We are kicking off the code release process internally but nothing is guaranteed. Google has been strict about weight release of generative models (safety & copyright issues). Stay tuned!

Profilbild von Yuheng Liu

Yuheng Liuvor 9 Monaten

Cool work!

Profilbild von Ghanshyam Vanol

Ghanshyam Vanolvor 5 Monaten

still waiting for realesed

Profilbild von Victor E Garcia

Victor E Garciavor 9 Monaten

nice stuff! i found the "Circular Latent Encoding" technique quite ingenious to ensure the seams of the 360 panorama stay consistent

Profilbild von X Æ A-12

X Æ A-12vor 9 Monaten

WTF ! 😲

Ähnliche Videos

Novel view synthesis has long been a core challenge in 3D vision. But how much 3D inductive bias is truly needed? —Surprisingly, very little! Introducing "LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias"—a fully transformer-based approach that enables scalable, generalizable, and fully data-driven novel view synthesis, from sparse posed inputs. 🧵(1/6) Project Page:

Novel view synthesis has long been a core challenge in 3D vision. But how much 3D inductive bias is truly needed? —Surprisingly, very little! Introducing "LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias"—a fully transformer-based approach that enables scalable, generalizable, and fully data-driven novel view synthesis, from sparse posed inputs. 🧵(1/6) Project Page:

Haian Jin

114,987 Aufrufe • vor 2 Jahren

Introducing Stable Virtual Camera: This multi-view diffusion model transforms 2D images into immersive 3D videos with realistic depth and perspective—without complex reconstruction or scene-specific optimization.

Introducing Stable Virtual Camera: This multi-view diffusion model transforms 2D images into immersive 3D videos with realistic depth and perspective—without complex reconstruction or scene-specific optimization.

Stability AI

197,625 Aufrufe • vor 1 Jahr

Check out CAT3D! Image(s)-to-3D in 1 minute! Given any number of real or generated images, CAT3D uses a multi-view diffusion prior to create consistent novel views. These views are used to reconstruct a 3D scene using NeRF/3DGS.

Check out CAT3D! Image(s)-to-3D in 1 minute! Given any number of real or generated images, CAT3D uses a multi-view diffusion prior to create consistent novel views. These views are used to reconstruct a 3D scene using NeRF/3DGS.

Philipp Henzler

12,752 Aufrufe • vor 2 Jahren

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

AK

304,708 Aufrufe • vor 3 Jahren

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:

AK

121,020 Aufrufe • vor 3 Jahren

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 Aufrufe • vor 3 Jahren

Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)

Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)

Vincent Sitzmann

75,715 Aufrufe • vor 4 Monaten

Introducing Video Generation in Omma - Text to video - Image to video Make videos, and add them to any website, app or 3d scene. Available Now for Pro/Max users.

Introducing Video Generation in Omma - Text to video - Image to video Make videos, and add them to any website, app or 3d scene. Available Now for Pro/Max users.

Omma AI

11,709 Aufrufe • vor 6 Monaten

Feed-forward 3D Gaussians from Visual Geometry Group (VGG) strike again! Flash3D has now been accepted to 3DV 2025: it is a method for feed-forward single-view 3D scene reconstruction. Project page: Code: Arxiv: A 🧵👇

Feed-forward 3D Gaussians from Visual Geometry Group (VGG) strike again! Flash3D has now been accepted to 3DV 2025: it is a method for feed-forward single-view 3D scene reconstruction. Project page: Code: Arxiv: A 🧵👇

Stan Szymanowicz

26,994 Aufrufe • vor 1 Jahr

3D-aware Conditional Image Synthesis abs: project page:

3D-aware Conditional Image Synthesis abs: project page:

AK

301,613 Aufrufe • vor 3 Jahren

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Thomas Kipf

98,036 Aufrufe • vor 2 Jahren

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

Matthias Niessner

18,447 Aufrufe • vor 5 Monaten

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2

MrNeRF

12,190 Aufrufe • vor 2 Jahren

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

Matthias Niessner

19,093 Aufrufe • vor 7 Monaten

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,934 Aufrufe • vor 1 Jahr

Check out our fun work on robust view synthesis! 🤩 Using a casual video as input, our method jointly 1⃣estimates accurate camera poses and 2⃣reconstructs multiple local radiance fields for a large scene. Project🕸️:

Check out our fun work on robust view synthesis! 🤩 Using a casual video as input, our method jointly 1⃣estimates accurate camera poses and 2⃣reconstructs multiple local radiance fields for a large scene. Project🕸️:

Jia-Bin Huang

35,538 Aufrufe • vor 3 Jahren

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

Jiaman Li

15,886 Aufrufe • vor 1 Jahr

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 Aufrufe • vor 2 Jahren

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Yu Xiang

57,572 Aufrufe • vor 1 Jahr