Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

📢Introducing 360Anything, our method for lifting any perspective image or video to gravity-aligned 360° panoramas without using any camera or 3D information. This enables consistent novel view synthesis and 3D scene reconstruction. Project page: 🧵

Dazitu_616's profile picture

Ziyi Wu

1,303 subscribers

64,086 görüntüleme • 9 ay önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

23 Yorum

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

*What is the task?* Unlike standard perspective data with a limited field of view, 360° panoramas capture the full scene with 360° horizontal and 180° vertical FoV. Our task: We take a perspective image or video and "hallucinate" the rest of that missing sphere. (2/N)

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

*Why panorama?* Standard video models struggle with object permanence—if a camera pans away and comes back, objects may disappear. With panoramas, the model is forced to generate everything in the scene. This serves as a "working memory" for consistent world generation. (3/N)

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

*Is our result 3D consistent?* Yes! We lift monocular video to 360° and directly train 3D Gaussian Splatting (3DGS) on it. 360Anything generates unobserved regions (e.g., behind the camera) with high consistency which blends seamlessly with observed areas in 3D. (4/N)

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

*Gravity-aligned generation* We always produce panoramas in a gravity-aligned canonical pose. Below we show results with large camera or object motion in the input video. 360Anything consistently tracks the camera trajectory indicated by the red border. (5/N)

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

*Method* Embracing the "bitter lesson", we replace geometric inductive biases in existing methods (e.g. camera projection) with a simple sequence-to-sequence DiT that learns camera geometry purely from data. 360Anything achieves new SOTA without requiring any camera info. (6/N)

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

Check out more results at: This work was done during my Student Researcher internship at @GoogleDeepMind Toronto, and it’s likely the last work of my PhD! Shout out to the amazing team @watson_nn, @taiyasaki, @fleet_dj, @marcusabrubaker, @srbhsxn! (N/N)

Sherwin Bahmani profil fotoğrafı
Sherwin Bahmani9 ay önce

Big guy 🚀

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

Bonus: seam artifact is a long-standing issue in pano gen. We attribute this to the zero-padding in Conv when encoding image/video latents with VAE, and propose a fix. Below, we concat both sides of the panorama to show they are continuous. Check out our paper for details!

Himanshu Kumar profil fotoğrafı
Himanshu Kumar9 ay önce

@Dazitu_616, your method for 360° panoramas is a clever approach for novel view synthesis; the project page will be useful.

Yong Zhang profil fotoğrafı
Yong Zhang9 ay önce

cool work! Where can I experience the model with custom inputs?

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

Thanks for your interest! We are kicking off the code release process internally but nothing is guaranteed. Google has been strict about weight release of generative models (safety & copyright issues). Stay tuned!

Jared Lambert profil fotoğrafı
Jared Lambert9 ay önce

Hey ziyi! I'm working on a project with @abundanceinst and this model is exactly what we need - I would love to talk to you about potential access! DM me!

Ulrich Forkel profil fotoğrafı
Ulrich Forkel6 ay önce

It's been almost three months—haven't you decided to release it as open source yet?

IAndrés profil fotoğrafı
IAndrés9 ay önce

🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯🤯

Debitto Lozano // MetAI CEO profil fotoğrafı
Debitto Lozano // MetAI CEO9 ay önce

Hi. If you are looking for AI testers in Japan let me know. I would love to try it(if possible).

Jason Diamond profil fotoğrafı
Jason Diamond9 ay önce

Will this generate Stereo 360°? Can it generate 180° stereo from a non fisheye mono or non fisheye stereo pair?

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

That would be an interesting direction--having more than a single input/output. Could be a future direction to explore (if the right data exists)

Mattias Johansson profil fotoğrafı
Mattias Johansson9 ay önce

Wow. Super. Any change we can get the code to play with?

Ziyi Wu profil fotoğrafı
Ziyi Wu9 ay önce

We are kicking off the code release process internally but nothing is guaranteed. Google has been strict about weight release of generative models (safety & copyright issues). Stay tuned!

Yuheng Liu profil fotoğrafı
Yuheng Liu9 ay önce

Cool work!

Ghanshyam Vanol profil fotoğrafı
Ghanshyam Vanol5 ay önce

still waiting for realesed

Victor E Garcia profil fotoğrafı
Victor E Garcia9 ay önce

nice stuff! i found the "Circular Latent Encoding" technique quite ingenious to ensure the seams of the 360 panorama stay consistent

X Æ A-12 profil fotoğrafı
X Æ A-129 ay önce

WTF ! 😲

Benzer Videolar

Novel view synthesis has long been a core challenge in 3D vision. But how much 3D inductive bias is truly needed? —Surprisingly, very little! Introducing "LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias"—a fully transformer-based approach that enables scalable, generalizable, and fully data-driven novel view synthesis, from sparse posed inputs. 🧵(1/6) Project Page:
0:36

Novel view synthesis has long been a core challenge in 3D vision. But how much 3D inductive bias is truly needed? —Surprisingly, very little! Introducing "LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias"—a fully transformer-based approach that enables scalable, generalizable, and fully data-driven novel view synthesis, from sparse posed inputs. 🧵(1/6) Project Page:

Haian Jin

114,987 görüntüleme • 2 yıl önce

Introducing Stable Virtual Camera: This multi-view diffusion model transforms 2D images into immersive 3D videos with realistic depth and perspective—without complex reconstruction or scene-specific optimization.
0:39

Introducing Stable Virtual Camera: This multi-view diffusion model transforms 2D images into immersive 3D videos with realistic depth and perspective—without complex reconstruction or scene-specific optimization.

Stability AI

197,625 görüntüleme • 1 yıl önce

Check out CAT3D! Image(s)-to-3D in 1 minute! Given any number of real or generated images, CAT3D uses a multi-view diffusion prior to create consistent novel views. These views are used to reconstruct a 3D scene using NeRF/3DGS.
0:38

Check out CAT3D! Image(s)-to-3D in 1 minute! Given any number of real or generated images, CAT3D uses a multi-view diffusion prior to create consistent novel views. These views are used to reconstruct a 3D scene using NeRF/3DGS.

Philipp Henzler

12,752 görüntüleme • 2 yıl önce

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:
0:24

Generative Novel View Synthesis with 3D-Aware Diffusion Models abs: project page:

AK

304,708 görüntüleme • 3 yıl önce

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:
0:42

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models abs: project page:

AK

121,020 görüntüleme • 3 yıl önce

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.
0:43

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization paper page: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

AK

179,061 görüntüleme • 3 yıl önce

Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)
0:32

Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)

Vincent Sitzmann

75,782 görüntüleme • 5 ay önce

Introducing Video Generation in Omma - Text to video - Image to video Make videos, and add them to any website, app or 3d scene. Available Now for Pro/Max users.
0:23

Introducing Video Generation in Omma - Text to video - Image to video Make videos, and add them to any website, app or 3d scene. Available Now for Pro/Max users.

Omma AI

11,709 görüntüleme • 7 ay önce

Feed-forward 3D Gaussians from Visual Geometry Group (VGG) strike again! Flash3D has now been accepted to 3DV 2025: it is a method for feed-forward single-view 3D scene reconstruction. Project page: Code: Arxiv: A 🧵👇
0:27

Feed-forward 3D Gaussians from Visual Geometry Group (VGG) strike again! Flash3D has now been accepted to 3DV 2025: it is a method for feed-forward single-view 3D scene reconstruction. Project page: Code: Arxiv: A 🧵👇

Stan Szymanowicz

26,994 görüntüleme • 1 yıl önce

3D-aware Conditional Image Synthesis abs: project page:
0:34

3D-aware Conditional Image Synthesis abs: project page:

AK

301,613 görüntüleme • 3 yıl önce

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵
0:30

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Thomas Kipf

98,036 görüntüleme • 2 yıl önce

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai
0:26

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢 Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior! We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views. GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion. Amazing work by Katharina Schmid, Nicolas von Lützow, Jozef, Angela Dai

Matthias Niessner

18,482 görüntüleme • 5 ay önce

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai
0:25

📢WorldAgents: 3D worlds only from 2D image models - without any training! We propose an agentic approach with a Director (VLM) to plan the scene, a Generator (Flux or NanoBanana) for new views, and a Verifier (VLM) for selection / 3D consistency. -> High-fidelity 3D worlds from a single text prompt. What's remarkable: our agents find consistent views from 2D image models to obtain 3D-consistent worlds; this shows that image models contain world priors - agents just need to find them! Great work by Ziya Erkoç Angela Dai

Matthias Niessner

19,093 görüntüleme • 7 ay önce

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2
1:07

WaterSplatting: Fast Underwater 3D Scene Reconstruction Using Gaussian Splatting Project: Method ⬇️ 1 I 2

MrNeRF

12,190 görüntüleme • 2 yıl önce

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:
0:50

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,960 görüntüleme • 1 yıl önce

Check out our fun work on robust view synthesis! 🤩 Using a casual video as input, our method jointly 1⃣estimates accurate camera poses and 2⃣reconstructs multiple local radiance fields for a large scene. Project🕸️:
1:41

Check out our fun work on robust view synthesis! 🤩 Using a casual video as input, our method jointly 1⃣estimates accurate camera poses and 2⃣reconstructs multiple local radiance fields for a large scene. Project🕸️:

Jia-Bin Huang

35,538 görüntüleme • 3 yıl önce

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:
0:21

🔥 Introducing MVLift: Generate realistic 3D motion without any 3D training data - just using 2D poses from monocular videos! Applicable to human motion, human-object interaction & animal motion. Joint work w/ Jiajun Wu & Karen 💡 How? We reformulate 3D motion estimation as generating consistent multi-view 2D pose sequences. Our framework uses 2D motion diffusion to progressively establish multi-view consistency, requiring only single-view 2D pose sequences for training. Project: Video with demonstration: Paper:

Jiaman Li

15,886 görüntüleme • 1 yıl önce

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency
0:21

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 görüntüleme • 2 yıl önce

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:
1:00

Introducing HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction. We built a multi-camera system and a semi-automatic method for annotating the shape and pose of hands and objects Project page:

Yu Xiang

57,604 görüntüleme • 1 yıl önce