Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

This AI paper just solved Google Earth's biggest problem. Satellites look down. Humans look across. That perspective gap is why 3D maps are limited to cities you can blanket with aerial flyovers. Skyfall-GS bridges the gap by synthesizing the views we never captured - rebuilding missing facades and street-level... show more

Bilawal Sidhu

104,214 subscribers

135,003 views • 6 months ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

So these researchers figured out you can basically hallucinate 3D cities into existence using just satellite photos & a diffusion model. The problem's pretty straightforward: satellites only see rooftops. Building facades? Invisible. Street-level detail? Doesn't exist. But people want flyable 3D environments, which means you need all that occluded geometry. When I worked on google maps photogrammetry, we could only use satellite-based 3D for isolated stuff like the pyramids - anything city-scale required airplane flyovers. Which is fine until you hit aerial-denied regions where you literally can't fly. Huge chunks of the world just unavailable. Their trick is honestly kind of beautiful. They train gaussian splats on satellite views, but as it descends toward ground level, the renders turn to absolute garbage - artifacts everywhere. Instead of fighting this, they just treat those nightmare renders as the input to a diffusion model. Basically - "hey FLUX, fix this mess." Then here's where it gets clever: they generate multiple diffusion samples per view instead of committing to one. Because any single denoising path is probably wrong in 3D space, but if you generate a couple and let the GS optimization find consensus across them, you get actual geometric consistency. They do this in episodes, curriculum style - start high, gradually descend (hence the name Skyfall-GS!). With each iteration the ground-level views get less fucked. By the end you've got real-time flyable cities that look surprisingly real, and the geometry still matches the satellite input. No 3D training data. No street-level photos. Just satellites + diffusion doing what it does best - filling in the blanks. It's like neural scene completion but actually practical, and it unlocks basically the entire world.

So these researchers figured out you can basically hallucinate 3D cities into existence using just satellite photos & a diffusion model. The problem's pretty straightforward: satellites only see rooftops. Building facades? Invisible. Street-level detail? Doesn't exist. But people want flyable 3D environments, which means you need all that occluded geometry. When I worked on google maps photogrammetry, we could only use satellite-based 3D for isolated stuff like the pyramids - anything city-scale required airplane flyovers. Which is fine until you hit aerial-denied regions where you literally can't fly. Huge chunks of the world just unavailable. Their trick is honestly kind of beautiful. They train gaussian splats on satellite views, but as it descends toward ground level, the renders turn to absolute garbage - artifacts everywhere. Instead of fighting this, they just treat those nightmare renders as the input to a diffusion model. Basically - "hey FLUX, fix this mess." Then here's where it gets clever: they generate multiple diffusion samples per view instead of committing to one. Because any single denoising path is probably wrong in 3D space, but if you generate a couple and let the GS optimization find consensus across them, you get actual geometric consistency. They do this in episodes, curriculum style - start high, gradually descend (hence the name Skyfall-GS!). With each iteration the ground-level views get less fucked. By the end you've got real-time flyable cities that look surprisingly real, and the geometry still matches the satellite input. No 3D training data. No street-level photos. Just satellites + diffusion doing what it does best - filling in the blanks. It's like neural scene completion but actually practical, and it unlocks basically the entire world.

Bilawal Sidhu

241,774 views • 7 months ago

This is pretty cool 👓 Google Announces Geospatial Creator powered by ARCore & Google Maps and it allows you to visualize, design, and publish world-anchored content by using Unity ! Few features available: 📌 Photorealistic 3D Tiles: Visualize the 3D geometry of the world and deploy location-anchored content accurately with high-resolution, photorealistic 3D tiles. 📌 Geospatial anchors: Place and anchor 3D content at any given latitude, longitude, and altitude with sub-meter accuracy in areas covered by Google Street View. 📌 Terrain anchors: Place and anchor 3D content with only latitude and longitude coordinates, using data from Google Maps to determine ground level. 📌 Rooftop anchors: Use rooftops to place and anchor 3D content with respect to the building geometry and terrain in areas covered by Google Street View. 📢 To get started with Unity follow these steps: #AR #Google

This is pretty cool 👓 Google Announces Geospatial Creator powered by ARCore & Google Maps and it allows you to visualize, design, and publish world-anchored content by using Unity ! Few features available: 📌 Photorealistic 3D Tiles: Visualize the 3D geometry of the world and deploy location-anchored content accurately with high-resolution, photorealistic 3D tiles. 📌 Geospatial anchors: Place and anchor 3D content at any given latitude, longitude, and altitude with sub-meter accuracy in areas covered by Google Street View. 📌 Terrain anchors: Place and anchor 3D content with only latitude and longitude coordinates, using data from Google Maps to determine ground level. 📌 Rooftop anchors: Use rooftops to place and anchor 3D content with respect to the building geometry and terrain in areas covered by Google Street View. 📢 To get started with Unity follow these steps: #AR #Google

Dilmer 👓

58,241 views • 3 years ago

3D, AI, NeRFs and oh my! Immersive view is now rolling out to 5 cities, with 5 more to follow. Experience the best of all views, with helpful info layered on top, so you can decide when & where to go. Proud to be part of the team-of-teams that made this maps milestone a reality!

3D, AI, NeRFs and oh my! Immersive view is now rolling out to 5 cities, with 5 more to follow. Experience the best of all views, with helpful info layered on top, so you can decide when & where to go. Proud to be part of the team-of-teams that made this maps milestone a reality!

Bilawal Sidhu

318,372 views • 3 years ago

📽️➡️🏠 Can an image become a consistent 3D world? Video Diffusion Models look stunning, but 3D consistency is still a nightmare… #WorldStereo is different. We gave the AI a "3D brain" using Geometric Memories. The result? ✅ Zero flickering. ✅ Perfect 3D consistency.

📽️➡️🏠 Can an image become a consistent 3D world? Video Diffusion Models look stunning, but 3D consistency is still a nightmare… #WorldStereo is different. We gave the AI a "3D brain" using Geometric Memories. The result? ✅ Zero flickering. ✅ Perfect 3D consistency.

Tengfei Wang

19,865 views • 3 months ago

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

AK

161,400 views • 2 years ago

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 views • 2 years ago

Apple Maps 3d is insanely impressive now (see video below) Google on the other hand decided to build an insanely clunky solution to display 3d maps to avoid using mobile GPUs: A Google server streams (!) a live video of its 3d rendered map to you, the result is a disastrous: a super laggy and clunky 3d view full of video compression artefacts with a response time of 2-3 seconds for every movement And I know WHY they built that, someone felt it was not inclusive to only have 3d maps that are high performing on iOS because Apple's GPUs are insanely fast and can deal with it (as you can see below in Apple Maps) but they wanted to support inferior phones, I get it But then they also decided to worsen the experience for iOS users who could easily run 3d natively on its GPU It's just an insane product decision, there's no way to cut it: they could have super smooth 3d like Apple Maps but neutered it with a clunky live stream Even more insane because Google does have native 3d on iOS in a completely different app that nobody uses: Google Earth Is 3d important? Not rly for nav no but it shows again something about Google's engineering decisions, like the redesign of the Google OAuth Login screen that made zero sense Nobody is speaking up against bad ideas in Google meetings, everyone's too scared of HR maybe, it's the opposite of a meritocracy and it's just super visible from using Google products right now

Apple Maps 3d is insanely impressive now (see video below) Google on the other hand decided to build an insanely clunky solution to display 3d maps to avoid using mobile GPUs: A Google server streams (!) a live video of its 3d rendered map to you, the result is a disastrous: a super laggy and clunky 3d view full of video compression artefacts with a response time of 2-3 seconds for every movement And I know WHY they built that, someone felt it was not inclusive to only have 3d maps that are high performing on iOS because Apple's GPUs are insanely fast and can deal with it (as you can see below in Apple Maps) but they wanted to support inferior phones, I get it But then they also decided to worsen the experience for iOS users who could easily run 3d natively on its GPU It's just an insane product decision, there's no way to cut it: they could have super smooth 3d like Apple Maps but neutered it with a clunky live stream Even more insane because Google does have native 3d on iOS in a completely different app that nobody uses: Google Earth Is 3d important? Not rly for nav no but it shows again something about Google's engineering decisions, like the redesign of the Google OAuth Login screen that made zero sense Nobody is speaking up against bad ideas in Google meetings, everyone's too scared of HR maybe, it's the opposite of a meritocracy and it's just super visible from using Google products right now

@levelsio

1,488,725 views • 1 year ago

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

Bilawal Sidhu

27,278 views • 6 months ago

3D capture is dope for delight, but the utility might be even more impactful. This digital twin of Boston fuses together aerial 3d scans + vector maps + building (BIM) data + historical crime data + utility waterlines + zoning data. You can answer questions like: 1. Which buildings were recently constructed in the Boston skyline? 2. How does crime density correlate with zoning types across the city? 3. Where are the oldest water mains located, and how do they relate to the city's infrastructure? 4. What would this busy street look like with dedicated biking lanes? It's wild the progress the geospatial world is making. And you can do almost all of this with Esri tools and a sprinkling of game engines.

3D capture is dope for delight, but the utility might be even more impactful. This digital twin of Boston fuses together aerial 3d scans + vector maps + building (BIM) data + historical crime data + utility waterlines + zoning data. You can answer questions like: 1. Which buildings were recently constructed in the Boston skyline? 2. How does crime density correlate with zoning types across the city? 3. Where are the oldest water mains located, and how do they relate to the city's infrastructure? 4. What would this busy street look like with dedicated biking lanes? It's wild the progress the geospatial world is making. And you can do almost all of this with Esri tools and a sprinkling of game engines.

Bilawal Sidhu

85,286 views • 2 years ago

Here's a look behind the scenes of how I used a 3d model, generated from an image, as a "driver" for ai animation. In the near future we'll see more 3d tools that support these types of workflows, with much more emphasis on enhancing user input instead of just letting the ai do everything. I used the 3d model to generate keyframes that I then animated in Luma Dream. By using a 3d model with a diffusion "layer" done with Krea, I got quite high quality frames with a relatively high consistency as well since the AI didn't have to "hallucinate" that much. I upscaled and detailed the frames using Magnific. Using semi-detailed 3d models as drivers for gen ai is very powerful, and it allows us to use models that are sculpted in a more free-flowing type of workflow that doesn't rely so much on high surface detailing or very time-consuming finish, but instead relies more on gestural 3d sculpting. #art #ai

Here's a look behind the scenes of how I used a 3d model, generated from an image, as a "driver" for ai animation. In the near future we'll see more 3d tools that support these types of workflows, with much more emphasis on enhancing user input instead of just letting the ai do everything. I used the 3d model to generate keyframes that I then animated in Luma Dream. By using a 3d model with a diffusion "layer" done with Krea, I got quite high quality frames with a relatively high consistency as well since the AI didn't have to "hallucinate" that much. I upscaled and detailed the frames using Magnific. Using semi-detailed 3d models as drivers for gen ai is very powerful, and it allows us to use models that are sculpted in a more free-flowing type of workflow that doesn't rely so much on high surface detailing or very time-consuming finish, but instead relies more on gestural 3d sculpting. #art #ai

Martin Nebelong

48,200 views • 1 year ago

3d gaussian splats + photogrammetry can work together beautifully – you don’t have to choose. I’m using a drone capture for the hero asset. Google maps aerial photogrammetry for the world around it.

3d gaussian splats + photogrammetry can work together beautifully – you don’t have to choose. I’m using a drone capture for the hero asset. Google maps aerial photogrammetry for the world around it.

Bilawal Sidhu

24,161 views • 27 days ago

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

AK

294,442 views • 2 years ago

holy crap! apple just beat google to the punch -- 3d gaussian splatting is coming to apple maps. these 3d scenes are made from oblique aerial imagery. but unlike blobby photogrammetry -- no more broccoli trees, no more melted powerlines -- ground level detail that actually holds up. here's hoping google maps/earth follows suite soon -- they have a significantly larger corpus of sensor data to work with. time to splat the world!

holy crap! apple just beat google to the punch -- 3d gaussian splatting is coming to apple maps. these 3d scenes are made from oblique aerial imagery. but unlike blobby photogrammetry -- no more broccoli trees, no more melted powerlines -- ground level detail that actually holds up. here's hoping google maps/earth follows suite soon -- they have a significantly larger corpus of sensor data to work with. time to splat the world!

Bilawal Sidhu

1,057,161 views • 4 days ago

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors paper page: present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images.

AK

305,643 views • 2 years ago

Google Maps just got an INSANE upgrade! Google just dropped AI powered "Ask Maps" This is the biggest Maps update in over a decade. Powered by Gemini. Rolling out today. 5 wild things you can now get out of it:

Google Maps just got an INSANE upgrade! Google just dropped AI powered "Ask Maps" This is the biggest Maps update in over a decade. Powered by Gemini. Rolling out today. 5 wild things you can now get out of it:

Min Choi

1,201,142 views • 3 months ago

The performance and fluidity of Apple Maps 'Look Around' compared to Google Maps 'Street View'

The performance and fluidity of Apple Maps 'Look Around' compared to Google Maps 'Street View'

˗ˏˋ Vinoth Ragunathan ˊˎ˗

290,916 views • 1 year ago

Very interesting Interactive live AI-generated world you can walk around in It does feel like something is pre-gen'd though cause you can switch from map to map and the maps stay the same, but it doesn't look like a 3d scanned world completely, more like semi-AI?

Very interesting Interactive live AI-generated world you can walk around in It does feel like something is pre-gen'd though cause you can switch from map to map and the maps stay the same, but it doesn't look like a 3d scanned world completely, more like semi-AI?

@levelsio

112,766 views • 1 year ago

🥁 The winners of the Google Maps Platform Awards are now live, and the Grand Prize goes to Innobrix! 🎉 The 3D configuration platform combines BIM data with Google Maps to provide real-world context for developments. 📢 Check it out along with the rest of our winners ⬇️

🥁 The winners of the Google Maps Platform Awards are now live, and the Grand Prize goes to Innobrix! 🎉 The 3D configuration platform combines BIM data with Google Maps to provide real-world context for developments. 📢 Check it out along with the rest of our winners ⬇️

Google Maps Platform

14,361 views • 9 months ago

People are just realizing something important. One of the largest real-world visual datasets for AI was built by millions of users… without them even knowing it. At Over the Reality, we believe this should work differently. We are building a global 3D map of the world through a community that maps real places on purpose, and gets rewarded for every meaningful contribution. People helping build the global 3D map of the world can earn from $1 to $3 for each location mapped with a smartphone, or up to $18 per hour using an Insta360 X5. Apply here:

People are just realizing something important. One of the largest real-world visual datasets for AI was built by millions of users… without them even knowing it. At Over the Reality, we believe this should work differently. We are building a global 3D map of the world through a community that maps real places on purpose, and gets rewarded for every meaningful contribution. People helping build the global 3D map of the world can earn from $1 to $3 for each location mapped with a smartphone, or up to $18 per hour using an Insta360 X5. Apply here:

Over the Reality 🌐

288,752 views • 2 months ago