Skip navigation
New paper from Apple - Sharp Monocular View Synthesis in Less than a Second
0:00 / 0:22

New paper from Apple - Sharp Monocular View Synthesis in Less than a Second

Tim Davison ᯅ

@timd_ca5K followers

Visit on X
1,769 likesDownloadOriginal post

133,200 views

New paper from Apple - Sharp Monocular View Synthesis in Less than a Second Mescheder et al. @ Apple just released a very impressive paper (congrats! 🎉🥳). You give it an image and it generates a really great looking 3d Gaussian representation. Uses depth pro. It's really good. The model is about 3GB, and takes ~5-10s on my M1 Max per image. Single feedforward pass network. The video is using "Metal Splatter" to view the .ply from ml-sharp on Apple Vision Pro. Some really wow moments today trying it on different scenes.

39 Comments

Comments from the original post on X (Twitter)

  • Tim Davison ᯅ's profile picture

    @timd_ca

    Project page:

    22 likes

  • softyoda's profile picture

    @softyoda

    It's fun

    43 likes

  • Tim Davison ᯅ's profile picture

    @timd_ca

    Wow, it did really well with this painting

    7 likes

  • Lee Macdonald's profile picture

    @MistahLee

    @Scobleizer For regular humans? Or for cracked coders?

    4 likes

  • Tim Davison ᯅ's profile picture

    @timd_ca

    @Scobleizer I wouldn't be surprised if this is the technique (or closely related) is what powers "spatial scenes" in the Apple Vision Pro Photos app. The code repository is pretty easy to use and setup, then just "sharp predict -i mountain.jpg -o splats" and copy to the Vision Pro

    7 likes

  • Amitay Keisar's profile picture

    @AmKeisar

    Now you just need to "take a picture" from the generated world and continue exploring without the need to reload the entire "world"

    4 likes

  • Tim Davison ᯅ's profile picture

    @timd_ca

    Yea! It's optimized for a parallax single perspective, so it's empty where the camera couldn't see. Would be so cool if they could fill in the missing pieces the way world models (such as World Labs) is doing. But I think the exciting thing is that it's fast.. just a few seconds on a MacBook Pro

    6 likes

  • Amitay Keisar's profile picture

    @AmKeisar

    I guess that is the next step, if not by apple probably from the open source community, but that is opening to so much cool tech!

  • dc's profile picture

    @d11cc3s

    Now if they could use this for satellite view on Apple Maps, that would be amazing.

    3 likes

  • SOUIC ᯅ's profile picture

    @SSOUIC

    Can normal people try this

    2 likes

  • Tim Davison ᯅ's profile picture

    @timd_ca

    Yea! The repository is python based. Easy to setup (you need to have conda/anacodna installed), run a few install commands, it was pretty seamless. Then the command is just "sharp predict -i mountain.jpg -o out" Takes a few seconds and copy the resulting .ply (Gaussian splat scene) to Apple Vision Pro (iCloud) and use the free app Metal Splatter or view in

    7 likes

  • SOUIC ᯅ's profile picture

    @SSOUIC

    The answer is no

    1 like

  • Pete Magazzu's profile picture

    @PeteMagazzu

    What’s this useful for?

    2 likes

  • Tim Davison ᯅ's profile picture

    @timd_ca

    Apple Vision Pro has a feature called spatial scenes. It lets you convert your photos into 3d, you view them through a portal. The effect breaks down if you move around too much from centre.

    3 likes

  • thetimwork's profile picture

    @thetimwork

    @its_thoth

    1 like

  • Joe Pea's profile picture

    @trusktr

    👏

    1 like

  • boy's profile picture

    @boymakesgames

    what is the size of the .ply it outputs?

    1 like

  • Tim Davison ᯅ's profile picture

    @timd_ca

    1.2 million Guassians. 66 MB. It's raw and full detail (32-bit), about 56 bytes per splat

    3 likes

  • boy's profile picture

    @boymakesgames

    not tooo bad

    1 like

  • Luis Bezzenberger's profile picture

    @bezzenberger

    Does this work for reflections?

    1 like