Gaussian Splatting in VR, all code written by me from scratch on DX12 + OpenVR. 2x2016x2240 res, no clever multi-view stuff yet so just running the whole render pipeline once per eye. Left the frame timer on so you can see I've been truthful about my perf :D

Now I've tried this I have a LOT of thoughts to share. 1) I was afraid the approximate affine transform of the gaussians into 2d would not be stereoscopically consistent. I am no longer concerned about that. By the time you get close enough you're out of stereo overlap.

2) I am thinking this is the future of geometry for VR. Why? - It is inherently anti-aliased, which is a big deal for VR - Its performance scales sublinearly with resolution - Geometric density in VR is more compelling than surface texture density

3) Even up close where the "illusion" of photorealism is broken, this is still really visually pleasant. It's like looking at brush strokes on a paint, vs. triangles when it's clipping plane city.

4) Photogrammetry / Novel view synthesis is not the most interesting use case, I want to be able to model these directly. Absolutely no reason you can't just replace SH colour with regular PBR material properties.

5) Speaking of Spherical Harmonics. For their massive data cost they don't add much. Bet you didn't even notice I disabled them for this video ;) (Not for perf reasons, they are by far one of the cheaper parts of the pipeline).

6) On aliasing, the fact we are "shading" per vertex let's us do interesting things like filtering the shaded geometry itself. In VR this could be a good way to combat specular aliasing, since we are always blending + averaging when using gaussian splats.

7) This performance is still unacceptably low. Let's put this in perspective, this is ~5M source primitives, and we're not doing any per vertex or pixel "shading". Not doing any sort of direct or indirect illumination.

8) So way slower than triangles. I think for this to be relevant for games, the geometry pipeline needs to be about 5-10x faster, and we need a way to scale to Billions of source primitives. Time to get working on that I guess.

Amazing work

Thanks!