Skip navigation
Depth-aware light injection in TypeGPU
0:00 / 0:31

Depth-aware light injection in TypeGPU

Konrad Reczko

@reczko_konrad7.6K followers

Visit on X
49,777 likesDownloadOriginal post

10,120,189 views

Depth-aware light injection in TypeGPU I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into the lighting pass. It never has to leave the GPU or go through any extra synchronization/interop step Inference, lighting and draw all go through the same command encoder.

50 Comments

Comments from the original post on X (Twitter)

  • Konrad Reczko's profile picture

    @reczko_konrad

    Here are some fun intermediate textures, the depth and constructed normals

    2,448 likes

  • Fernando's profile picture

    @nandogarza

    Really nice. But you shouldn't cast a shadow when the light source is between you and the wall

    544 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    True, but it looks cool :D Doubles as fake godrays

    687 likes

  • Marc Krenn's profile picture

    @marc_krenn

    dope! what depth model are you using?

    150 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    Repacked DepthArt

    321 likes

  • Marc Krenn's profile picture

    @marc_krenn

    thanks, man! :)

    3 likes

  • dax's profile picture

    @thdxr

    cool

    116 likes

  • Rehan Ali Shah's profile picture

    @RehanAS21

    Can you share the link of the code you are using for this video. I want to try it.

    71 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    Sure! I’m building it as a TypeGPU example so it will be fully open and runnable just like my selfie segmentation inference

    362 likes

  • vogel's profile picture

    @ryanvogel

    woah this is insane wtf holyshit

    43 likes

  • グスタボ 🇧🇷's profile picture

    @g0ustalo

    can this add dynamic backgrounds in webcam?

    30 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    Yeah, you get raw depth and pixel color you can do whatever you want with it

    57 likes

  • Mustafa.'s profile picture

    @oprydai

    this is sickk

    23 likes

  • Rubén's profile picture

    @enrub_en

    You invented Navi! :)

    23 likes

  • Marc's profile picture

    @mrousavy

    Does this use the TrueDepth camera? VisionCamera?

    22 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    This on web but I will port it to Vision Camera on RN too

    58 likes

  • Marc's profile picture

    @mrousavy

    Nice! let me know how it goes, VisionCamera gives you depth frames via the CameraDepthOutput:

    6 likes

  • Konrad Reczko's profile picture

    @reczko_konrad

    I actually wanted to run my custom network and just use the external texture interface via react-native-webgpu. Worked nicely for me last time :D

    1 like

  • Marc's profile picture

    @mrousavy

    wdym exactly? no depth streaming? and which external texture interface do you mean, NativeBuffer?

    1 like

  • Konrad Reczko's profile picture

    @reczko_konrad

    Yup, exactly, I just take the raw camera info and run the inference via TypeGPU compute shaders

    3 likes