Depth-aware light injection in TypeGPU I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into... the lighting pass. It never has to leave the GPU or go through any extra synchronization/interop step Inference, lighting and draw all go through the same command encoder.show more

Here are some fun intermediate textures, the depth and constructed normals

Really nice. But you shouldn't cast a shadow when the light source is between you and the wall

True, but it looks cool :D Doubles as fake godrays

dope! what depth model are you using?

Repacked DepthArt

thanks, man! :)

cool

Can you share the link of the code you are using for this video. I want to try it.

Sure! I’m building it as a TypeGPU example so it will be fully open and runnable just like my selfie segmentation inference

woah this is insane wtf holyshit

can this add dynamic backgrounds in webcam?

Yeah, you get raw depth and pixel color you can do whatever you want with it

this is sickk

You invented Navi! :)

Does this use the TrueDepth camera? VisionCamera?

This on web but I will port it to Vision Camera on RN too

Nice! let me know how it goes, VisionCamera gives you depth frames via the CameraDepthOutput:

I actually wanted to run my custom network and just use the external texture interface via react-native-webgpu. Worked nicely for me last time :D

wdym exactly? no depth streaming? and which external texture interface do you mean, NativeBuffer?

Yup, exactly, I just take the raw camera info and run the inference via TypeGPU compute shaders

ah yes that is the perfect approach for stuff like this. if you build this with VisionCamera can you show me the code? :)

Of course, I love the lib btw, I can run code on the camera frames in a way that never reaches JS outside of native handles, brilliant

whoa

Dayummm!! Where's the code

Soon, I have to clean it up and handle licensing the weights

Waiting!!

wtf have you built?!?!?!

😳😍👌

what kind of sorcery is this

yoo this is so cool, is it opensource?

Soon (it will be joining the family of TypeGPU examples I will post an update when it’s ready

very cool. I can see it being very useful for live streaming.

@grok_owner if you could plz implement this in your livestream to have a fairy fly around you the chat would appreciate it

Brought some stable fluids to the party. Can confirm: TypeGPU is, ineed, fire 🔥🔥

Wow, definitely gonna check this model out

Sorcery

This is so sick!!!

that's smooth 🤘

@sivori Amazing!

the lighting physics are the most impressive part. very cool.

dude you are fucking brilliant

This project is super cool. Like a AR game haha.

Can we use lidar sensor on iphone for accurate depth ??

That would be amazing, on web I doubt it but on RN maybe

wizardry

The shadow effect annoys me, when the hand is between the camera and the light source the light would be directly hitting the wall there should not be a shadow casted from the hand

Yeah I get that a lot, good news is the shadow is togglable and I already took the feedback and tweaked the shadows so they are less offensive while still getting the “godray” effect which I like

Niceee, all the best!

I ve literally been working on something like this for web design. Your implementation is brilliant

brilliant stuff, konrad!